digiKam 6.0.0 beta 1 is released

Dear digiKam fans and users, following the long stage of integrating a lots of work from students during the Summer of Code we are proud to announce the first beta of digiKam 6.0.0.

Improvements and Fixes

The next major version 6.0.0 looks promising, and will introduce new features as:

• Full support of video files management working as photos.
• An integration of all import/export web-service tools in LightTable, Image editor and Showfoto.
• Raw file decoding engine supporting new cameras.
• Similarity data is now stored in a separate file.
• Simplified web-service authentication using OAuth protocol.
• New tools to export to Pinterest, OneDrive and Box web-services.
• The capability to re-organize the icon-view contents manually.

A huge factoring of source code has been done to reduce external dependencies again with the goal to simplify application compilation, packaging and maintenance for the next years.

The reports already closed due to new implementations are more than 350 files, but nothing is completed yet, as we plan a few more beta releases before publishing the final 6.0.0 release.

Video Files Management as Photo

Since a very long time, digiKam users would like to be able to manage videos just like photos. The digital camera world adds video support in all devices, with plenty of formats. The challenge to deal with these video media is to extract all the main metadata and populate the database. The Exiv2 shared library used in background by digiKam has video support which has never been finalized and the implementation is still unstable. Enabling video metadata support with Exiv2 would crash digiKam quickly in production.

So we have been forced to find an alternative to Exiv2 for video support. In release 5.5.0 we started to use the QtAV framework to play video media in digiKam. We chose QtAV because this framework directly uses ffmpeg codecs which de facto supports all formats very well. In contrast, the QtMultimedia framework requires extra platform codecs that end users need to install explicitly to obtain functional video support. We don’t want to force digiKam users to install extra programs for that. This must work automatically when digiKam is installed. digiKam must work as simple way to deal with video, as for photo.

So, after losing plenty of time with Exiv2 about video files, without any satisfying results, we started to use ffmpeg as metadata parser. FFmpeg provides a C API that we can use directly in digiKam to populate the database. It’s fast and powerful enough to catch all major information about video media. After three weeks of coding and testing, an FFmpeg wrapper for digiKam database interface has been finalized and is fully operational, without seeing any crash during collections scanning.

A video thumbnailer based exclusively on FFmpeg was also written. The previous one, using QtAV API introduced a high time latency while icon-view is populated with video media. The advanced search tool in digiKam has been patched to deal with all new metadata extracted from video media and stored in the database.

Web-service Tools Available Everywhere

With this new 6.0.0 release, we added the capability to use all export tools everywhere in digiKam graphical interfaces. In fact, this is not only limited to export tools, but all main tools available in digiKam album view as the metadata and the geolocation editors, need to be available everywhere. This means one should be able to use these tools not only from album-view, but in Image Editor, in Light Table and in Showfoto.

So a new interface has been created, to access core application data, independently. Remember that Showfoto does not use a database, so we need a high level interface to define how to read or write data in core application. This interface is implemented with database supp port in digiKam and with more basic low level metadata access with Showfoto.

Currently, all tools are embedded in core application as well, but we plan for the near future to introduce a new plugin interface, only limited for digiKam uses. Experience showed, that sharing this kind of interface with other applications introduces complexity, instability and more work to maintain and stabilize the code. We want to be free to change it as we need in time without introducing dysfunctions in other 3rd party applications.

The Marble project does it very well. There are plenty of plugins, used to integrate new features everywhere. The plugins are separated shared libraries loaded on demand at run-time. An API permits to deal with the place to plug the tool and extend the application. This is exactly what digiKam needs for the future. For example, the batch queue manager will need to support all export tools to be include in users workflow automatically. Typically, the user wants to touch image step by step, automatically, and in parallel, (resize, fix color, convert, remove metadata, etc.), and at end to export the results to the cloud, without having to ask any information from the end user. The login information will be previously set in the workflow. A similar concept already exists in pro-photo application like LR.

New Camera Supported by Raw Files Engine

digiKam try to be the most powerful with all files provided by digital camera. Raw files support is a big challenge. Some applications have been especially created only to support RAW files from camera, as this kind of support is complex, long and hard to maintain in time.

Raw files are not like JPEG. Nothing is standardized, and camera makers are free to change everything inside these digital container without documentation. Raw files permit to re-invent the existing, to implement hidden features, to cache metadata, to require a powerful computer to process data. When you buy an expensive camera, you must expect that the image provided are seriously pre-processed by the camera firmware and ready to use immediately.

This is true for JPEG, not RAW files. Even if JPEG is not perfect, it’s well standardized and well documented. For Raw, for each new camera release, the formats can change as it depends in-depth on camera sensor data not processed by camera firmware. This require an intensive reverse-engineering that digiKam team cannot support as well. This is why we use the powerful libraw library to post-process the Raw files on the computer. This library include complex algorithms to support all different Raw file formats.

In this 6.0.0 beta1, we use the new version libraw 0.19 which introduce more than 200 new Raw formats, especially the most recent camera models available on the photo market. See the list below for details:

• Apple: Phone 8, iPhone 8 plus, iPhone X
• BlackMagic: URSA Mini 4k, URSA Mini 4.6k, URSA Mini Pro 4.6k
• Canon: PowerShot A410, A540, D10, ELPH 130 IS, ELPH 160 IS, SD750, SX100 IS,SX130 IS, SX160 IS, SX510 HS, SX10 IS, IXUS 900Ti PowerShot G1 X Mark III, G9 X Mark II, EOS 6D Mark II, EOS 77D, EOS 200D, EOS 800D, EOS M6, EOS M100
• Casio: EX-ZR4100/5100
• DJI: Phantom4 Pro/Pro+, Zenmuse X5, Zenmuse X5R
• FujiFilm: S6500fd, GFX 50S, X100f, X-A3, X-A5, X-A10, X-A20, X-E3, X-H1, X-T20
• Hasselblad: H6D-100c, A6D-100c
• Huawei: P9 (EVA-L09/AL00), Honor6a, Honor9, Mate10 (BLA-L29)
• Leica: CL, M10, TL2
• LG: V20 (F800K), VS995
• Nikon: D850, D5600, D7500, Coolpix B700
• Olympus: E-PL9, E-M10 Mark III, TG-5
• OnePlus: A3303, A5000
• Panasonic: DMC-FZ45, DMC-FZ72, DC-FZ80/82, DC-G9 (std. res mode only), DC-GF10/GF90, DC-GH5, DC-GX9, DC-GX800/850/GF9, DMC-LX1, DC-ZS70 (DC-TZ90/91/92, DC-T93), DC-TZ100/101/ZS100, DC-TZ200/ZS200
• PARROT: Bebop 2, Bebop Drone
• Pentax: KP
• PhaseOne: IQ3 100MP Trichromatic
• Samsung: Galaxy Nexus, Galaxy S3, S6 (SM-G920F), S7, S7 Edge, S8 (SM-G950U)
• Sony: A7R III, A9, DSC-RX0, DSC-RX10IV
• Yi: M1
• YUNEEC: CGO3, CGO3P
• Xiaoyi: YIAC3 (YI 4k)

This Libraw version is able to process in totality more than 1000 RAW formats. You can find the complete list in digiKam and Showfoto through the Help/Supported Raw Camera dialog. Thanks to Libraw team to share and maintain this experience.

New Similarity Database File

From the beginning of support of fuzzy and duplicate search (similarity searches) in digiKam, the data required to perform those searches were located in the core database which occupied a large part of the memory. Due to numerous improvements and extension of similarity search functionality in version 5.x, the impact on the core database became too strong. This lead to some performance issues including time latency. Also, further improvements of similarity searches depend on optimized database structures.

Thus, the structures needed for similarity searches were analyzed by Swati Lodha, an Indian student graduated from LNM Institute of Information Technology, during her Google Summer of Code project in 2017. The outcome of the project was the separation of the relevant structures and algorithms to form and access an additional database, the similarity database.

The work of Swati was then stabilized and finalized by Maik Qualmann and Mario Frank for digiKam 6.0.0.

The new database allows the similarity searches to work in the background while the core system is working on other tasks with much less lag. This also prevents the core database from bloating. The new database and implementation can be extended more easily with additional similarity search algorithms that may be better than the currently used. Also, we would now be able to work on the similarity searches performance in future.

Google Summer of Code Projects 2018

In February 2018, we started to select our next Google Summer of Code participants with few open projects for students. Three have been chosen for this summer: Tarek Talaat from Egypt, Thanh Trung Dinh from Vietnam, and Yungjie Liu from China.

Re-organize Icon-view Contents Manually

Yungjie Liu, a master student in the School of Software, Tsinghua University, works with us for the second time during this summer to implement a way to sort icon-view contents manually.

This feature was a long time wish in bugzilla, with many votes to see this functionality implemented in digiKam core. It’s now done for next 6.0.0 and everything should be functional for testing with this beta 1 release. The icon items position is recorded in digiKam database to be restored between session. You need of course to select the right sort option in the items menu.

Liu Yungjie has well explained how to use this new feature in this blog entry.

Web-Services Tools Factoring and Oauth Authentication.

Thanh Trung Dinh, an undergraduate student in Computer Engineering, from Université de Technologie de Compiègne, works with us for the first time during this summer to implement web service authentication using O2 library (Qt-based library implementing OAuth protocol), and develop a new wizard unifying all web services.

digiKam has already supported an export tool, allowing users to upload videos and photos to social networks, online storage or sharing platforms. The objectives of this project was:

• Reimplementing the current authentication flow for some web services in digiKam, that still use OAuth 1.0 protocol (requires user to copy-paste long token to wizard), and replacing it with new flow using OAuth2, which is more secure and much simpler (eliminating the copy-paste step),
• Simplifying and factoring code base to resolve inconsistency between core codes and user interface,
• Developing a new GUI, unifying all web services in one place.

OAuth was already implemented by Maik Qualmann in three web services: dropbox, imgur and flickr. Thanh continued to implement it on Google Drive, Google Photos, Facebook and Smugmug. Therefore, anytime users log in to one of these services above, they will be directed to web service login page. Loging in normally and then all the authentication will be realized automatically by digiKam. When users go back to digiKam, they can use export tool immediately.

Factoring code base is important mainly for the sake of reducing developer’s work in code maintaining. In addition, it is the foundation to implement the new GUI, which places all web services inside a single wizard. User will be able to choose web service they want to use and the account that they want to login. Then, with a browser embedded directly in the wizard, users can login there directly, which demonstrates a better performance than opening a separate browser for login. Moreover, other operations that can be done with old wizard will be ported to this new one, in order to keep user experience not only the same, but also better than the old one.

Helped by Maik Qualmann, Thanh has well completed the project and wrote a resume, including some interesting screenshots of new features in this report

New Web-service Tools For Pinterest, OneDrive and Box

Tarek Talaat, a student graduated from Computer and Systems Engineering department at faculty of engineering AinShams, work with us for the first time while this summer to implement new web-service export tools.

digiKam already supports exporting images to multiple web-services. The purpose of this project was to extend this support to make more web-service tools available to our users.

Tarek added three new web-services to digiKam export tools: Onedrive cloud service, Pinterest social network and Box remote store web-service.

Users can now select the desired web-service from the export menu, login to their accounts, provide authorization for digiKam through OAuth protocol, and upload photos to the desired web-service.

Helped by Mohamed Anwer, Tarek Talaat has well completed this project and write a resume in this report.

The Final Words: How to Test This Beta Version

Due to many changes in database schema, do not use this beta1 in production yet and backup your database files before testing it. It’s recommended to try this beta1 with a small collection of items first, and to add new photo and video step by step.

Thanks to all users for your supports and help during the beta stage. We plan other beta releases during this year, to complete the final release of 6.0.0 in December.

digiKam 6.0.0 beta1 source code tarball, Linux 32/64 bits AppImage bundles, MacOS package and Windows 32/64 bits installers can be downloaded from this repository.

Happy digiKam testing…