digiKam 6.0.0 is released

Dear digiKam fans and users, following the long stage of integrating a lots of work from students during the Summer of Code, and after 2 years of intensive development, we are proud to announce the new digiKam 6.0.0.

Improvements and Fixes

The new major version 6.0.0 introduce these new features:

• Full support of video files management working just like photos.
• An integration of all import/export web-service tools in LightTable, Image editor and Showfoto.
• Raw file decoding engine supporting new cameras.
• Similarity data is now stored in a separate database file.
• Simplified web-service authentication using OAuth protocol.
• New tools to export to Pinterest, OneDrive and Box web-services.
• The capability to re-organize the icon-view contents manually.

A huge factoring of source code has been done to reduce external dependencies again with the goal to simplify application compilation, packaging and maintenance for the future.

The amount of closed reports for this release is 630 files.

In parallel to purging pending files in bugzilla, the whole digiKam source code (1.2M lines of C++ code) is parsed by three static source code analyzers (open-source): Clang scan-build, CppCheck, and Krazy. We have reduced to the minimum all reports generated by these tools and discovered some important in-deep dysfunctions. This also permits to review automatically all code written by students, before transitioning these implementations to production.

Exiv2 0.27 support

With this new release we are following the Exiv2 project which has released their new stable release 0.27.0. This library is a main component of digiKam used to interact with file metadata, like populating database contents, updating item textual information, or to handling XMP side-car for read only files.

digiKam 6.0.0 has been fixed to support the new API of this library.

Time-Adjust Tool

With this new release the famous TimeAdjust tool is back to quickly patch time-stamp of items, without having to use the Batch Queue Manager workflow. Adjusting the date is a common task that wont necessarily require as complex a tool as BQM. In all cases, even if TimeAdjust tool is now available everywhere in AlbumView, ImageEditor, LightTable, and Showfoto, a workflow with the possibility to fix date is always available in BQM as with previous releases.

DrMinGW Support

digiKam under Windows is more and more popular, and the number of reports in bugzilla has grown up seriously. This is not to imply that digiKam under Windows is less stable than on Linux, but end users start to use more advanced features day by day and discover new dysfunctions specific to Windows. After all, the Windows version is still in beta stage since many years and a lots of problems have been already fixed. Also, the port to Qt5 with less KDE dependencies has seriously stabilized the application and we now enter the last stage to stabilize all source code on this platform well.

To complete this task, digiKam is cross-compiled for Windows under Linux through MinGW (thanks to MXE project) and we include a special package dedicated to catch all technical information generated when a crash occurs, and to include better C++ exception support. DrMinGW is the tool dedicated to this task, similar to crash-course used under Linux to post debug traces in a report when a problem appears at runtime.

This will speed-up the investigations in the future to process Windows bug entries, without having to ask end-users to install a debugger and run the application in it. A recent example of DrMinGW feedback running digiKam under Windows can be seen on this file, with a clear crash located in Exiv2 shared library.

Video Files Management as Photo

Since a very long time, digiKam users would like to be able to manage videos just like photos. The digital camera world adds video support in all devices, with plenty of formats. The challenge to deal with these video media is to extract all the main metadata and populate the database. The Exiv2 shared library used in background by digiKam has video support which has never been finalized and the implementation is still unstable. Enabling video metadata support with Exiv2 would crash digiKam quickly in production.

So we have been forced to find an alternative to Exiv2 for video support. In release 5.5.0 we started to use the QtAV framework to play video media in digiKam. We chose QtAV because this framework directly uses ffmpeg codecs which de facto supports all formats very well. In contrast, the QtMultimedia framework requires extra platform codecs that end users need to install explicitly to obtain functional video support. We don’t want to force digiKam users to install extra programs for that. This must work automatically when digiKam is installed. digiKam must work as simple way to deal with video, as for photo.

So, after losing plenty of time with Exiv2 about video files, without any satisfying results, we started to use ffmpeg as metadata parser. FFmpeg provides a C API that we can use directly in digiKam to populate the database. It’s fast and powerful enough to catch all major information about video media. After three weeks of coding and testing, an FFmpeg wrapper for digiKam database interface has been finalized and is fully operational, without seeing any crash during collections scanning.

A video thumbnailer based exclusively on FFmpeg was also written. The previous one, using QtAV API introduced a high time latency while icon-view is populated with video media. The advanced search tool in digiKam has been patched to deal with all new metadata extracted from video media and stored in the database.

digiKam has the capability to play video files as well, without requiring an extra video player and dedicated codecs (thanks to FFmpeg and QtAV frameworks). One major dysfunction reported with previous beta release was about the embeded video slider used to jump into the stream contents which do not work anymore. This regression, introduced by changes in QtAV API require a long investigation to restore the feature.

Iconview : Separate Album Contents Feature

With this release, the Album management gets a new feature: separating icon-view items by properties. The default option does not separate items and provides a flat list. The new model can show album contents, automatically separated using virtual sub-albums, accordingly by mime types (RAW, JPEG, PNG, TIFF), or by the month they were shot in. To use this feature, go to the View/Separate Items menu entry.

Web-service Tools Available Everywhere

With this new 6.0.0 release, we added the capability to use all export tools everywhere in digiKam graphical interfaces. In fact, this is not only limited to export tools, but all main tools available in digiKam album view as the metadata and the geolocation editors, need to be available everywhere. This means one should be able to use these tools not only from album-view, but in Image Editor, in Light Table and in Showfoto.

So a new interface has been created, to access core application data, independently. Remember that Showfoto does not use a database, so we need a high level interface to define how to read or write data in core application. This interface is implemented with database support in digiKam and with more basic low level metadata access with Showfoto.

Currently, all tools are embedded in core application as well, but we plan for the near future to introduce a new plugin interface, only limited for digiKam uses. Experience showed, that sharing this kind of interface with other applications introduces complexity, instability and more work to maintain and stabilize the code. We want to be free to change it as we need in time without introducing dysfunctions in other 3rd party applications.

The Marble project does it very well. There are plenty of plugins, used to integrate new features everywhere. The plugins are separated shared libraries loaded on demand at run-time. An API permits to deal with the place to plug the tool and extend the application. This is exactly what digiKam needs for the future. For example, the batch queue manager will need to support all export tools to be include in users workflow automatically. Typically, the user wants to touch image step by step, automatically, and in parallel, (resize, fix color, convert, remove metadata, etc.), and at end to export the results to the cloud, without having to ask any information from the end user. The login information will be previously set in the workflow. A similar concept already exists in pro-photo application like LR.

New Camera Supported by Raw Files Engine

digiKam try to be the most powerful with all files provided by digital camera. Raw files support is a big challenge. Some applications have been especially created only to support RAW files from camera, as this kind of support is complex, long and hard to maintain in time.

Raw files are not like JPEG. Nothing is standardized, and camera makers are free to change everything inside these digital container without documentation. Raw files permit to re-invent the existing, to implement hidden features, to cache metadata, to require a powerful computer to process data. When you buy an expensive camera, you must expect that the image provided are seriously pre-processed by the camera firmware and ready to use immediately.

This is true for JPEG, not RAW files. Even if JPEG is not perfect, it’s well standardized and well documented. For Raw, for each new camera release, the formats can change as it depends in-depth on camera sensor data not processed by camera firmware. This require an intensive reverse-engineering that digiKam team cannot support as well. This is why we use the powerful libraw library to post-process the Raw files on the computer. This library include complex algorithms to support all different Raw file formats.

In this 6.0.0, we use the new version libraw 0.19 which introduce more than 200 new Raw formats, especially the most recent camera models available on the photo market. See the list below for details:

• Apple: Phone 8, iPhone 8 plus, iPhone X
• BlackMagic: URSA Mini 4k, URSA Mini 4.6k, URSA Mini Pro 4.6k
• Canon: PowerShot A410, A540, D10, ELPH 130 IS, ELPH 160 IS, SD750, SX100 IS,SX130 IS, SX160 IS, SX510 HS, SX10 IS, IXUS 900Ti PowerShot G1 X Mark III, G9 X Mark II, EOS 6D Mark II, EOS 77D, EOS 200D, EOS 800D, EOS M6, EOS M100
• Casio: EX-ZR4100/5100
• DJI: Phantom4 Pro/Pro+, Zenmuse X5, Zenmuse X5R
• FujiFilm: S6500fd, GFX 50S, X100f, X-A3, X-A5, X-A10, X-A20, X-E3, X-H1, X-T20
• Hasselblad: H6D-100c, A6D-100c
• Huawei: P9 (EVA-L09/AL00), Honor6a, Honor9, Mate10 (BLA-L29)
• Leica: CL, M10, TL2
• LG: V20 (F800K), VS995
• Nikon: D850, D5600, D7500, Coolpix B700
• Olympus: E-PL9, E-M10 Mark III, TG-5
• OnePlus: A3303, A5000
• Panasonic: DMC-FZ45, DMC-FZ72, DC-FZ80/82, DC-G9 (std. res mode only), DC-GF10/GF90, DC-GH5, DC-GX9, DC-GX800/850/GF9, DMC-LX1, DC-ZS70 (DC-TZ90/91/92, DC-T93), DC-TZ100/101/ZS100, DC-TZ200/ZS200
• PARROT: Bebop 2, Bebop Drone
• Pentax: KP
• PhaseOne: IQ3 100MP Trichromatic
• Samsung: Galaxy Nexus, Galaxy S3, S6 (SM-G920F), S7, S7 Edge, S8 (SM-G950U)
• Sony: A7R III, A9, DSC-RX0, DSC-RX10IV
• Yi: M1
• YUNEEC: CGO3, CGO3P
• Xiaoyi: YIAC3 (YI 4k)

This Libraw version is able to process in totality more than 1000 RAW formats. You can find the complete list in digiKam and Showfoto through the Help/Supported Raw Camera dialog. Thanks to Libraw team to share and maintain this experience.

New Similarity Database File

From the beginning of support of fuzzy and duplicate search (similarity searches) in digiKam, the data required to perform those searches were located in the core database which occupied a large part of the memory. Due to numerous improvements and extension of similarity search functionality in version 5.x, the impact on the core database became too strong. This lead to some performance issues including time latency. Also, further improvements of similarity searches depend on optimized database structures.

Thus, the structures needed for similarity searches were analyzed by Swati Lodha, an Indian student graduated from LNM Institute of Information Technology, during her Google Summer of Code project in 2017. The outcome of the project was the separation of the relevant structures and algorithms to form and access an additional database, the similarity database.

The work of Swati was then stabilized and finalized by Maik Qualmann and Mario Frank for digiKam 6.0.0.

The new database allows the similarity searches to work in the background while the core system is working on other tasks with much less lag. This also prevents the core database from bloating. The new database and implementation can be extended more easily with additional similarity search algorithms that may be better than the currently used. Also, we would now be able to work on the similarity searches performance in future.

Google Summer of Code Projects 2018

In February 2018, we started to select our next Google Summer of Code participants with few open projects for students. Three have been chosen for this summer: Tarek Talaat from Egypt, Thanh Trung Dinh from Vietnam, and Yungjie Liu from China.

Re-organize Icon-view Contents Manually

Yungjie Liu, a master student in the School of Software, Tsinghua University, works with us for the second time during this summer to implement a way to sort icon-view contents manually.

This feature was a long time wish in bugzilla, with many votes to see this functionality implemented in digiKam core. It’s now done for this 6.0.0 and everything is functional. The icon items position is recorded in digiKam database to be restored between session. You need of course to select the right sort option in the items menu.

Liu Yungjie has well explained how to use this new feature in this blog entry.

Web-Services Tools Factoring and Oauth Authentication.

Thanh Trung Dinh, an undergraduate student in Computer Engineering, from Université de Technologie de Compiègne, works with us for the first time during this summer to implement web service authentication using O2 library (Qt-based library implementing OAuth protocol), and develop a new wizard unifying all web services.

digiKam has already supported an export tool, allowing users to upload videos and photos to social networks, online storage or sharing platforms. The objectives of this project was:

• Reimplementing the current authentication flow for some web services in digiKam, that still use OAuth 1.0 protocol (requires user to copy-paste long token to wizard), and replacing it with new flow using OAuth2, which is more secure and much simpler (eliminating the copy-paste step),
• Simplifying and factoring code base to resolve inconsistency between core codes and user interface,
• Developing a new GUI, unifying all web services in one place.

OAuth was already implemented by Maik Qualmann in three web services: dropbox, imgur and flickr. Thanh continued to implement it on Google Drive, Google Photos, Facebook and Smugmug. Therefore, anytime users log in to one of these services above, they will be directed to web service login page. Loging in normally and then all the authentication will be realized automatically by digiKam. When users go back to digiKam, they can use export tool immediately.

Factoring code base is important mainly for the sake of reducing developer’s work in code maintaining. In addition, it is the foundation to implement the new GUI, which places all web services inside a single wizard. User will be able to choose web service they want to use and the account that they want to login. Then, with a browser embedded directly in the wizard, users can login there directly, which demonstrates a better performance than opening a separate browser for login. Moreover, other operations that can be done with old wizard will be ported to this new one, in order to keep user experience not only the same, but also better than the old one.

Helped by Maik Qualmann, Thanh has well completed the project and wrote a resume, including some interesting screenshots of new features in this report

New Web-service Tools For Pinterest, OneDrive and Box

Tarek Talaat, a student graduated from Computer and Systems Engineering department at faculty of engineering AinShams, work with us for the first time while this summer to implement new web-service export tools.

digiKam already supports exporting images to multiple web-services. The purpose of this project was to extend this support to make more web-service tools available to our users.

Tarek added three new web-services to digiKam export tools: Onedrive cloud service, Pinterest social network and Box remote store web-service.

Users can now select the desired web-service from the export menu, login to their accounts, provide authorization for digiKam through OAuth protocol, and upload photos to the desired web-service.

Helped by Mohamed Anwer, Tarek Talaat has well completed this project and write a resume in this report.

The Final Words

Thanks to all users for your support, all contributors, all students, and testers who permit to complete this important release during a very long code incubation. 2 years of development is a very long stage, and it’s always difficult to maintain a goal for a huge open source project as digiKam. The future is already here, with the next 6.1.0 which will include new important features. But it’s for later, be patient…

digiKam 6.0.0 source code tarball, Linux 32/64 bits AppImage bundles, MacOS package and Windows 32/64 bits installers can be downloaded from this repository.

Happy digiKaming…