digiKam has a new tool to convert RAW camera files to DNG…
Yes! a DNG converter for Linux...
For a long time, many people asked me to implement a RAW camera image converter to Digital NeGative (DNG). Why should one support DNG under Linux? To improve interoperability with other photo-management programs such as Adobe tools, and to improve Digital Asset Management (DAM), as Gerhard Kulzer discusses in this nice tutorial and in particular in Part 2.
I have already talked about DNG in another ticket. My first viewpoint about DNG SDK from Adobe has been wrong: missing tutorials, weird API documentation, wrong source code licence, etc... But after having found this nice web site from Barry Pearson which provides a huge collection of papers about DNG, I had a second look at the DNG SDK.
My first problem with DNG is to be able to extract un-demosaiced RAW image data to pass to DNG sdk. Here, dcraw cannot help really. By chance, after the port of libkdcraw to LibRaw, I have found an easy way to extract these data, plus lots of relevant meta-information, fully suitable for the DNG SDK. I would like to Alex Tutubalin from LibRaw for pointing this out.
After few weeks of studying the DNG SDK code to learn the API and the code from this little application dedicated to convert Canon Powershoot RAW files to DNG, I have finally found the way to write the first RAW to DNG converter for Linux, which support a lots of major RAW camera formats, as NEF, CR2, MRW, PAF, etc...
A DNG codec has been written around the DNG SDK and a command line tool to perform unit-tests. After a lots of conversions tests using official Adobe DNG converter and my program, a new kipi-plugin is born... See below for a screenshot of this tool in action...
The tool transfers RAW data to DNG, creates a JPEG preview and a thumbnail, sets a lots of TIFF-EP/DNG tags with original raw files metadata, does a backup of all Exif metadata and makernotes, and creates a set of XMP tags, accordingly. The generated is a valid DNG file, which is checked by Adobe DNGValidate tool...
The plan for the future is to improve the DNG tags creation following special makernotes entries, to support new RAW file formats, and to add a new option to backup the original RAW file within the DNG.
To conclude: DNG SDK is not perfect, but not too bad. I hope that Adobe will publish a future version using a real open source license such as BSD (ie, in the same way as the XMP SDK is published).
digiKam description extended on my website
I've added digiKam to the list of DNG Converters, and added it to the History page as the first Linux DNG Converter.
http://www.barrypearson.co.uk/articles/dng/products.htm#converters
http://www.barrypearson.co.uk/articles/dng/history.htm#year5
Thanks Barry...
Thanks Barry...
Note that the digiKam DNG converter compile fine also under Windows and MacOS-X, as it use Qt4 which is multi-platform API.
digiKam
I've updated the page again
I've changed the page to make it clear that is supports Windows and MacOS-X.
some hints
Thanks for the first open DNG converter!
I'm working on my own raw converter NOT based on dcraw. At the moment my converter supports only Canon CR2 files, but I think I have already some hints for you after a quick look at the source code:
- color matrix: you are using the color matrix from dcraw, dcraw is using only one color matrix and this matrix is a copy of one of the matrices of the Adobe DNG Converter. As described in the chapter "Mapping Camera Color Space to CIE XYZ Space" of the DNG specification two matrices could be used and as you are already using the Adobe matrices why not using them completely?
- camera calibration: Adobe calculates the color matrices with a reference cam. Every camera has a different calibration. The calibration matrix can be calculated by comparing the 5200K neutral white balance values from the reference cam against your 5200K values (the same values for both calibration matrices). (Only tested with Canon cams!)
bergi
color matrix and calibration
> - color matrix: you are using the color matrix from dcraw, dcraw is using only one color >matrix and this matrix is a copy of one of the matrices of the Adobe DNG Converter. As >described in the chapter "Mapping Camera Color Space to CIE XYZ Space" of the DNG >specification two matrices could be used and as you are already using the Adobe matrices >why not using them completely?
Sure, but how to get this values from Adobe. There are harcoded in DNG converter binary program...
The only way that i can see is to run Adobe DNG convert over a huge RAW file collection and to extract color matrix as well using Exiftool for ex. I suspect that color matrix from dcraw.c have been generated like this, or i miss something ?
> - camera calibration
I have not yet deal with this values. It's on my todo list...
digiKam
> Sure, but how to get this
> Sure, but how to get this values from Adobe. There are harcoded in DNG converter binary program...
The only way that i can see is to run Adobe DNG convert over a huge RAW file collection and to extract color matrix as well using Exiftool for ex. I suspect that color matrix from dcraw.c have been generated like this, or i miss something ?
I have done it this way and I also think the dcraw matrices were generated this way. I have done this already for some Canon cams. You can find them in nice readable xml/rdf format in the camdefinitions folder of my raw converter. Would be cool to agree to a common format.
Perhaps you already know this page, but there are some more raw examples: http://www.rawsamples.ch/
> I have not yet deal with this values. It's on my todo list...
Feel free to ask when you have reached this item on your todo list.
ok. i can see it.
>I have done it this way and I also think the dcraw matrices were generated this way. I have >done this already for some Canon cams. You can find them in nice readable xml/rdf format in >the camdefinitions folder of my raw converter. Would be cool to agree to a common format.
ok. i can see it. xml is the right way to store these values. It will be fine to use the same file format. But where to store this file ? LibRaw ?
>Perhaps you already know this page, but there are some more raw examples: >http://www.rawsamples.ch/
yes i know. there are also others RAW repositories on the web.
Gilles
> ok. i can see it. xml is
> ok. i can see it. xml is the right way to store these values. It will be fine to use the same file format. But where to store this file ? LibRaw ?
Yes, LibRaw is in my opinion the right place, but this will be a bigger change. Until now everything is in the source code.
>ok. i can see it. xml is the
>ok. i can see it. xml is the right way to store these values. It will be fine to use the same file format. But where to store this file ? LibRaw ?
LibRaw will not store _additional_ DNG-related data due to three reasons:
* Copyright issues. LibRaw already have one set of DNG ColorMatrix tables derived from dcraw code. In this case we rely on Coffin's copyright clearance and free status of dcraw itself.
* the only programs that needs/uses two color matrices in DNG file are Adobe programs (Lightroom and Camera Raw). If you already use Adobe programs, you can also use free Adobe DNG converter.
* The 'DNG processing model' (quoting from Adobe staff member in dpreview forums) introduced in DNG 1.2 is much more complex, than in DNG 1.0/1.1. Should LibRaw store all (possibly copyrighted) Adobe color data in source code? I don't think so.
We (LibRaw team) dislike 'DNG processing model' because of very poor image quality produced by Adobe products (try compare with Raw Photo Processor): high noise levels (produced by mixing low noise Green with high noise Red channels), false colors (produced by using ProPhoto as internal space) and poor quality of color profiles (produced by small gamut of color targed used).
In my thoughts, the only place to store DNG-related color data is in DNG converter itself. I've added cam_xyz[] field to LibRaw interface only because it was a very small step (2 lines of code and 5 lines in documentation).
> * the only programs that
> * the only programs that needs/uses two color matrices in DNG file are Adobe programs (Lightroom and Camera Raw). If you already use Adobe programs, you can also use free Adobe DNG converter.
You can only know about open source raw converters. Until now all open source raw converters are based on dcraw and I think it's wrong to use only one matrix just because dcraw doesn't implement the full DNG standard.
> * The 'DNG processing model' (quoting from Adobe staff member in dpreview forums) introduced in DNG 1.2 is much more complex, than in DNG 1.0/1.1. Should LibRaw store all (possibly copyrighted) Adobe color data in source code? I don't think so.
My extracted color matrices are equal in every digit to the dcraw matrices. I don't know much about the copyright in this situation, but the matrices are extracted from a DNG file (the owner of this DNG file is not Adobe!) and not from the binary. My idea was also not to store it in the source, I'm storing it in a xml/rdf file. This makes it also easier to add your own matrices.
> We (LibRaw team) dislike 'DNG processing model' because of very poor image quality produced by Adobe products (try compare with Raw Photo Processor): high noise levels (produced by mixing low noise Green with high noise Red channels), false colors (produced by using ProPhoto as internal space) and poor quality of color profiles (produced by small gamut of color targed used).
I implemented my own raw converter just because of the nice colors I got from Lightroom. For example dcraw completely ignores the camera calibration. With the DNG model two different cameras from the same model will produce the same colors. I think the noise you are talking about is a result of a bad demosaic filter, dcraw has already a better one, so you don't have to care about it. Small gamut? You know this page? http://www.brucelindbloom.com/index.html?WorkingSpaceInfo.html
> In my thoughts, the only place to store DNG-related color data is in DNG converter itself. I've added cam_xyz[] field to LibRaw interface only because it was a very small step (2 lines of code and 5 lines in documentation).
If they are there just for information you are right, but it would be nice if LibRaw would also use them.
> I don't know much about the
> I don't know much about the copyright in this situation, but the matrices are extracted from a DNG file (the owner of this DNG file is not Adobe!)
Adobe refused to use Nikon white balance extracted and decrypted from NEF files (those also do not belong to Nikon) saying they do not want to violate DMCA.
> Adobe refused to use Nikon
> Adobe refused to use Nikon white balance extracted and decrypted from NEF files (those also do not belong to Nikon) saying they do not want to violate DMCA.
That's a different thing. To extract the white balance you have to decrypt them, the algorithm is unknown, so you have to decompile Nikons binaries.
> To extract the white
> To extract the white balance you have to decrypt them, the algorithm is unknown, so you have to decompile Nikons binaries.
Well, not exactly. One can simply use dcraw or make a set of planned experiments (like some actually did) to get the way to decipher WB.
>My idea was also not to
>My idea was also not to store it in the source, I'm storing it in a xml/rdf file. This makes it also easier to add your own matrices.
External database is a very good idea.
>For example dcraw completely ignores the camera calibration.
You can use camera ICC profile with dcraw.
>Small gamut? You know this page? http://www.brucelindbloom.com/index.html?WorkingSpaceInfo.html
ProPhoto working space has very (very-very) large gamut. But 24-square ColorChecker (used for profiling; also CC24-profiling built in DNG Profile editor) has very small gamut. So, you need to extrapolate your picture data into wide-gamut space using low-gamut profile data. Result is far from perfect.
>If they are there just for information you are right, but it would be nice if LibRaw would also use them.
LibRaw is not for color processing. LibRaw is for RAW data extraction (very small and limited). Of course, without interpolation-postprocessing code RAW data is not useful, so we've included dcraw postprocessing code into LibRaw (as quick and durty solution). This code is 'good enough' (look around: 90% of raw-enabled software uses either unmodified dcraw code or external dcraw binary and users are happy /enough/), we do not plan to extend this code.
The next steps with LibRaw will be
- get maximum from raw data (masked pixels, de-banding) - work in progress now;
- get some from exif data - metadata required for raw processing (tone curves, wb settings and so).
> External database is a very
> External database is a very good idea.
Than we should agree to a common format. My proposal is xml/rdf, you can look into my source for examples.
> ProPhoto working space has very (very-very) large gamut. But 24-square ColorChecker (used for profiling; also CC24-profiling built in DNG Profile editor) has very small gamut. So, you need to extrapolate your picture data into wide-gamut space using low-gamut profile data. Result is far from perfect.
OK, we can do it better if we use our own matrices created with a better color checker. But at the moment dcraw isn't better in this topic, because it's using Adobes matrices.
> LibRaw is not for color processing. LibRaw is for RAW data extraction (very small and limited). Of course, without interpolation-postprocessing code RAW data is not useful, so we've included dcraw postprocessing code into LibRaw (as quick and durty solution). This code is 'good enough' (look around: 90% of raw-enabled software uses either unmodified dcraw code or external dcraw binary and users are happy /enough/), we do not plan to extend this code.
You just don't want to extend the dcraw code or you really don't want to include more postprocessing code? If you would include it you can take my code, you will have to change it in some places because it's based on my metadata model, but it's GPL.
> - get some from exif data - metadata required for raw processing (tone curves, wb settings and so).
Tone curves would be cool. I think Canon has one in the makernotes, but some preprocessing is required to use it with linear gamma.
White balance belongs to the postprocessing, you should take a look at my code.
>Than we should agree to a
>Than we should agree to a common format. My proposal is xml/rdf, you can look into my source for examples.
In general case, camera color may (and should?) be described in terms of ICC profile. With high quality
profiles you don't need to have two rgb-xyz matrices for different color temperatures and interpolation between these matrices (as in DNG processing model). Camera does not change with light source.
For DNG case you need to store two 3x3 or 3x4 matrices for DNG 1.1 and some another data for DNG 1.2. These data is identified by make/model pair. So, any documented (and extensible) format is good enough. I do not like XML much due to (de)serialization overhead on large data amounts, but for several hundreds of camera models it does not make sense.
>You just don't want to extend the dcraw code or you really don't want to include more postprocessing code?
LibRaw goal is to read and decode RAW files (image data + metadata need for postprocessing phase). dcraw_* postprocessing calls added only for possible 'immediate use' in 3rd-party apps such as digiKam :).
The next phase will be RAW analysis tools (like Rawnalyze)
Full RAW converter (with focus on quality, so with FP-math, accurate ICC code and so on) is not small task. We definitely will do either (extensible) postprocessing library or (extensible) GUI converter or both. Definitely, not in this year.
> In general case, camera
> In general case, camera color may (and should?) be described in terms of ICC profile. With high quality
profiles you don't need to have two rgb-xyz matrices for different color temperatures and interpolation between these matrices (as in DNG processing model). Camera does not change with light source.
I agree, but we should support both because building a big ICC profile database will take some time.
> For DNG case you need to store two 3x3 or 3x4 matrices for DNG 1.1 and some another data for DNG 1.2. These data is identified by make/model pair. So, any documented (and extensible) format is good enough. I do not like XML much due to (de)serialization overhead on large data amounts, but for several hundreds of camera models it does not make sense.
What it is your proposal? What I don't want is a text format like the channels.conf from VDR. There are many different extension, but everyone incompatible to the other. Is a database (SQLite) an option for you?
> LibRaw goal is to read and decode RAW files (image data + metadata need for postprocessing phase). dcraw_* postprocessing calls added only for possible 'immediate use' in 3rd-party apps such as digiKam :).
The next phase will be RAW analysis tools (like Rawnalyze)
Full RAW converter (with focus on quality, so with FP-math, accurate ICC code and so on) is not small task. We definitely will do either (extensible) postprocessing library or (extensible) GUI converter or both. Definitely, not in this year.
I'm working already on a raw converter, but not based on dcraw because I want full control. I also have already tested camera ICC profiles, but the Canon profiles are only lookup tables with a gamma curve applied. I will investigate later more in this topic, but first I want to make my converter more usable. Definitely we should stay in contact.
Are you sure that no metadata
Are you sure that no metadata is lost in your converter? How did you verify this?
Exiftool and Exiv2...
I check metadata with Exiftool and Exiv2.
Of course not all metadata are not yet backported properlly. But at least 80-90% of original are saved. I use Exiv2 to read original and DNG sdk api to write to DNG. In the future i will use Exiv2 to write in DNG, because Adobe sdk is limited.
digiKam
My reason to not use DNG is
My reason to not use DNG is just this. Its never sure that all meta is ported correctly, even with adobe converter.
Hello. I work on the Adobe
Hello. I work on the Adobe Camera Raw and Lightroom teams as a computer scientist / engineer. I also work on the DNG SDK, DNG Profile Editor, DNG processing model, DNG spec, etc.
The DNG Converter copies over ALL the tags from the original file (CR2, NEF, ARW, etc.) into the DNG, even the ones it does not understand. This includes private MakerNote information for which Adobe generally has very little information. By copying it over directly, bit for bit, we do not throw out proprietary data.
thank you sohbet
thank you
sohbet
you are using the color
you are using the color matrix from dcraw, dcraw is using only one color matrix and this matrix is a copy of one of the matrices of the Adobe DNG Converter. The calibration matrix can be calculated by comparing the 5200K neutral white balance values from the reference cam against your 5200K values (the same values for both calibration matrices).
Well, I have done it this way
Well, I have done it this way and I also think that the dcraw matrices were generated this way.