DNG encoding

[dgalland]

I think it's essential for any photo device to offer proper raw/dng encoding. In Picamera2, DNG encoding is done by a third party PiDNG library. This has the merit of existing, but it's far from optimal, as the encoding is extremely slow, probably due to the use of Python, despite good use of numpy.
I think the Raspberry Foundation should take on the task of making this encoding more efficient.
I'd also like to point out that when using DNG encoding, do not use CSI2P compressed formats, as decompression before encoding is catastrophically slow!
Regards

[davidplowman]

Do you have a specific example script that you could share that you feel is slow? I have in the past occasionally helped people get quite good performance saving DNG files, such as here: https://forums.raspberrypi.com/viewtopic.php?t=360334 (sorry, quite a long thread, but we were able to get some pretty blazing speeds using a thread pool and an NVMe drive).

My recollection was that you have to avoid using packed or compressed formats. Of course it's extremely sensitive to the speed of whatever disk you're writing too - that Samsung NVMe was just awesome and blew away my SATA SSDs. I don't recall that I thought the speed of PiDNG was an issue, but it was a while ago. We do know the author of PiDNG, and I'm sure he would be amenable to improvements if we could identify any.

[dgalland]

I'm coming back to this discussion because the status of the PiDNG version is not clear to me. While doing some tests of a compression of the raw (uncompressed) request buffer outside of picamera2 I noticed that the github version of PiDNG generates a bad DNG, I don't know why?
I also see that the rpicam-apps github has a C++ function that does the same processing, wouldn't it make sense to use it from picamera2?

[davidplowman]

I'm coming back to this discussion because the status of the PiDNG version is not clear to me.

It's a 3rd party library, like many others that we use, but we do know the author as he's a Pi user and is normally helpful to us in implementing features or fixing problems.

While doing some tests of a compression of the raw (uncompressed) request buffer outside of picamera2 I noticed that the github version of PiDNG generates a bad DNG, I don't know why?

I'm not aware of any problems myself, though we would recommend installing the PiDNG from our apt package as that's what we've tested. If there are bugs, I would encourage them to be reported, otherwise they might not get fixed. If you have no luck reporting them in the PiDNG repo, you could report them here and we could see if there's anything we can do to help.

I also see that the rpicam-apps github has a C++ function that does the same processing, wouldn't it make sense to use it from picamera2?

In C++ we use libtiff because it's the only library I know of that even comes close to writing DNG files. However, I'm not very keen on it. It has some strange behaviours that I don't understand requiring hacky workarounds to produce usable DNG files. Recently, also, there was an issue where some versions of libtiff "misplaced" some of the tags that are necessary for DNG files, breaking them completely. It seems to me that maintenance of DNG file support is not always great, and DNG files are a bit of a 2nd class citizen there. It also has no Python bindings (AFAIK) so there's be further work to do. I would rate our odds of getting things fixed and improved in PiDNG as a lot higher.

But in general - if there are any issues, please report them. We'll always do what we can to fix and improve things.

[dgalland]

Thanks for your answer, no problems actually with the installed version of PiDNG. It was just a remark about a potential problem with the Github version, there is one more SampleFormat tag and it's normal but the DNG file does not seem correct. I did not investigate further. Nothing urgent in any case.

[davidplowman]

Sure, no problem. But if anything does crop up that isn't working properly, please do let us know! :)

[mu-files-dev]

You might be interested in this new python package that is fairly fast for DNG encode - https://github.com/mu-files/mu-image/tree/main/muimg

[davidplowman]

Hi, and thanks for the info. If you were interested in supplying a "how to save raw camera image using mu-files" example Python script, we'd be very happy to merge and distribute it with our other examples. Do ask if you have any questions about that. Obviously what we have at the moment seems to work well for us, do you think there are any specific benefits that we could point to? Thanks!

[mu-files-dev]

Thanks for this reply. That sounds like a great idea. Can you point me to the examples that you distribute (especially any you do with saving DNG) so that I can see the format? Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: davidplowman ***@***.***> Sent: Tuesday, May 5, 2026 1:13:30 AM To: raspberrypi/picamera2 ***@***.***> Cc: mu-files ***@***.***>; Comment ***@***.***> Subject: Re: [raspberrypi/picamera2] DNG encoding (Discussion #1057) Hi, and thanks for the info. If you were interested in supplying a "how to save raw camera image using mu-files" example Python script, we'd be very happy to merge and distribute it with our other examples. Do ask if you have any questions about that. Obviously what we have at the moment seems to work well for us, do you think there are any specific benefits that we could point to? Thanks! — Reply to this email directly, view it on GitHub<#1057 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BSWFDDBEKOQGPYY3L2S7YCL4ZGPCVAVCNFSM6AAAAACYOTKDRKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMOBRGM3DINA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[davidplowman]

Sure, there are a number of examples, including this one. However, I might be tempted just to use this code:

import time
from picamera2 import Picamera2, MappedArray

picam2 = Picamera2()

raw = {'format': 'SBGGR12'}  # make sure we get an uncompressed buffer
config = picam2.create_still_configuration(raw=raw, buffer_count=2)
picam2.configure(config)
picam2.start()

time.sleep(1)

with picam2.captured_request() as request:
    with MappedArray(request, 'raw') as m:
        print(request.get_metadata())
        print(m.array)
        print(picam2.camera_configuration()['raw'])

The print statements at the end show you how to get the metadata associated with that capture (exposure time, colour matrix etc.), the raw image array (without copying it), and the buffer format (do you have a Raspberry Pi of some kind?). Hopefully you can just replace those lines with code that saves a DNG. Please ask if you have any questions!

[davidplowman]

Alternatively, if you want to send me the runes for saving a numpy array as a DNG file, maybe with fixed exposure/gain, colour balance etc. I could probably fill in the rest. We're just talking a few lines of code, I assume? Thanks.

[mu-files-dev]

I should have something in the next few days. I do have a raspberry pi. This project started with me wanting to save astro images captured with a USB camera through a pi. Before sending over the code I wanted to test with one of supported cameras. So I ordered an Pi HQ Camera and should get it today.

And yes it should just be a few lines of code - a bit of metadata copying followed by a call to write_dng_from_array(...)

[mu-files-dev]

I uploaded an example here: https://github.com/mu-files/mu-rasppi

The important function is write_muimg() in picamera2_capture.py

The code also benchmarks PiDNG and muimg on a captured image. I added some details to the README with benchmark results and advantages/disadvantages of each library.

Note1: I needed to create a branch of muimg due to python version issues. The tifffile package muimg depends on is at python 3.12 so had to down-version that in the branch.
Note2: I could not get jpeg compress in PiDNG to work. It produces a valid image in my benchmark but it is garbled. Would like to update if you have a suggested fix.

[davidplowman]

Thanks for doing this, that's very interesting. A few of things to discuss...

Firstly, the latest Raspberry Pi OS (Trixie) uses Python 3.13 and numpy 2.2.something, so you shouldn't need any compatibility branches or anything. I've been using the regular versions of everything just fine - maybe just double-check what OS version you have?

The latest Raspberry Pi OS may also be using a more recent version of PiDNG. But your benchmarks are still showing approximately a 50% performance improvement, so that's still very good. I've never used the PiDNG compression modes, so can't really tell you anything about it, and our users have never asked!

I also put together an example showing what I think is probably the most direct way to save DNG files on the Pi, if users want the extra performance or features. Perhaps you could let me know if I've done anything obviously wrong there. I'm happy to host an example like this, or for you to do it (instead or as well) if that were useful.

I probably can't integrate muimg directly into Picamera2 because of the license (interesting as that might be). I think users will have to download and install it for themselves, being fully aware what the licensing requirements are. Anyway, maybe let me know if you're happy simply for me to host that example in our repo like this, or would like to do anything different. Thanks!

[mu-files-dev]

Thanks for producing this example. Did it run with no problem for you?

I think that I have an older image on my raspberry pi and I'm embarrassed to say I asked Claude how to install my code and it gave me the impression that Trixie was still a beta. I'll try to re-image soon.

It would be great if you could host it and if you see fit, give it a bit of promotion. If you do that please let me know and I will add a link from your github.

Overall the example looks great. A few notes in looking through it:

• You don't need the CameraCalibration matrices (they default to identity). You especially don't need CameraCalibration2 since there is no corresponding ColorMatrix2. Or you don't need to set the calibration matrices or illuminant at all as write_dng will set SRGB to connection-space matrices if none are provided.
• If you want you can use python "native" types when setting the metadata - eg you can just set ccm directly without converting to rationals
• brucelindbloom site link does not work. It needs to be http (doesn't offer https)
• We are sticking with the current license for now, but may change to something FOSS in the future. My preference is to describe the current license as - muimg "free to use except for large commercial enterprises" rather than the comment in the code - "it may not be free to use" (which is a bit more ambiguous).

On compression - for certain scenarios that don't require max write throughput I think that it is interesting. Every DNG coming out of Adobe's RAW convertor uses lossless jpeg. I started this project wanting to store 15s timelapse images in RAW. I found that the JXL lossy mode gave me access to the RAW footage in a more storage friendly format that was still "perceptually" lossless. I can save about 3months worth of these images on my external drive.

Finally any feedback on the package would be great. This was just released and any input would be valuable.

[davidplowman]

Thanks for that, I'll make those changes. Yes, it all runs perfectly for me on Trixie - which I think was released in October last year!

I'll merge the example onto our "next" branch, and it'll appear on our "main" branch when we next do a release.

Finally, can I perhaps ask what mu-files is - a company, organisation or individual? Just curious!

Thanks.