Support for Debayering RAW16?

[AlaskanAstro]

In my quest for greater dynamic range in what really are nice camera sensors these days, I'm wondering if there is possibility of support for debayering the RAW16 data? 8 bit color can be a bit limiting with large, bright gradients especially with long night exposures. I have no idea what kind of compute power that asks of a Pi.

My understanding is the currently the RGB24 is a debayered RAW8.

[EricClaeys]

The ZWO library only supports 16-bit mono. The only choice for color is 8-bit per color.

[AlaskanAstro]

Gotcha. Would an external debayer step need to be a module of some kind?

[EricClaeys]

Unless I don't understand the ZWO library, the camera can't produce a 16-bit per channel color image on Linux, so you're out of luck. I've always thought this was strange since most newer sensors do 10, 12, or 16 bit color, yet all we can do on Linux is 8 bit.
This would be a good question on a ZWO forum.

[AlaskanAstro]

So in my brief research it looks like ZWO does not have RGB48 in their SDK. And outside comments are that their debayer algorithm isn't that great anyway so debayering the RAW16 into color with a separate process yields a better result anyway.

This ZWO forum post is where I got my answer: https://bbs.zwoastro.com/d/13050-reading-out-16-bit-rgb-with-sdk/8

[EricClaeys]

Everything I see that ZWO has says that the size of a RAW16 image is:
width * height * 2
which is a mono 16-bit image.

A 16-bit color image would be
width * height * 2 * 3
so the forum post is confusing to me.

[aaronwmorris]

The RAW16 image for color cameras are still 2 dimensional images, it just has the raw uint16 values from the sensor for each pixel. You have to run the image through a debayering algorithm, supplying the correct bayer pattern, the resulting image will be 3 dimensional, albeit, still with uint16 values. OpenCV has the method you needed to debayer the images. The results still need to be downsampled to 8 bits if you want to save as JPEG.

https://docs.opencv.org/4.9.0/de/d25/imgproc_color_conversions.html

[AlaskanAstro]

My understanding is the ideal process is:
Capture RAW16
Debayer
Stretch to target brightness with black clipping and midpoint adjust
Save jpeg in 8 bit

[EricClaeys]

@aaronwmorris, I don't understand how a color camera can output all 3 colors with 16-bit fidelity using a single 16-bit integer rather than 3 16-bit integers (for a total of 48 bits / pixel). According to the document you posted, with RAW16 each color gets only 5 or 6 bits, which is worse than RGB24 where they get 8 bits each. All ZWO's documentation including their ASICamera2.h file mentions RAW16 bit being for mono.

Are you able to produce 16-bit per color images in Indi Allsky in Linux? If so, how?

[aaronwmorris]

Yes, I can produce a 16-bit color image with indi-allsky from a 2 dimensional RAW source (FITS data). Most image data from INDI is 16-bit RAW data by default. Although, it is very rare for someone to be able to use a 16-bit color image because support for 16-bit data is not consistent with programs. The most common way to use the 16-bit data is for processing it before it is down-sampled to 8-bits.

You do have to use the correct debayering pattern, there are 4 commonly used: RGGB, GRGB, BGGR, and GBRG. The ZWO SDK will be able to provide the bayer pattern for the sensor for each camera.

The RAW16 data for RGGB looks like this:

RGRGRG...
GBGBGB...
RGRGRG...
GBGBGB...
...

The debayering algorithm is able to take the these RAW values for the individual color pixels and produce a synthetic RGB (BGR for OpenCV) output. When you output RGB24, the camera is doing all of this internally, but it is following the same exact process. The debayering algorithm is the reason RGB data contains chrominance noise (color speckles) because the process for converting the data is never perfect.

This is some rough Python code. It should be fairly easy to convert this to C.

import cv2
import numpy as np

cfa_bgr_map = {
    'RGGB' : cv2.COLOR_BAYER_BG2BGR,
    'GRBG' : cv2.COLOR_BAYER_GB2BGR,
    'BGGR' : cv2.COLOR_BAYER_RG2BGR,
    'GBRG' : cv2.COLOR_BAYER_GR2BGR,
}

# debayer RGGB pattern for ASI224
image_bgr_16bit = cv2.cvtColor(raw16_data, cfa_bgr_map['RGGB'])

# 16-bit PNG
cv2.imwrite('image16.png', image_bgr_16bit, [cv2.IMWRITE_PNG_COMPRESSION, 9])

# downsample to 8 bits (ZWO cameras return full 16-bit data)
image_bgr_8bit = np.right_shift(image_bgr_16bit, 8).astype(np.uint8)

# 8-bit jpeg
cv2.imwrite('image8.jpg', image_bgr_8bit, [cv2.IMWRITE_JPG_QUALITY, 90])

[EricClaeys]

@aaronwmorris,
Am I understanding correctly that indi-allsky can produce a 16-bit per color (i.e., 48 bits total) image from a 16-bit image? This means red, green, and blue all have up to 65K different shades. If so, where are the other 24 bits coming from?

Perhaps we're using different terminology. To me, a 16-bit image means each pixel's value is encoded in 16 bits so the size of the image is 16-bits X height X width. Typically, that's a mono image, but could be a color image with each color having 5 or 6 bits of data (which would most likely have severe banding).

[aaronwmorris]

@EricClaeys You are right that there is no "free lunch" here, but the bits from the 16-bit mono bayered image are not divided evenly between the colors. Debayering is a lossy algorithm, though.

Debayering basically takes the individual [monochrome] pixels, and interpolates an R, G, and B value based on the neighboring [different] color pixel values. Data is effectively borrowed from the neighboring pixels to produce the 3 values for one camera pixel. The colors are approximations. This is why using LRGB filters in astrophotography is superior for producing color images--you get a real R, G, and B values for each pixel, instead of an interpolated value.

All modern camera sensors produce monochrome data, even color cameras. The RGB24 data is calculated from this monochrome data. When you request RGB24 from the driver, the camera is just performing the same debayering calculations before returning the data.

There are other realities to deal with. Most all sky cameras are going to be 10, 12, or 14 bits. 10 bits has a max value of 1023, 12 is 4095, and 14 is 16383. The ZWO driver always upscales this data so that the max value is 65535. You already have banding of data straight out of the camera, although this does not really matter once dithered to 8-bits.

Unfortunately (for indi-allsky) not all drivers scale the data to the full 16-bit space. This can even be a problem with libcamera, although I think the newest versions have started upscaling the data for the DNG files.

[AlaskanAstro]

Just adding onto this two things:

The original question I posted hinges on taking advantage of the modern sensor's 12 and 14 bit ADCs.

While lossy, modern debayer algorithms are remarkably good at "filling in the gaps" especially for broadband data, which is what all sky data is.

[aaronwmorris]

@AlaskanAstro My guess is this type of change would require significant effort to handle RAW data.

I will relate this to my experiences handing RAW data in indi-allsky. A lot of image processing utilities are built on the assumption you are dealing with unsigned 8-bit integers and fail when presented with 16-bit data. Many of of the modules in allsky are designed around this same assumption. Even OpenCV is inconsistent in handling different bit depths. Some functions handle uint16, some require normalized data (32-bit floats between 0 and 1). The entire allsky module pipeline would have to be scrutinized to ensure it can handle uint16 data. This would very likely break a lot of 3rd party modules.

The indi-allsky image processing pipeline had to be carefully designed multi-stage process. The first few stages is where RAW data is handled (dark frame subtraction, stretching, stacking, etc) and the latter stages are for when the data has been scaled to 8-bits (star detection, line detection, cropping, flipping, scaling, etc).