Bad pixel map implementation

[manugr788]

Dear,

In order to carry out end-to-end simulations (L0->L1), we would be very interested in implementing a bad pixel map function in Platosim, if possible fairly quickly. PCOT has recently been working on this and has defined two lists of bad pixel status, a basic list with only two modalities and a more complex list with all possible cases (see Confluence PCOT page).

We would be very keen to see the binary bad pixel map (using binary status) implemented in Platosim in much the same way as the PRNU map, i.e. with a BadPixelMap dataset, BadPixelMapSeed and WriteBadPixelMap parameters included in the yaml file. In addition, a new statistical parameter could control the frequency of bad pixels (statistical law TBD).

As with the PRNU, the output bad pixel map must have the same size as the sub-field, but it is very important for us that the bad pixel map is generated for the entire CCD on the basis of the BadPixelMap Seed, and then extracted for the imagette position. This will allow us to run simulations of full CCD images and simulations of multiple imagettes with a bad pixel map consistent between the full CCD image and the imagettes using the same BadPixelMap Seed.

Don't hesitate to ask me questions if this isn't clear and you need more details.

[JorisDeRidder]

Hi Manu, OK, just to be sure you only need 0/1 flags to mark the bad pixels, correct? Not the different status flags that are mentioned in the Confluence page, correct?

[manugr788]

Hi Joris,
In fact, we need at least, as soon as possible, a first version of the bad pixel map implemented in Platosim with 0/1 flags to mark bad pixels. But this could be further improved with a parameter to choose between 0/1 maps or the PCOT bad pixel map you mentioned above (although I suppose the statistical laws for all these modalities are not yet well known), and even a third choice to load a bad pixel map from a file could be an interesting option too if it's easy to implement quickly (with an example of bad pixel map file in inputfiles directory or to download somewhere)

[manugr788]

Dear Dries,
do you know approximatively when the first version of this feature might be implemented ? That's way, we'll know whether we should start the next end2end simulation without this feature right away, or whether we should wait a little while and include the platosim bad pixel map in our next simulation.

[JorisDeRidder]

Hi Manu, Dries is currently on holidays until the end of the week, but he has started to work on it, and expects to have a first implementation next week. In a first implementation the bad pixels will be added at the last stage, i.e. after implementing all other effects. This means that we are not including at this moment the effect of bad pixels on the CTI, on the BFE, on FWS, etc.

[manugr788]

ok, perfect, we will wait for Dries's implementation of bad pixel map.
BTW are you aware of the bad pixel maps and PRNU maps (currently only F-CAM) produced by Philipp & Sascha ?
You can find the bad pixel maps on the KUL server under /DLR/PCOT/MPDB_Calibration_Parameters/Analogue_Chain/Fast BadPixelMap

[Dries-Seynaeve]

Hi Manu,

I pushed a first version to a separate branch called BadPixelMap. This follows a Bernoulli distribution to generate a binary bad pixel map.

This can be configured in CCD/BadPixelMap where you can set if you want to include the bad pixel map and the chance of having a bad pixel. (0.1% by default)

The seed can be set in RandomSeeds/BadPixelSeed, and you can specify if you want to save the bad pixel map in the output HDF5 file in ControlHDF5Content/WriteBadPixelMap. This saves both the bad pixel map for the entire CCD and for the subfield.

Let me know if these features work for you and if you want something else added. I wasn't sure if you want to be able to give a bad pixel map as input as well. If so, I can add this.

Kind regards
Dries

[manugr788]

Thanks a lot Dries, this will help us a lot to test our pipeline with the functionality of the bad pixel map (binary) .
In the future, it has been decided that the on-ground L0->L1 pipeline will use the more complex PCOT table mentioned above, i.e. actually a PixelQualityMap with a bit mask of 14 bits. I guess this will be more complex to implement as we don't know exactly the physics behind these processes, and it might be simpler to add a functionality to load a pixel quality map from a file covering all the CCD (4510*4510), and extract the relevant pixels for the subfield.

Just one point, in your current implementation of the bad pixel map, can you confirm that if we run 2 simulations, one for a full image and one for a 6*6 imagette with the same BadPixelSeed, we'll get the bad pixel(s) at the same place ?

[Dries-Seynaeve]

Hi Manu,

Yes, that should be the case. We always generate the bad pixel map for the entire CCD and then we select from this map the pixels that lie on the subfield. So with the same seed, you should always have the same bad pixel map for the entire CCD.

[manugr788]

Hi Dries,

I have just tested the new Bad Pixel feature implemented in the BadPixelMap branch. It looks OK except that it seems that a bad pixel is coded with a value of 0 while a good pixel is coded with a value of 1. I was expecting the opposite since it's a map of bad pixels and we would like to find bad pixels such as file["BadPixelMap/CCD"][:] == True.

With the default parameters (badPixelParameter: 0.001), I got:

>>> (file["BadPixelMap/CCD"][:] == True).sum()
np.int64(20319914)
>>> (file["BadPixelMap/CCD"][:] == False).sum()
np.int64(20186)

[manugr788]

Moreover:

• the data type of BadPixelMap/CCD and BadPixelMap/Subfield datasets is floating point while I was expecting int32 (to prepare further PixelQualityMap dataset)
• when I simulate a full image 45104510 and I get a BadPixelMap/CCD dataset of size 45104510, which is correct, but a BadPixelMap/Subfield dataset of size 100*100, why ?

[Dries-Seynaeve]

Hello Manu,

Yes, there was an error in how the subfield map was saved to the HDF5 file. I fixed this and also changed it so that bad pixels are not registered as 1.

Kind regards
Dries