Qc pipeline

[BradleySappington]

Create and implement rough ideas for quality control checks for the dark pipeline.

These checks are currently algorithmically accurate, however numerically undecided.

[BradleySappington]

@rcosenti-stsci @rgcosentino - dark_quality_control.py is the file that I'd like you to look at to get those QC checks dialed in. We can also merge this in now if it looks good and go back and re-evaluate how we want that QC check to work after its implemented.

[rgcosentino]

Just a few notes. The change for Javier need to check this has been confirmed. It should be saturation step and then refpix. Everything else looks good.

[rgcosentino]

` def update_data_quality_array(self, hot_pixel_rate=0.015, warm_pixel_rate=0.010, dead_pixel_rate=0.0001):
# TODO evaluate options for variables like this and sigma clipping with a parameter file?
"""
The hot and warm pixel thresholds are applied to the dark_rate_image and the pixels are identified with their respective
DQ bit flag.

    Parameters
    ----------
    dead_pixel_rate: float; default = 0.0001 DN/s or ADU/s
        The dead pixel rate is the number of DN/s determined from detector characterization to be the level at
        which no detectable signal from dark current would be found in a very long exposure.
    hot_pixel_rate: float; default = 0.015 DN/s or ADU/s
        The hot pixel rate is the number of DN/s determined from detector characterization to be 10-sigma above
        the nominal expectation of dark current.
    warm_pixel_rate: float; default = 0.010 e/s
        The warm pixel rate is the number of DN/s determined from detector characterization to be 8-sigma above
        the nominal expectation of dark current.
    """

    self.hot_pixel_rate = hot_pixel_rate
    self.warm_pixel_rate = warm_pixel_rate
    self.dead_pixel_rate = dead_pixel_rate

    logging.info("Flagging dead, hot, and warm pixels and updating DQ array.")
    # Locate hot and warm pixel num_i_pixels, num_j_pixels positions in 2D array
    self.mask[self.data_cube.rate_image >= self.hot_pixel_rate] += self.dqflag_defs["HOT"]
    self.mask[(self.data_cube.rate_image >= self.warm_pixel_rate)
              & (self.data_cube.rate_image < self.hot_pixel_rate)] += self.dqflag_defs["WARM"]
    self.mask[self.data_cube.rate_image <= self.dead_pixel_rate] += self.dqflag_defs["DEAD"]

`
I would look at dark.py and how we set the flags for identifying hot, warm and dead pixels. So here's a thought now, do we just look at the flagged pixels in the dq array of the object? I think yes. We wouldn't want to go back into the data and find them again right?

I also think for the config file start off with something crazy like 5 or 10% of the total number of pixels in array

1% is 168000, 5% is 778000 and 10% is 1.7e6 or 1,680,000 number of pixels. For right now pick one "value" to be in the config - do 5% maybe and leave a comment that 1% is 168,000

[BradleySappington]

I completely forgot we had this set up!

[rgcosentino]

The hot, warm and dead dark rate thresholds for each detector should be in their respective QC config files. Get that value from there. Use the DQ flags for hot, warm and dead to count those pixels in separate checks and compare.

[rgcosentino]

in dark.py and the update_data_quality array method, I would like to have globals or these config QC parameters for hot, warm, and dead pixel rate be defined in the config file, and that config value overrides the default value or is provided as input to the method. Store it here or somehow and the number of hot, warm, and dead pixels.

So maybe
"num_hot_pix_check": type:boolean - this is the true false to do the QC check on
"max_num_hot_pix_value": type:int - this is the integer value for 5% of pixels 839,000
"hot_pix_rate": type:float - and this is the float value of hot pixels 0.1

Do you see how we need these or how each is used differently