Bad wavelengths in extract2d cutouts for wfss data

[stscijirahub]

Issue JP-4348 was created on JIRA by Ned Molter:

The wavelength backplane to the virtual slits extracted for WFSS data in the Extract2dStep is wrong, and sometimes wrong to the point of being highly unreasonable.  See the attached images.

The wavelength backplane is found by assuming that all direct-image flux is at the slit reference point, and then figuring out the wavelength value needed to disperse from the reference point into a given dispersed-plane pixel.  This is wrong for two related reasons:

Multiple direct-image pixels at multiple different wavelengths contribute to a given dispersed pixel.  This is true for point sources due to the PSF, and it’s even more true for extended sources.

For some bright sources with extended PSFs as well as some extended sources, there are dispersed-plane pixels where the real contribution from the reference point does not exist at all.

See the attached images.  There is an extended source, with a large footprint in the segmentation map. (The multiple sources should be deblended, see JP-4339, but the present issue will remain an issue.)  The file jw03362010001_15101_00006_slit0_wavelength_default.png shows the wavelength solution, with the colorbar being in microns.  The cutout is ~800 pixels long in the dispersion direction, much longer than a single-pixel dispersion would be, and the wavelengths accordingly extend far beyond the ~0.8-1.0 um that would be expected in this filter.  Some of the wavelengths even go negative.

Two possible solutions I see are:

• Force the wavelength backplane to be NaN outside the wavelengths defined in the WAVELENGTHRANGE file.  This would make a slit cutout like this one have mostly NaN wavelengths.  The 1-D spectral extraction would be constrained to within the expected range, but it would lose a lot of signal and fail to account for the fact that each dispersed pixel encodes multiple direct-image pixels/wavelengths.
• Use the wfss_contam step to compute the wavelengths expected to contribute in each dispersed pixel, and use those to override the wavelength backplane of the slit when wfss_contam is turned on.  This could either be a 3rd dimension of the image containing multiple wavelengths and their associated weights at each dispersed pixel (in which case the algorithm of extract1d needs to be reconsidered), or it could be an average wavelength in each dispersed pixel (easier to implement as extract1d need not change, would still be inaccurate but perhaps better than the current state).

There may be other solutions.  I'd love to hear thoughts from the WFSS experts.

[stscijirahub]

Comment by David Law on JIRA:

Tagging Russell Ryan Norbert Pirzkal Gael Noirot 

[stscijirahub]

Comment by Gael Noirot on JIRA:

My two cents:

• Better segmentation would help mitigate the issue a lot (it seems to be the main culprit right now).
• Avoiding to trim the wavelength array should be preferred, as extended sources will have real data outside of the final 1D wavelength array in the cal files (since based on the reference pixel only).
• Adding info in the data products about which wavelengths contribute to a given grism pixel would be useful. For each grism pixel, this could also include the corresponding direct image pixel IDs that contribute, the relative flux weights from the direct image pixels, and the relative grism response for the contributing wavelengths. Although all of that can be retrieved outside the box, so a helper function to do so might be more useful.
• How the 1D wavelength array is assigned in the cal files should be clearly described in some documentation.
• For most basic analysis with smallish sources (which can represent a good fraction of extragalactic sources people are interested in), assigning the centroid pixel wavelength to the 1D collapsed spectrum is fine (eg, getting a redshift, getting a line flux).
• Most users who want/need to do advanced/refined analysis will likely use forward-modeling techniques which won't be affected by this issue.

[stscijirahub]

Comment by Ned Molter on JIRA:

Thanks Gael those comments are helpful.  I'm confused though what you mean about "Adding info in the data products about which wavelengths contribute to a given grism pixel."  If I understand correctly, the only way to get this is through contam modeling based on the direct image, i.e., you need to run the wfss_contam step.  How do you envision that being retrieved by a helper function?

RE advanced/refined analysis, of course the pipeline is not likely able to support the most complex workflows, but wfss_contam is already a basic form of forward modeling.  I'd be curious to hear your thoughts on more advanced techniques.

Do any of the specific options I mentioned appeal to you, or something different?

[stscijirahub]

Comment by Gael Noirot on JIRA:

The helper can be a function that takes the cal file, and associated direct image, seg map, and catalog as input, and gives the wavelength contributions at each pixel in the cal file as output, which I think is possible since what needs to be known is only the dispersion. The contam step can already do that though, I agree.
The more advanced technique I was referring to is basically being able to use any template of interest for any given region/pixel in the direct image, and have a way to do some fitting of the data via this forward modeling, and sampling any parameter space of interest from the templates while doing so. The "multi-band"/pixel-based method being worked on for the contam step can in principle allow that.

For the two options written in the description, I would avoid doing the first (my second bullet point below). And for the second option, having the contributions at each pixel as additional info rather than replacing the current wavelength array might be preferred (my third and fifth bullet points below). Users can then choose how to use/weight those contributions if they need/want to.

[stscijirahub]

Comment by Ned Molter on JIRA:

Ok, thanks Gael.  So it sounds like the action items are:

improve segmentation, which will be handled by separate tickets JP-4164 and JP-4339

Document better how the wavelength array is assigned in extract2d

Use the wfss_contam step to additionally provide all the wavelengths and wavelength weights that contribute to a given dispersed-frame pixel.

Do you agree with this list?

For #⁠3, if the WFSS teams agree we should indeed go that route, there are some decisions to make.  What format will this information take in the file?  How, if at all, will the extra information be leveraged when running extract_1d?

Do you want to pursue the idea of a helper function further, or does just using the contam step for this seem fine?

[stscijirahub]

Comment by Gael Noirot on JIRA:

Seems good to me.
I don't have preferences for the format, and extract_1d can simply carry the info around, without doing anything special to it. I'm also fine if the contamination step handles generating the info rather than a helper function.

[stscijirahub]

Comment by Jo Taylor on JIRA:

Just wanted to chime in that on the NIRISS team, we are also considering modifying the wavelength limits in our wavelengthrange and photom files. This may help some for some sources, but I agree that the biggest issue is overly large segmentation maps.

[stscijirahub]

Comment by Ned Molter on JIRA:

Gael Noirot I'm still not sure we're on the same page here.  My suggestion above would be to make the 2-D extraction cutouts from extract2d contain, instead of a single-valued wavelength backplane, a list of wavelengths and associated weights per pixel.  Please let me know if that's unclear.

I'm not sure what you mean by "carry the info around" in extract1d - we need to do something with the information in order to turn it into a 1-d spectrum.  Even if we continue to use the old wavelength backplane for the spectral extraction, i.e., not modify the algorithm, we'd still need to find some way to reduce the wavelength-weight pairs (several per 2-d pixel) to be associated with the 1-d spectral data points (wavelength, flux).

[stscijirahub]

Comment by Gael Noirot on JIRA:

Ned Molter I mean that the current 1D/2D products can stay the same, simply with added extensions/entries that give the contributing wavelengths and associated weights/infos (eg, direct image pixel IDs, relative flux, relative response, ...) for each data point along the spectra. This can be given/carried around in a table format for instance. Users can then determine how best to use that info to modify the wavelength array if they need/want to.