Source catalog step reference file usage & prefetching

[schlafly]

Eric Younkin noticed that reference files needed only by the source catalog step were not being prefetched by CRDS. I suspect that the prefetching mechanism is triggered by lines like these:
https://github.com/spacetelescope/romancal/blob/main/romancal/saturation/saturation_step.py#L33
which the source catalog step does not use. The source catalog step uses the aperture reference file, the dust map reference file, and the PSF reference file. We should try to add those to the step analogously to their use in the saturation step.

@stscieisenhamer , does that look like the right mechanism to you?

[schlafly]

@stscijgbot-rstdms

[stscijgbot-rstdms]

This issue is tracked on JIRA as RCAL-1401.

[stscijgbot-rstdms]

Comment by Jonathan Eisenhamer on JIRA:

Comments from the slack converstation:

I just did a quick look at the code and I see the following:

The actual use of the reference is fine. I believe there is a roman datamodel-specific getreferences equivalent, but since we’re not dealing with a datamodel, there is no need for that here.
However, in the source_catalog_step itself, the reference type is not in the prefetch. But I think that may not actually work since the required information, in this case FIELD , is not in any dataheader. But that is OK. See below.

The prefetch that pipelines do on references is only to ensure that the needed references are present before potentially long, compute-intensive operations are done before discovering after much time/resource is spent that a reference file is missing.

So, basically to have the prefetch of the dustmap references occur, the FIELD keyword will need be specified. This may not be possible through the standard mechanism. However, it can be done if there is an explicit call to getreferences at the beginning of the step, doing the same conceptual work as the prefetch.

[mairanteodoro]

The call for the dust maps happens in a lazy way, only when RomanSourceCatalog.dust_ebv is evaluated.

Specifically, the FITS dust maps are not read during SourceCatalogStep.process() setup. They are opened lazily only if the dust_ebv column is actually needed while building the catalog:

1. SourceCatalogStep.process() creates RomanSourceCatalog(...).
2. Later, catobj.catalog triggers catalog assembly.
3. During _run_measurements() / _assemble_catalog(), any requested column is accessed.
4. When dust_ebv is accessed, the @lazyproperty runs:
   • resolves the CRDS dust map paths via _get_sfd_map_paths()
   • then calls _get_dust_ebv(...)
5. Inside _get_dust_ebv(...), the maps are actually read here:

romancal/romancal/source_catalog/_source_catalog.py

Lines 570 to 575 in 11c1271

	with fits.open(map_paths[pole]) as hdulist:
	imwcs = WCS(hdulist[0].header)
	x, y = imwcs.wcs_world2pix(l[pole_mask], b[pole_mask], 0)
	out[pole_mask] = map_coordinates(
	hdulist[0].data, [y, x], order=order, mode="nearest"
	).astype(np.float32)

So the short answer is: the dust maps are read only on demand, when dust_ebv is computed, not earlier.

The relevant entry point is here:

romancal/romancal/source_catalog/_source_catalog.py

Lines 437 to 464 in 11c1271

	def dust_ebv(self):
	"""
	Return per-source dust E(B-V) from SFD maps referenced by CRDS.
	This property queries CRDS for north/south SFD map references,
	evaluates E(B-V) at each source `ra`/`dec` position, and returns
	one value per source.
	Returns
	-------
	result : `~numpy.ndarray`
	Array of dtype ``float32`` and shape ``(n_sources,)``.
	If any step fails (CRDS lookup, file I/O, WCS transform, or
	interpolation), returns all-NaN values with the same shape.
	Warns
	-----
	Emits a warning before returning all-NaN values on failure.
	"""
	try:
	dust_map_paths = self._get_sfd_map_paths()
	return self._get_dust_ebv(self.ra, self.dec, dust_map_paths)
	except Exception as exc:
	log.warning(
	f"Failed to compute dust_ebv from SFD reference maps: {exc}; "
	"setting dust_ebv to NaN."
	)
	return np.full(self.n_sources, np.nan, dtype=np.float32)

And the catalog is triggered here, near the end of the step:

romancal/romancal/source_catalog/source_catalog_step.py

Line 225 in 11c1271

cat = catobj.catalog

So, in order to follow @stscieisenhamer's suggestion, we'd need to extract the call for the dust maps out of the lazy property and place it before the source catalog step starts any actual work.

[schlafly]

I think Jonathan's point is more that the mechanism used here:
https://github.com/spacetelescope/romancal/blob/main/romancal/source_catalog/source_catalog_step.py#L79
won't work because the north/south selector isn't in the image metadata (and we want both anyway). Let's at least try adding the PSF reference file to that list so that it can be prefetched.

[braingram]

Could we override _get_crds_parameters for the step or perhaps prefetch?