RCAL-1380: Populate SFD-estimated extinction in catalogs.

[mairanteodoro]

Resolves RCAL-1380

This PR adds support for populating a new dust_ebv extinction column in Roman source and multiband catalogs using SFD dust maps resolved through CRDS. It updates the catalog schema and source-catalog generation logic, adds tests and regression checks for the new field.

Note

This adjusts CI/tox setup to use temporary git-based dependency installs needed for the change. We need to revert it before merging this PR. Done (unit tests will fail until the RAD PR is merged).

Regression tests

• https://github.com/spacetelescope/RegressionTests/actions/runs/25570563604 (05/08/26; all tests passing except for one that needs to have the truth file updated later).

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. 24Q4_B15 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
    • if your change breaks existing functionality, also add a changes/<PR#>.breaking.rst news fragment
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

❌ Patch coverage is 98.11321% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 77.30%. Comparing base (947f9d1) to head (2b07dbc).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
romancal/source_catalog/_source_catalog.py	98.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2313      +/-   ##
==========================================
- Coverage   80.55%   77.30%   -3.26%     
==========================================
  Files         155      155              
  Lines        9398     9451      +53     
==========================================
- Hits         7571     7306     -265     
- Misses       1827     2145     +318     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[schlafly]

Sorry, I worry I made the dependencies more complicated. Have we changed the procedure there? My memory is that normally we update pyproject.toml to point to your specific rad branch that has the needed updates. I don't understand the ci-gitdeps stuff. Alternatively run regtests overriding the rad dependency to point to your branch and point to the successful regtest run.

[mairanteodoro]

Sorry, I worry I made the dependencies more complicated. Have we changed the procedure there? My memory is that normally we update pyproject.toml to point to your specific rad branch that has the needed updates.

That's usually the case for direct dependencies. However, RAD is not a direct dependency of romancal and editing pyproject.toml won't honor the required version for RAD set in that file (it will be installed via RDM).

@zacharyburnett and I had a quick chat about this issue and he will draft a PR to try and solve this issue of RAD and RDM depending on each other while being two separate packages. For now, patching tox.ini/tests.yml is the temporary solution that I was able to come up with. At least it shows that the guard won't be needed when the RAD changes are in.

Alternatively run regtests overriding the rad dependency to point to your branch and point to the successful regtest run.

The regtests are fine because we can provide additional parameters that will override the requirements:
https://github.com/spacetelescope/RegressionTests/actions/runs/25570563604

In that regtest run, the only failure is for the test_multiband_catalog: romancal.regtest.test_multiband_catalog, and that is because we need to update the truth file later.

[schlafly]

Looks good to me. One minor comment inline. Thanks Mairan!

[schlafly]

The oldestdeps python 3.11 tests were failing with NoneType in various source catalog steps. We probably need a rad version requirement to do this; we're saying that's okay here since it should pass in the regtests with the pyproject / roman_datamodels pointing to the right rad version.

[zacharyburnett]

@zacharyburnett and I had a quick chat about this issue and he will draft a PR to try and solve this issue of RAD and RDM depending on each other while being two separate packages. For now, patching tox.ini/tests.yml is the temporary solution that I was able to come up with. At least it shows that the guard won't be needed when the RAD changes are in.

I drafted spacetelescope/roman_datamodels#676 to see what making a monorepo of rad and roman_datamodels would look like, but from talking with @braingram and @WilliamJamieson I don't think that's a viable solution either unfortunately. For now we'll just have to work with this dependency chain as-is

[schlafly]

Thanks, approved. I'll let you back out the pyproject toml changes and merge when the corresponding rad PR is in. We'll also need to okify the differences; I'll let you handle that when you merge?

[mairanteodoro]

We'll also need to okify the differences; I'll let you handle that when you merge?

Sounds good.

[braingram]

@zacharyburnett and I had a quick chat about this issue and he will draft a PR to try and solve this issue of RAD and RDM depending on each other while being two separate packages. For now, patching tox.ini/tests.yml is the temporary solution that I was able to come up with. At least it shows that the guard won't be needed when the RAD changes are in.

I drafted spacetelescope/roman_datamodels#676 to see what making a monorepo of rad and roman_datamodels would look like, but from talking with @braingram and @WilliamJamieson I don't think that's a viable solution either unfortunately. For now we'll just have to work with this dependency chain as-is

The unit tests are failing with a TypeError because:

romancal/romancal/source_catalog/_source_catalog.py

Line 652 in 1937ff3

catalog[column].info.description = definition["description"]

is attempting to get the "description" where definition is None. Merging spacetelescope/rad#870 won't fix this as:

• romancal is requiring the released rdm version:

  romancal/pyproject.toml

  Line 28 in 1937ff3

  "roman_datamodels>=0.31.0, <0.32",

• the development version of rdm is using the released version of rad: https://github.com/spacetelescope/roman_datamodels/blob/210ae0eb8d44d7e16647c30176eb29e4f68c7d01/pyproject.toml#L19

We will need:

• this PR to be updated to point to rdm main
• a rdm PR setting the required rad version to rad main

@mairanteodoro would you update the rdm requirement in this PR? I'll open a rdm PR setting the rad requirement to main.

[mairanteodoro]

Merging spacetelescope/rad#870 won't fix this as:

• romancal is requiring the released rdm version:

  romancal/pyproject.toml

  Line 28 in 1937ff3

  "roman_datamodels>=0.31.0, <0.32",

• the development version of rdm is using the released version of rad: https://github.com/spacetelescope/roman_datamodels/blob/210ae0eb8d44d7e16647c30176eb29e4f68c7d01/pyproject.toml#L19

We will need:

• this PR to be updated to point to rdm main
• a rdm PR setting the required rad version to rad main

@mairanteodoro would you update the rdm requirement in this PR? I'll open a rdm PR setting the rad requirement to main.

Yep, I can do that. I assumed that we would have a released version soon, so I didn't think of changing the requirements.