JP-4035: Fit trace model and oversample IFU data

[melanieclarke]

Resolves JP-4035

This PR will introduce a new spec2 and spec3 pipeline step called adaptive_trace_model, which will model the spectral trace in a 2D spectral image with a set of 1D basis spline fits in each column along the dispersion axis. Optionally, the step will also use the trace model to oversample the input data by a specified factor (3 is recommended). Only IFU data is supported so far.

Cube build is updated here to accommodate the oversampled data. Oversampled data will have a new frame added at the front of the WCS pipeline, to transform from oversampled pixels to detector pixels. All remaining transforms should work transparently from there, so all that is needed in cube build is to look for the right frame in the input and make sure that detector and image indices are separated correctly.

I added the new step before pixel_replace, with the idea that the new trace model might be a useful alternative for pixel replacement. So far, I updated pixel_replace to check for the right input coordinate frame in case of oversampled input data, but otherwise am not using the new profile anywhere other than the oversampling step. From INS testing, it looks like having the new step before pixel_replace results in a better trace model anyway.

Trace modeling and oversampling are based on David Law's initial draft, here:
https://github.com/drlaw1558/jwst/tree/ifu_oversample

Requires spacetelescope/stdatamodels#632

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

❌ Patch coverage is 97.31438% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.95%. Comparing base (bbf1e5f) to head (80d2cbe).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/cube_build/ifu_cube.py	72.72%	9 Missing ⚠️
jwst/adaptive_trace_model/trace_model.py	98.94%	5 Missing ⚠️
jwst/pixel_replace/pixel_replace.py	50.00%	2 Missing ⚠️
jwst/pipeline/calwebb_spec3.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10066      +/-   ##
==========================================
+ Coverage   85.98%   86.95%   +0.97%     
==========================================
  Files         368      370       +2     
  Lines       38457    39041     +584     
==========================================
+ Hits        33066    33948     +882     
+ Misses       5391     5093     -298     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[melanieclarke]

Regtests here:
https://github.com/spacetelescope/RegressionTests/actions/runs/20794108070

Diffs are as expected for spec2 and spec3: the new cal step keyword appears and is set to SKIPPED.

There are a couple small numerical diffs in the new regtest (test_miri_mrs_spec3_oversample) because I generated the truth files locally on my Mac. These can be okified with the rest.