JP-4285: Cube Build updates by jemorrison · Pull Request #10433 · spacetelescope/jwst

jemorrison · 2026-04-07T18:00:20Z

See comment below:needs update to calspec2 parameter reference file.

This PR addresses how output filenames are created in cube_build. The 'pipeline' parameter was removed from cube along with the option to create 'single' output_type cubes. The 'single' type cubes had been used early in the mission in outlier detection to try and flag bad pixels. In the mode outlier detection called 'cube_build' to create a set of cube for each band. Each input file was mapped to the single band, all the single band s3d files where median combined and blotted back to the detector frame where bad pixels were flagged. The method of outlier detection for IFU data was replaced did not flag bad pixels well and was replaced soon after launch. The hooks to make this types of 'single' ifu cubes was left in cube_build. This pr removed the unnecessary code.
For calspec2 the internal 'step' routines create the default IFU cube names.
For calspec3 cube - cube build uses the asn name and information on the bands to create a string for the output name.

REMAINING ISSUE update to Parameter reference file.
In cube_build the 'output_type' has now been defaulted to 'band'. Only when running calspec2 on MIRI data do
we want the 'output_type' parameter = 'multi'
To run the calspec2 pipeline on miri data - the parameter reference file needs to be updated to
set 'output_type' = multi.
We could hard code it in the calwebb_spec2 code - but it is probably better to have a parameter reference file.
For miri the current calwebb_spec3 parameter reference file is set to
class: jwst.cube_build.cube_build_step.CubeBuildStep
name: cube_build
parameters: {output_type: band, skip: false}

If NIRSPec wants cubes other than 'band' from calspec3 then they should also create a parameter reference file
for calspec3.

Tasks

codecov · 2026-04-07T18:34:36Z

Codecov Report

❌ Patch coverage is 82.75862% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.61%. Comparing base (f86541f) to head (0af9fe3).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/cube_build/cube_build_step.py	60.00%	4 Missing ⚠️
jwst/cube_build/data_types.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10433      +/-   ##
==========================================
+ Coverage   86.40%   86.61%   +0.20%     
==========================================
  Files         373      373              
  Lines       40112    39927     -185     
==========================================
- Hits        34658    34582      -76     
+ Misses       5454     5345     -109

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jemorrison · 2026-04-20T17:39:03Z

I running the regression test to see if I need to fix any tests.

jemorrison · 2026-04-20T17:48:34Z

@drlaw1558 One other thing while we are cleaning up the interface for cube build. Currently the parameter 'linear_wave' only works for NIRSpec data. This allows the user to switch between linear and non-linear wavelengths dimensions. The default value is True. For MIRI data, cube_build looks at the data that is to make the cube and decides if it should be linear or non-linear. In general, ifu cubes of a single band are linear and multi channel or multi band cubes can be non-linear. Cubes from spec2 are non-linear and cubes from spec3 are linear. But the user can run cube_build offline and create non-linear cube from spec3 associations by selecting the bands or channels for the cube.
Do you want to keep cube build determining if the ifu cubes have a linear or non-linear wavelength or move control to the 'linear_wave' parameter ?

melanieclarke

This is a very nice simplification of the code!

Can you please add some unit tests for the expected filenames in various cases?

Also, I think we will need a breaking change note, since this removes parameters from the spec.

…etection method

jemorrison · 2026-05-04T14:32:38Z

Currently the calwebb_spec2.py file sets output_type = 'multi' for MIRI MRS data and for NIRSpec it is set to 'band'
Cube_build_step defaults output_type to band - so these changes should not cause any problems.

jemorrison · 2026-05-04T15:47:54Z

@melanieclarke I wrote a unit test for testing suffix names but it uses from unittest.mock import MagicMock
I don't think other unit tests use MagicMock so if you don't want it done this way let me know.
I needed a way to call ifu_cube and not have a ton of set up just to test the filename suffix.

jemorrison · 2026-05-04T16:46:51Z

@melanieclarke Changed the test to use monkey patch.

jemorrison · 2026-05-04T16:47:43Z

@melanieclarke changed the test to use monkey patch.
I do not have much experience with monkey patch so let me know if that test is ok or you want something else.

melanieclarke

Tests look good, thanks! A few other minor clean up items, but we're getting close.

melanieclarke · 2026-05-05T17:53:17Z

 ``pipeline [integer]``
  ``cube_build`` can be called from :ref:`calwebb_spec2 <calwebb_spec2>`, :ref:`calwebb_spec3 <calwebb_spec3>`, or stand-alone. The output IFU cubes
  have a different name depending which pipeline calls it. This parameter defaults to 3 which follows the rules
  for creating cubes based on the :ref:`calwebb_spec3 <calwebb_spec3>` pipeline and is also the behavior when running
  ``cube_build`` stand-alone. When the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline calls
  ``cube_build``, it sets ``pipeline=2``. When running stand-alone or in
  :ref:`calwebb_spec3 <calwebb_spec3>` pipeline, the parameter ``pipeline=3`` is set for ``cube_build``.


Looks like this section still needs to be removed.

There are also some comments below about the "pipeline rules" for output_type that I think need to be updated.

melanieclarke · 2026-05-05T17:57:33Z

+"""
+Unit test for Cube Build testing testing determining the correct suffix for the filename.
+"""


Suggested change

"""

Unit test for Cube Build testing testing determining the correct suffix for the filename.

"""

"""Test output filename suffixes for the cube_build step."""

melanieclarke · 2026-05-05T17:59:08Z

+
+
+def test_define_cubename_suffix_miri_1band(cube_instance):
+    """Test MIRI suffix logic by cube_instance attributes."""


Suggested change

"""Test MIRI suffix logic by cube_instance attributes."""

"""Test MIRI suffix logic for one band and channel."""

melanieclarke · 2026-05-05T17:59:25Z

+
+
+def test_define_cubename_suffix_miri(cube_instance):
+    """Test MIRI suffix logic for multiple bands and channels"""


Suggested change

"""Test MIRI suffix logic for multiple bands and channels"""

"""Test MIRI suffix logic for multiple bands and channels."""

melanieclarke · 2026-05-05T17:59:40Z

+
+    suffix = cube_instance.define_cubename_suffix()
+
+    # Assert (MIRI logic sorts subchannels according to increasing wavelength)


Suggested change

# Assert (MIRI logic sorts subchannels according to increasing wavelength)

# MIRI logic sorts subchannels according to increasing wavelength

melanieclarke · 2026-05-05T17:59:51Z

+
+
+def test_define_cubename_suffix_nirspec(cube_instance):
+    """Test NIRSpec logic fir 1 grating and 1 band ."""


Suggested change

"""Test NIRSpec logic fir 1 grating and 1 band ."""

"""Test NIRSpec logic for 1 grating and 1 band ."""

melanieclarke · 2026-05-05T18:03:06Z

Checking coverage locally, it looks like there are a few more cases we could add here:

NRS with a duplicate grating name to remove

NRS with multiple gratings

NRS internal_cal

melanieclarke · 2026-05-05T18:10:21Z

+    def define_cubename_suffix(self):
        """
-        Define the base output name.
+        Determine the filename suffix name consisting of channels/sub channels or gratings/filters.


Suggested change

Determine the filename suffix name consisting of channels/sub channels or gratings/filters.

Determine the filename suffix consisting of channels/sub channels or gratings/filters.

melanieclarke · 2026-05-05T18:26:58Z

Can you please also add a breaking change note, since we're removing step parameters?

jemorrison requested a review from a team April 7, 2026 18:00

github-actions Bot added the cube_build label Apr 7, 2026

jemorrison marked this pull request as draft April 7, 2026 18:00

github-actions Bot added pipeline MIRI testing spec2 pipeline spec3 pipeline regression_testing labels Apr 20, 2026

jemorrison force-pushed the cb_check branch from 7614241 to 7f60b12 Compare April 20, 2026 17:38

jemorrison force-pushed the cb_check branch from 7f60b12 to 5faa539 Compare April 21, 2026 15:56

github-actions Bot added NIRSPEC documentation labels Apr 21, 2026

jemorrison added this to the Build 13.0 milestone Apr 22, 2026

jemorrison marked this pull request as ready for review April 22, 2026 21:43

jemorrison requested review from emolter, melanieclarke, penaguerrero, pllim and tapastro as code owners April 22, 2026 21:43

jemorrison requested a review from drlaw1558 April 22, 2026 21:47

jemorrison force-pushed the cb_check branch from ed38e63 to 0351af8 Compare April 22, 2026 21:48

melanieclarke reviewed Apr 23, 2026

View reviewed changes

jemorrison force-pushed the cb_check branch from 0351af8 to a1a427b Compare April 23, 2026 16:51

jemorrison added 3 commits May 4, 2026 07:14

added print

b7b95cb

testing

a0166a9

Loaded for comments

45dc715

jemorrison added 7 commits May 4, 2026 07:14

Loaded for comments

32530bd

clean up interface and remove single type cubes used in old outlier d…

825a27e

…etection method

add change log

56967ee

cleaned up tests

7144ced

fix syntax error in test and update docs

464b501

update calspec2 tests in regression tests

4145b0c

updates from review

9e5af96

jemorrison force-pushed the cb_check branch from a1a427b to 9e5af96 Compare May 4, 2026 14:27

add unit test for suffix

92bee69

change test to monkey patch

36df3c3

emolter reviewed May 4, 2026

View reviewed changes

Comment thread jwst/cube_build/tests/test_cube_suffix.py Outdated

update unit test

0af9fe3

melanieclarke reviewed May 5, 2026

View reviewed changes

tapastro changed the title ~~Cube Build updates~~ JP-4285: Cube Build updates May 6, 2026



		def test_define_cubename_suffix_miri_1band(cube_instance):
		"""Test MIRI suffix logic by cube_instance attributes."""

	"""Test MIRI suffix logic by cube_instance attributes."""
	"""Test MIRI suffix logic for one band and channel."""



		def test_define_cubename_suffix_miri(cube_instance):
		"""Test MIRI suffix logic for multiple bands and channels"""


		suffix = cube_instance.define_cubename_suffix()

		# Assert (MIRI logic sorts subchannels according to increasing wavelength)

	# Assert (MIRI logic sorts subchannels according to increasing wavelength)
	# MIRI logic sorts subchannels according to increasing wavelength



		def test_define_cubename_suffix_nirspec(cube_instance):
		"""Test NIRSpec logic fir 1 grating and 1 band ."""

	"""Test NIRSpec logic fir 1 grating and 1 band ."""
	"""Test NIRSpec logic for 1 grating and 1 band ."""

	Determine the filename suffix name consisting of channels/sub channels or gratings/filters.
	Determine the filename suffix consisting of channels/sub channels or gratings/filters.

Conversation

jemorrison commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

codecov Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jemorrison commented Apr 20, 2026

Uh oh!

jemorrison commented Apr 20, 2026

Uh oh!

melanieclarke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jemorrison commented May 4, 2026

Uh oh!

jemorrison commented May 4, 2026

Uh oh!

jemorrison commented May 4, 2026

Uh oh!

jemorrison commented May 4, 2026

Uh oh!

Uh oh!

melanieclarke left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jemorrison commented Apr 7, 2026 •

edited

Loading

codecov Bot commented Apr 7, 2026 •

edited

Loading