Decrease redundant/irrelevant lines in regtest reports

[stscijgbot-jp]

Issue JP-4007 was created on JIRA by Ned Molter:

The regression test diffs are mostly full of unnecessary information, which makes it hard to find what we are really interested in.  For one example: [^AssertionError.pdf]where green indicates the things I actually want to see.  It amounts to ~5 lines out of 3 pages of things I'd consider mostly extraneous to figuring out what actually went wrong.

[stscijirahub]

This issue is tracked on JIRA as JP-4007.

[stscijirahub]

Comment by Ned Molter on JIRA:

I was asked to clarify the intent of this ticket. Copying from Slack

 
99% of the time we are not interested in: * which HDUs and header keywords were not compared, as this never changes

• that the thing asserting False is a fitsdiff object
• the fitsdiff default kwargs, as those basically never change and mostly repeat which hdus and header keywords not to change
• a print of the entire test. For regtests the reason for the failure is seldom a problem with the test setup, and the test can easily be found by searching for the name

Some of the above is repeated multiple times.  The things we are interested in are also repeated multiple times: * the filenames being compared

• the fact that the headers contain a different number of entries
• the actual diff itself

I don't know what's best here and you'll have to play with multiple options.  I agree there are potential cases where more information is useful, so it might be nice to have some flag to turn on additional reporting or something.

There were other related issues at the time, but I hope they will mostly be cleaned up by stfitsdiff.  One that comes to mind is that, when adding or deleting an extension, instead of noticing the extension missing the code will attempt to compare with the same extension {}index{}.  i.e. if you remove AREA from {}a{}, then it'll attempt to compare VAR_RNOISE from a with AREA from b

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Opened spacetelescope/RegressionTests#323 for initial discussions.

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Updated scope:

1. See if we can vendor test-summary-action instead of having it living in a branch of Brett Graham's fork: spacetelescope/RegressionTests#324 . If not, then consider moving the code from Brett's fork to spacetelescope proper.
2. Once we know where test-summary-action ends up and moved it, we can then update it to not have duplicate output with the "E traceback details" lines. This could be either changing how junit spits out the XML or how the action ingest the XML.
3. Unrelated to test-summary-action but we have to wait till stfitsdiff code stabilizes and in production before we can update the diff output to
   a. not show a list of ignored header keywords
   b. shorter path to filenames being compared

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Example log from stfitsdiff as of 2026-03-17: https://github.com/spacetelescope/RegressionTests/actions/runs/23146532890/job/67236128582

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Update:

• Vendoring option (https://github.com/spacetelescope/RegressionTests/pull/324) was abandoned as it did not quite work and The Internet is too confusing on this topic, especially given the reusable workflow workflow has extra complexity.
• Brett G is unwilling to make further changes to his test-summary-action fork nor does he think it should go into spacetelescope. So if we must change how test-summary-action fork, he recommends that we get rid of test-summary-action altogether in RegressionTest workflows and instead reimplement what we want from scratch in Python; unclear how that is going to tie into test.yml at this point. Should we pursue this?

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Or maybe look into fine-tuning the junit stuff for JWST first in pyproject.toml ? https://docs.pytest.org/en/stable/reference/reference.html#confval-junit_family

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Part of the scope is blocked by https://jira.stsci.edu/browse/JP-4260

• Enable stfitsdiff in regression tests #10311
• JP-4260 - Enable stfitsdiff in nightly regression tests #10300

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Another experiment: I forked Brett's fork against his wishes... spacetelescope/RegressionTests#325

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Brett G has a counter-proposal: spacetelescope/RegressionTests#326

[stscijirahub]

Comment by Pey-Lian Lim on JIRA:

Putting this on hold. Blocked by:

• spacetelescope/RegressionTests#326
• JP-4260 - Enable stfitsdiff in nightly regression tests #10300