Skip to content

JP-4323: Improve performance of tso median calculation in outlier detection#10452

Merged
melanieclarke merged 5 commits into
spacetelescope:mainfrom
braingram:tso_med
Apr 16, 2026
Merged

JP-4323: Improve performance of tso median calculation in outlier detection#10452
melanieclarke merged 5 commits into
spacetelescope:mainfrom
braingram:tso_med

Conversation

@braingram
Copy link
Copy Markdown
Collaborator

@braingram braingram commented Apr 14, 2026

Resolves JP-4323

On main jwst/regtest/test_miri_image_tso.py::test_log_tracked_resources_miri_image_tso (based on regtest runs):

  • takes 173 seconds
  • has peak memory usage of 31G

With this PR the test:

  • takes 51 seconds
  • has peak memory usage of 4G

Running the same test locally the runtime pattern agrees (95 on main, 33 with PR).

Regtests: https://github.com/spacetelescope/RegressionTests/actions/runs/24400023732
all pass. Here's a plot of the memory usage for the latest run:
Screenshot 2026-04-14 at 4 43 09 PM
We still have something leaking but at least the peak is reduced.

Tasks

  • If you have a specific reviewer in mind, tag them.
  • add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
  • Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
    • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
      • if your change breaks step-level or public API (as defined in the docs), also add a changes/<PR#>.breaking.rst news fragment
    • update or add relevant tests
    • update relevant docstrings and / or docs/ page
    • start a regression test and include a link to the running job (click here for instructions)
      • Do truth files need to be updated ("okified")?
        • after the reviewer has approved these changes, run okify_regtests to update the truth files
  • if a JIRA ticket exists, make sure it is resolved properly

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.76%. Comparing base (34543ea) to head (06de8e1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10452      +/-   ##
==========================================
- Coverage   85.77%   85.76%   -0.01%     
==========================================
  Files         372      372              
  Lines       40065    40063       -2     
==========================================
- Hits        34364    34362       -2     
  Misses       5701     5701              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@braingram braingram marked this pull request as ready for review April 14, 2026 20:47
@braingram braingram requested a review from a team April 14, 2026 20:47
@braingram braingram requested a review from emolter as a code owner April 14, 2026 20:47
@emolter emolter added this to the Build 13.0 milestone Apr 15, 2026
Comment thread changes/10452.outlier_detection.rst
Copy link
Copy Markdown
Collaborator

@emolter emolter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM just one comment about the changelog

@drlaw1558
Copy link
Copy Markdown
Collaborator

Ah, this is probably why I had to put the SOSS notebook onto a 64 GB github CI runner. Great to see the lower memory footprint plus faster runtime, thanks!

Copy link
Copy Markdown
Collaborator

@melanieclarke melanieclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you! One small request for more explanation.

Comment thread jwst/outlier_detection/tso.py
@melanieclarke melanieclarke changed the title Improve performance of tso median calculation in outlier detection JP-4323: Improve performance of tso median calculation in outlier detection Apr 16, 2026
@melanieclarke melanieclarke enabled auto-merge (squash) April 16, 2026 20:41
@melanieclarke melanieclarke merged commit e9d87eb into spacetelescope:main Apr 16, 2026
30 checks passed
@braingram braingram deleted the tso_med branch April 16, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants