Skip to content

Implement alternative uncertainty propagation methods for full model uncertainty #535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

MoAly98
Copy link
Collaborator

@MoAly98 MoAly98 commented Apr 7, 2025

Closes #221.
Blocked by #505.
This PR implements two alternative methods to propagate parameter uncertainties to a model, following the discussion in #221. The two implemented methods are:

  • Error propagation by using the model's Jacobian (with finite-differences for derivatives) as implemented by iminuit
  • Error estimation by bootstrapping the model predictions

Both methods can be accessed from the model_utils.yield_stdev function directly, or via the model_utils.prediction method. In both cases, the user simply specifies the name of the method that should be used for propagating the uncertainty. In case of bootstrapping, the user can also specify the seed for the bootstrap random number generator and the number of samples to be produced.

To-do:

  • initial implementation of Jacobi method
  • initial implementation of bootstrap method
  • consistent variance array across all three methods of array propagation and harmonise transforming that to per-bin and per-channel quantities
  • Modularise the code by splitting the methods into individual functions
  • Wrap some common operations into modular methods
  • Add tests
  • Test closure in a model where closure is expected (model linear in all parameters, norm factors-only setup, all mods are linearly interpolated,..) -- See scikit-hep/pyhf/#2584

@MoAly98 MoAly98 self-assigned this Apr 7, 2025
Copy link

github-actions bot commented Apr 7, 2025

Status: Blocked ❌

Issues blocking this PR:


This comment was automatically written by the Blocking Issues bot, and this PR will be monitored for further progress.

@github-actions github-actions bot added the blocked Waiting for another PR/issue to be merged/closed. label Apr 7, 2025
Copy link

codecov bot commented Apr 7, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (bb682b4) to head (6a8af67).

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #535   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           22        22           
  Lines         2098      2159   +61     
  Branches       291       303   +12     
=========================================
+ Hits          2098      2159   +61     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MoAly98 MoAly98 requested review from alexander-held and removed request for alexander-held April 10, 2025 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Waiting for another PR/issue to be merged/closed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Alternative methods for model uncertainty calculation
1 participant