Skip to content

A noise-free curve yields a zero-width confidence interval, which is correct arithmetic and a misleading claim #125

Description

@marcohost33-maker

Found in round 20 of PR #121, recorded rather than fixed because the honest fix is a decision about
what an interval is allowed to mean, not a patch.

What happens

Feed fitting/bootstrap.py a curve that the model describes exactly, with no noise: the estimated
residual scale is about 1e-15, every parametric resample is essentially the same curve, and the
returned BCa interval has width 0.0.

Why this is not simply a bug

Inside the parametric bootstrap this is arithmetically correct: the resampling distribution really
does have no spread. The defect is at the interface. A confidence interval of width zero is read by
every downstream consumer as maximal certainty, when what actually happened is that the
uncertainty model had nothing to work with.

That makes it the same family as the findings this PR closed — the number is not wrong, the
claim carried by the number is.

Options, none of them free

  1. Refuse to emit a finite interval below a residual-scale floor and mark uncertainty unavailable.
    Honest, but introduces a threshold that must itself be justified.
  2. Emit the interval and a companion flag saying the resampling distribution was degenerate.
    Cheapest, but relies on consumers reading the flag.
  3. Leave as is and document. Weakest: a documented limit prevents nothing.

Maintainer decision — this changes what a published interval means.

Discrimination requirement whichever route is taken

The chosen behaviour must fire on the noise-free curve and stay silent on a curve with ordinary
noise. Both directions in one test; the silent direction is the one that is usually forgotten.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions