Skip to content

READINESS_REDESIGN_PHASE1_ACUTE_RISK_FEASIBILITY.md

Latest commit

 

History

History
72 lines (53 loc) · 4.75 KB

READINESS_REDESIGN_PHASE1_ACUTE_RISK_FEASIBILITY.md

File metadata and controls

72 lines (53 loc) · 4.75 KB
Raw

Readiness Redesign — Phase 1 Acute Risk event_t1_t3 Classifier Feasibility

Auto-generated by analysis/phase6_acute_risk_feasibility/feasibility.py. Re-run any time.

Methodology

  • Target: acute_risk / event_t1_t3 (binary classifier label).
  • Test slice: source_2025_current only (2025-01-01 → 2026-05-15).
  • Features: Acute Risk own-features from feature_snapshots (no cross-sub_score signals on this iteration).
  • Model: L2-regularised logistic regression (Newton-Raphson IRLS, intercept not penalised). Alpha grid [0.01, 0.1, 1.0, 10.0, 100.0].
  • Primary split: chronological 70/30.
  • Sensitivity: expanding walk-forward monthly blocks.
  • Bootstrap: STRATIFIED (positives and negatives resampled separately), 1000 iterations.
  • Alpha selection: inner train/val split (chronological 80/20 within the train period); chosen alpha refit on full train and scored ONCE on test — no hyperparameter leakage.
  • Primary metric: precision at recall = 0.5. Tied predictions evaluated as whole buckets (include-all-or-none).
  • Floor: event_base_rate precision at recall = 0.5 on the same test rows. Persistence is intentionally excluded — it scores high on forward-window labels via window-overlap, not predictive signal (documented in floors report).
  • Success criterion (stricter than continuous): model precision@R=0.5 LOWER CI bound must exceed floor precision@R=0.5 UPPER CI bound. Intervals must NOT overlap. Single-point lift below CI overlap is statistical noise.

Primary 70/30 split

  • Total eligible rows: 361 (after dropping 40 with missing features)
  • Train: 253 rows, 2025-03-16 → 2026-01-24
  • Test: 108 rows, 2026-01-25 → 2026-05-13
  • Test positives: 21 (0.194 base rate)

Inner train/val for alpha selection

Train period split chronologically 80/20: train' = 202 rows, val = 51 rows. Validation metrics per alpha — chosen alpha is the one that maximises val precision@R=0.5 (fallback to highest val AUC if precision is undefined on this fold):

L2 α val precision@R=0.5 val AUC chosen
0.01 0.364 0.614
0.1 0.400 0.602 ✓
1.0 0.400 0.589
10.0 0.364 0.577
100.0 0.348 0.588

Model vs floor on primary test (single evaluation each)

precision@R=0.5 95% stratified bootstrap CI lift over base rate AUC captured pos
event_base_rate floor 0.273 [0.193, 0.394] 1.40 0.612 12
L2 logistic α=0.1 (chosen via val) 0.229 [0.200, 0.750] 1.18 0.657 11

Decision

Verdict: no production model yet. Model precision@R=0.5 point estimate 0.229 (CI [0.200, 0.750]) vs floor point 0.273 (CI [0.193, 0.394]). CIs overlap. Per the agreed criterion (model lower CI must exceed floor upper CI), this is not a candidate. Possible next steps before escalating: cross-sub_score features (acute event lag features, recovery deterioration counts), different event_t1_t3 criterion, or accept event_base_rate as the deployable layer.

Scope of the verdict. This is a statement about the current chronological test tail (2026-01-25 → 2026-05-13, 21 positives in 108 rows), not a global claim about the event_t1_t3 label. If the walk-forward section below shows materially higher precision on earlier months with denser positives, that points to seasonality or regime dependence rather than a useless label. The production decision is governed by the primary chronological split because that is the tail the live system would score; revisit when more positives accumulate.

Sensitivity — expanding walk-forward, monthly blocks

Sanity check against the single primary split. Each row trains on every month strictly before test_month, evaluates on that month.

Alpha is selected per month via an inner train/val split inside the cumulative-train window — never on the held-out month. Floor precision@R=0.5 is reported on the same month rows as the model so the two columns are directly comparable.

test_month n_train n_test n_pos chosen α model precision@R=0.5 floor precision@R=0.5
2025-05 42 29 13 0.1 1.000 0.636
2025-06 71 27 9 0.01 0.500 0.294
2025-09 111 26 10 100.0 0.278 0.750
2025-10 137 31 9 0.1 0.312 0.227
2025-11 168 30 14 0.01 1.000 0.533
2025-12 198 31 5 0.01 0.375 0.130
2026-01 229 30 11 100.0 0.600 0.250
2026-02 259 28 6 1.0 0.300 0.273
2026-03 287 31 7 100.0 0.500 0.308
2026-04 318 30 8 0.01 0.286 0.800

Mean across monthly tests — model: 0.515, floor: 0.420.

Materially different from the primary split would indicate the 70/30 caught an unusually favourable/unfavourable test tail.