Skip to content

READINESS_REDESIGN_PHASE1_CHRONIC_LABEL_FEASIBILITY.md

Latest commit

 

History

History
70 lines (51 loc) · 4.67 KB

READINESS_REDESIGN_PHASE1_CHRONIC_LABEL_FEASIBILITY.md

File metadata and controls

70 lines (51 loc) · 4.67 KB
Raw

Readiness Redesign — Phase 1 Chronic chronic_label Classifier Feasibility

Auto-generated by analysis/phase5_chronic_label_feasibility/feasibility.py. Re-run any time.

Methodology

  • Target: chronic_load / chronic_label (binary classifier label).
  • Test slice: source_2025_current only (2025-01-01 → 2026-05-15).
  • Features: Chronic Load own-features from feature_snapshots (no cross-sub_score signals on this iteration).
  • Model: L2-regularised logistic regression (Newton-Raphson IRLS, intercept not penalised). Alpha grid [0.01, 0.1, 1.0, 10.0, 100.0].
  • Primary split: chronological 70/30.
  • Sensitivity: expanding walk-forward monthly blocks.
  • Bootstrap: STRATIFIED (positives and negatives resampled separately), 1000 iterations.
  • Alpha selection: inner train/val split (chronological 80/20 within the train period); chosen alpha refit on full train and scored ONCE on test — no hyperparameter leakage.
  • Primary metric: precision at recall = 0.5. Tied predictions evaluated as whole buckets (include-all-or-none).
  • Floor: event_base_rate precision at recall = 0.5 on the same test rows. Persistence is intentionally excluded — it scores high on forward-window labels via window-overlap, not predictive signal (documented in floors report).
  • Success criterion (stricter than continuous): model precision@R=0.5 LOWER CI bound must exceed floor precision@R=0.5 UPPER CI bound. Intervals must NOT overlap. Single-point lift below CI overlap is statistical noise.

Primary 70/30 split

  • Total eligible rows: 322 (after dropping 27 with missing features)
  • Train: 225 rows, 2025-02-26 → 2026-01-13
  • Test: 97 rows, 2026-01-14 → 2026-05-01
  • Test positives: 30 (0.309 base rate)

Inner train/val for alpha selection

Train period split chronologically 80/20: train' = 180 rows, val = 45 rows. Validation metrics per alpha — chosen alpha is the one that maximises val precision@R=0.5 (fallback to highest val AUC if precision is undefined on this fold):

L2 α val precision@R=0.5 val AUC chosen
0.01 0.875 0.965 ✓
0.1 0.778 0.949
1.0 0.778 0.931
10.0 0.778 0.929
100.0 0.778 0.926

Model vs floor on primary test (single evaluation each)

precision@R=0.5 95% stratified bootstrap CI lift over base rate AUC captured pos
event_base_rate floor 0.850 [0.682, 1.000] 2.75 0.794 17
L2 logistic α=0.01 (chosen via val) 0.750 [0.593, 1.000] 2.42 0.936 15

Decision

Verdict: no production model yet. Model precision@R=0.5 point estimate 0.750 (CI [0.593, 1.000]) vs floor point 0.850 (CI [0.682, 1.000]). CIs overlap. Per the agreed criterion (model lower CI must exceed floor upper CI), this is not a candidate. Possible next steps before escalating: cross-sub_score features (acute event lag features, recovery deterioration counts), different chronic_label criterion, or accept event_base_rate as the deployable layer.

Scope of the verdict. This is a statement about the current chronological test tail (2026-01-14 → 2026-05-01, 30 positives in 97 rows), not a global claim about the chronic_label label. If the walk-forward section below shows materially higher precision on earlier months with denser positives, that points to seasonality or regime dependence rather than a useless label. The production decision is governed by the primary chronological split because that is the tail the live system would score; revisit when more positives accumulate.

Sensitivity — expanding walk-forward, monthly blocks

Sanity check against the single primary split. Each row trains on every month strictly before test_month, evaluates on that month.

Alpha is selected per month via an inner train/val split inside the cumulative-train window — never on the held-out month. Floor precision@R=0.5 is reported on the same month rows as the model so the two columns are directly comparable.

test_month n_train n_test n_pos chosen α model precision@R=0.5 floor precision@R=0.5
2025-06 64 8 2 100.0 0.143 1.000
2025-07 72 27 8 0.01 1.000 0.174
2025-08 99 22 3 0.01 0.143 1.000
2025-09 121 15 2 100.0 1.000 0.333
2025-12 191 21 3 100.0 0.667 0.286
2026-01 212 20 15 0.01 0.800 0.727
2026-03 260 31 13 0.01 0.875 1.000
2026-04 291 30 13 0.1 1.000 0.909

Mean across monthly tests — model: 0.703, floor: 0.679.

Materially different from the primary split would indicate the 70/30 caught an unusually favourable/unfavourable test tail.