Skip to content

READINESS_REDESIGN_PHASE1_PASSIVE_FEASIBILITY.md

Latest commit

 

History

History
83 lines (62 loc) · 3.88 KB

READINESS_REDESIGN_PHASE1_PASSIVE_FEASIBILITY.md

File metadata and controls

83 lines (62 loc) · 3.88 KB
Raw

Readiness Redesign — Phase 1 Passive rolling_3d Model Feasibility

Auto-generated by analysis/phase2_passive_feasibility/feasibility.py. Re-run any time.

Methodology

  • Target: passive_efficiency / rolling_3d

  • Test slice: source_2025_current only (2025-01-01 → 2026-05-15)

  • Features: Passive own-features from feature_snapshots (no cross-sub_score signals on this iteration).

  • Models: OLS + Ridge over alpha grid [0.01, 0.1, 1.0, 10.0, 100.0]. No Lasso, no trees.

  • Primary split: chronological 70/30.

  • Sensitivity: expanding walk-forward monthly blocks.

  • Bootstrap: block bootstrap with 14-day contiguous blocks, 1000 iterations. Preserves autocorrelation that a shuffled bootstrap would destroy.

  • Standardisation: per-feature z-score, fitted on train and applied to test (no leakage).

  • Floor to beat: ewma_45d MAE point 3.1911 bpm, lower CI bound 2.9263 (from floors report on full test slice).

  • Success criterion: model MAE on primary test must beat 2.9263 bpm — the lower CI bound, NOT the point estimate. Beating the point estimate but not the CI lower bound is statistical noise.

Primary 70/30 split

  • Total rows: 315 (after dropping 18 rows with missing features)
  • Train: 220 rows, 2025-02-18 → 2026-02-03
  • Test: 95 rows, 2026-02-04 → 2026-05-13

Inner train/val for alpha selection (no test leakage)

Train period split chronologically 80/20: train' = 176 rows, val = 44 rows. Validation MAE per alpha — chosen alpha is the one that minimises val MAE; refit on the full train period and scored ONCE on test:

Ridge α val MAE chosen
0.01 4.5597
0.1 4.4705
1.0 4.1768
10.0 3.9482
100.0 3.8976 ✓

Model results on primary test (single evaluation each)

model MAE (bpm) 95% block-bootstrap CI
EWMA45 baseline (on same split) 3.0978 [2.5623, 3.9615]
OLS 4.2342 [3.2166, 5.4398]
Ridge α=100.0 (chosen via val) 3.0783 [2.4532, 3.9364]

Decision

Best linear model on primary test: Ridge α=100.0 with MAE = 3.0783 bpm, upper CI = 3.9364.

Floor to beat: 2.9263 bpm (lower bound of EWMA45 CI on the full test slice).

Verdict: no production model yet. Best linear model MAE (3.0783) does not beat the floor lower CI (2.9263). Per the agreed criterion this is not a candidate, even if the model's MAE happens to be below the floor's point estimate. Possible next steps before escalating to a tree model: cross-sub_score features (Recovery rolling_3d, sleep_debt_7d, sustained_hr_load) added to the feature set; alternate target encoding; or accept that EWMA45 is the production layer for Passive rolling_3d and shift focus to other sub_scores.

Sensitivity — expanding walk-forward, monthly blocks

Sanity check against the single primary split. Each row trains on every month strictly before test_month, evaluates on that month.

test_month n_train n_test OLS MAE best Ridge MAE
2025-04 38 21 40.1115 2.5754
2025-05 59 23 14.1116 3.8851
2025-06 82 22 3.8572 3.9031
2025-07 104 8 1.5474 1.5514
2025-09 112 22 3.0572 3.1811
2025-10 134 27 2.7711 2.7727
2025-11 161 21 2.4110 2.3281
2025-12 182 22 3.5209 3.4238
2026-01 204 13 6.0975 4.8504
2026-02 217 28 3.0410 2.7729
2026-03 245 31 2.4479 2.3969
2026-04 276 30 4.6963 3.6954
2026-05 306 9 3.2646 2.2427

Across all monthly tests:

  • OLS mean MAE: 6.9950, median MAE: 3.2646
  • Ridge α=0.01 mean MAE: 5.0288
  • Ridge α=0.1 mean MAE: 4.3054
  • Ridge α=1.0 mean MAE: 3.8758
  • Ridge α=10.0 mean MAE: 3.3964
  • Ridge α=100.0 mean MAE: 3.1910

If walk-forward mean MAE is materially different from the primary test MAE, the primary split caught an unusually favourable or unfavourable test tail.