Context
benchmarks/_common/search.py:search has an optional log_test_trajectory: bool flag (added in PR #117) that records each trial's held-out test metric as the Optuna trial user-attr test_metric during the search — a pure diagnostic, never fed into selection. When present, benchmarks/_common/sensitivity_report.py:incumbent_test_curve reads that attr and needs no re-eval for Curve B. The committed studies predate the flag, so Curve B currently falls back to the (slow) incumbent re-eval.
Goal
Run dedicated sensitivity studies with log_test_trajectory=True so Curve B comes for free from storage, and retire the incumbent re-eval path as the default.
Where
benchmarks/_common/search.py:search (flag already implemented).
benchmarks/sensitivity.py (would consume stored attrs; no re-eval).
benchmarks/_common/sensitivity_report.py:incumbent_test_curve (already prefers stored test_metric, falls back to re-eval).
Acceptance criteria
References
Context
benchmarks/_common/search.py:searchhas an optionallog_test_trajectory: boolflag (added in PR #117) that records each trial's held-out test metric as the Optuna trial user-attrtest_metricduring the search — a pure diagnostic, never fed into selection. When present,benchmarks/_common/sensitivity_report.py:incumbent_test_curvereads that attr and needs no re-eval for Curve B. The committed studies predate the flag, so Curve B currently falls back to the (slow) incumbent re-eval.Goal
Run dedicated sensitivity studies with
log_test_trajectory=Trueso Curve B comes for free from storage, and retire the incumbent re-eval path as the default.Where
benchmarks/_common/search.py:search(flag already implemented).benchmarks/sensitivity.py(would consume stored attrs; no re-eval).benchmarks/_common/sensitivity_report.py:incumbent_test_curve(already prefers storedtest_metric, falls back to re-eval).Acceptance criteria
log_test_trajectory=True.# re-eval = 0.References
docs/superpowers/specs/2026-07-15-hp-search-sensitivity-curves-design.md(follow-up 3).