Skip to content

bench: adopt log_test_trajectory for sensitivity runs (drop Curve-B re-eval) #121

Description

@davorrunje

Context

benchmarks/_common/search.py:search has an optional log_test_trajectory: bool flag (added in PR #117) that records each trial's held-out test metric as the Optuna trial user-attr test_metric during the search — a pure diagnostic, never fed into selection. When present, benchmarks/_common/sensitivity_report.py:incumbent_test_curve reads that attr and needs no re-eval for Curve B. The committed studies predate the flag, so Curve B currently falls back to the (slow) incumbent re-eval.

Goal

Run dedicated sensitivity studies with log_test_trajectory=True so Curve B comes for free from storage, and retire the incumbent re-eval path as the default.

Where

  • benchmarks/_common/search.py:search (flag already implemented).
  • benchmarks/sensitivity.py (would consume stored attrs; no re-eval).
  • benchmarks/_common/sensitivity_report.py:incumbent_test_curve (already prefers stored test_metric, falls back to re-eval).

Acceptance criteria

  • A sensitivity study set is run with log_test_trajectory=True.
  • Curve B for those studies reconstructs with # re-eval = 0.
  • Docs note Curve B is now storage-only for runs using the flag.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestfollow-upDeferred work captured as a self-contained issue

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions