bench: evaluate redesigning the experiment backend against the honest-scholar contract

[davorrunje]

Context

The honest-scholar integration (#136) binds experiment_backend: benchmarks/ in .honest-scholar/config.yml — the repo-local harness that implements the run/evidence/tables/is-current contract, whose result JSONs findings.md cite as evidence. PR #127 (design/benchmark-experiment-orchestration, open draft) proposes a benchmark-orchestration design and is intentionally kept open as the experiment-backend implementation spec. A redesign of this backend against the honest-scholar contract may be warranted.

Goal

Evaluate and, if warranted, redesign the experiment backend so it cleanly satisfies the honest-scholar experiment-backend contract — in particular stable run-refs and an is-current (staleness) check that findings.md/ledger.md can rely on.

Where

• .honest-scholar/config.yml (experiment_backend block; run_ref_scheme: path).
• benchmarks/ — run.py, search.py, _common/results.py, benchmarks/results/.
• Contract reference: honest-scholar resources/contracts/experiment-backend.md (in the plugin).
• PR Design spec: benchmark experiment orchestration #127 (design to reconcile).

Proposed approach

Compare #127's orchestration design against the honest-scholar experiment-backend contract; decide whether to adopt #127 as-is, adapt it, or redesign. Define a durable run-ref scheme (the current binding uses path) and an is-current implementation so staleness is checkable before sign-off. Update .honest-scholar/config.yml if the interface changes.

Acceptance criteria

• Documented decision on Design spec: benchmark experiment orchestration #127 (adopt / adapt / redesign / close).
• Run-ref scheme and is-current check specified (and implemented, or split into a follow-up).
• .honest-scholar/config.yml experiment_backend block matches the chosen design.

References

• chore: adopt honest-scholar as the research-workflow driver (install + research-init adopt) #136, Design spec: benchmark experiment orchestration #127
• .honest-scholar/config.yml, benchmarks/