|
| 1 | +# Harness Checkpoint Experiment Compare |
| 2 | + |
| 3 | +## Traceability |
| 4 | + |
| 5 | +- ADR ID: `ADR-0004` |
| 6 | +- Status: Proposed |
| 7 | +- Decision date: 2026-08-17 |
| 8 | +- Related specs: [Harness Coding Compare](../specs/2026-08-15-harness-coding-compare.md), |
| 9 | + [Session Checkpoint Executor PoC](../specs/2026-08-16-session-checkpoint-executor-poc.md), |
| 10 | + [Harness UI Studio](../specs/2026-08-15-harness-ui-studio.md) |
| 11 | +- Implementation spec: [Checkpoint-anchored multi-lane harness |
| 12 | + experiments](../specs/2026-08-17-harness-studio-checkpoint-compare.md) |
| 13 | +- Source abstraction: [Checkpoint-backed Compare Sources and |
| 14 | + Materialization](checkpoint-backed-compare-sources.md) |
| 15 | + |
| 16 | +## Context |
| 17 | + |
| 18 | +Two evidence systems exist today and neither can express a mixed-origin, |
| 19 | +multi-lane experiment: |
| 20 | + |
| 21 | +- `harness-compare.v1` freezes a fixture, runs exactly two variants |
| 22 | + (`baseline`/`candidate`) serially against a synthetic Git repository, and |
| 23 | + derives a single verdict. The manifest carries one global `runtime.model`, |
| 24 | + so per-lane model variation is inexpressible, and the treatment axis is |
| 25 | + limited to `harness | runtime-profile`. |
| 26 | +- The session executor owns the checkpoint contract |
| 27 | + (`session-execution-plan-v1`): base commit and tree plus session file |
| 28 | + digest, entry id, and branch digest. Execution results land on |
| 29 | + `refs/better-harness/session-executions` namespaced refs and never switch |
| 30 | + the user's branch. |
| 31 | + |
| 32 | +Studio users want to pick a checkpoint, replay the observed historical |
| 33 | +trajectory, run two fresh agents (different harness or different model) from |
| 34 | +the same checkpoint in parallel, and compare the three trajectories side by |
| 35 | +side. The first executable adapter resolves a Git commit and materializes |
| 36 | +detached worktrees, but ADR-0005 keeps those mechanics out of the generic |
| 37 | +product contract. The temptation is to add a "sandbox checkpoint" type, copy |
| 38 | +source-specific fields into a new experiment schema, or let a single global |
| 39 | +verdict summarize a three-lane view. Each of those blurs provenance or |
| 40 | +fabricates attribution. |
| 41 | + |
| 42 | +## Decision |
| 43 | + |
| 44 | +- **One checkpoint definition per adapter.** The current session-executor plan |
| 45 | + is the first checkpoint contract. Future source adapters may own other |
| 46 | + versioned checkpoint formats under ADR-0005. A sandbox is a materialization |
| 47 | + of the referenced checkpoint for one lane's execution, never a new checkpoint |
| 48 | + type. Experiment documents hold a `checkpointRef` (path plus digest) and never |
| 49 | + copy or reinterpret checkpoint fields. |
| 50 | +- **The `.harness` grammar does not change.** The upgrade lives entirely in |
| 51 | + the experiment and compare layer. `harness-compare.v1` remains the frozen |
| 52 | + fixture, two-variant path; it is not extended to cover checkpoints. |
| 53 | +- **New `harness-experiment.v1` manifest.** The version starts at v1; no |
| 54 | + prior experiment schema exists. It declares one shared task (prompt hash, |
| 55 | + grader) and N lanes. A lane has `origin: "observed"` (a recorded trajectory |
| 56 | + reference plus its starting checkpoint digest; no sandbox is created) or |
| 57 | + `origin: "execute"` (a harness id plus a per-lane runtime profile and model; |
| 58 | + a sandbox is materialized from the shared checkpoint). The host, the visible |
| 59 | + tool set, and the run policy are shared across lanes rather than per-lane, so |
| 60 | + a lane cannot move the host and confound every contrast at once. |
| 61 | +- **Treatment axes are derived, never author-declared.** A contrast names |
| 62 | + only the lanes it compares. The runner computes the axis by diffing the |
| 63 | + lanes' harness id, runtime profile, and model. Exactly one differing axis |
| 64 | + (`harness`, `runtime-profile`, or `model` — extending the existing |
| 65 | + taxonomy) permits an attribution verdict; more than one yields |
| 66 | + `multi-axis`, which is descriptive only and can never produce a harness |
| 67 | + accept/reject. |
| 68 | +- **The statistical evidence bar is preserved.** Execute lanes carry a trial |
| 69 | + count, and per-contrast verdicts are decided by the existing matched-pair |
| 70 | + decision policy, including the two-matched-pair floor. A contrast whose |
| 71 | + lanes ran once each is attributable in principle but reports |
| 72 | + `insufficient_evidence`, never a promotion; the experiment schema is not a |
| 73 | + bypass for the compare evidence bar. |
| 74 | +- **An identically configured pair is descriptive, not invalid.** Where |
| 75 | + `harness-compare.v1` rejects a manifest whose variants move nothing, an |
| 76 | + experiment accepts it and reports `no-axis-moved`. Two identical lanes from |
| 77 | + one checkpoint measure run-to-run variance, which is the noise floor every |
| 78 | + other contrast is read against. |
| 79 | +- **Checkpoint completeness is a gate, not an assumption.** A checkpoint |
| 80 | + anchors a commit and tree, not the untracked files or dirty state the |
| 81 | + historical trajectory may have started from. Materialization records a |
| 82 | + completeness receipt: either a clean-tree assertion or a captured |
| 83 | + dirty-state patch applied to every fresh lane. Without it, fresh lanes may |
| 84 | + not claim to share the historical lane's starting condition. |
| 85 | +- **Observed lanes are contextual evidence by default.** A historical |
| 86 | + trajectory participates in an attribution verdict only when its runtime, |
| 87 | + model, harness revision, environment receipts, and task identity (prompt |
| 88 | + and prior session context) all match the fresh lanes. Absent any of those |
| 89 | + — and the prompt almost never matches — it is displayed as context in a |
| 90 | + descriptive contrast. Grades are optional on observed lanes. |
| 91 | +- **Execution discipline for parallel lanes.** All lanes pass preflight |
| 92 | + (checkpoint digest, base commit/tree, session digest) before any lane |
| 93 | + starts. Worktree materialization is serialized to avoid Git lock |
| 94 | + contention; execution then runs in parallel with `Promise.allSettled`, so |
| 95 | + one lane's failure never discards another lane's evidence. Every event |
| 96 | + carries `experimentId`, `laneId`, and `runId`. Each lane persists its own |
| 97 | + `HarnessRevision`, runtime/model receipt, sandbox receipt, trajectory, |
| 98 | + patch, and grade. Results stay on namespaced refs; adopting a result is a |
| 99 | + later explicit action. |
| 100 | +- **Studio hosts experiments, not one global verdict.** The stateless |
| 101 | + run-per-request `/agui` endpoint is insufficient. Studio gains a |
| 102 | + server-side experiment registry (create an experiment, subscribe to |
| 103 | + per-lane event streams, cancel), persists evidence per experiment |
| 104 | + directory, previews which axes a configuration moves before running, and |
| 105 | + renders one verdict per contrast. A three-lane view never shows a single |
| 106 | + aggregate verdict. |
| 107 | +- **Tool-chain correlation is explicit evidence, not a similarity score.** |
| 108 | + Studio normalizes each ACP-derived tool call into tool name, resource target, |
| 109 | + and canonical arguments, then aligns lane sequences one-to-one and in order. |
| 110 | + The UI labels a counterpart `exact`, `same-resource`, `same-tool`, or `none` |
| 111 | + and shows the neighbouring calls around it. This makes a shared file read and |
| 112 | + a shared Read → Edit → Test path visible without presenting fuzzy similarity |
| 113 | + as provenance or causal proof. |
| 114 | + |
| 115 | +## Consequences |
| 116 | + |
| 117 | +Mixed-origin comparison becomes expressible without expanding the core |
| 118 | +`.harness` DSL or forking the checkpoint contract. The compare aggregate |
| 119 | +taxonomy gains a `model` axis and per-contrast decisions; consumers of |
| 120 | +`harness-compare-result.v1` are unaffected because `harness-compare.v1` |
| 121 | +remains frozen. Observed trajectories gain a principled, limited role: |
| 122 | +context by default, baseline only under full identity. The cost is a second |
| 123 | +manifest schema to maintain, a Studio server that now holds run state, and |
| 124 | +the obligation to keep the derived-axis rule and evidence floor enforced in |
| 125 | +the runner rather than trusted from the document. Implementation still |
| 126 | +requires the dated spec, acceptance scenarios, and test evidence mandated by |
| 127 | +`AGENTS.md`. |
0 commit comments