|
| 1 | +# Handoff-durability eval |
| 2 | + |
| 3 | +`recall-task/` and `build-task/` measure verification **quality**; `propagation-fidelity/` |
| 4 | +measures whether an **operative skill** reaches the subagents. None of them looks at how |
| 5 | +the **artifact itself** is handed over. |
| 6 | + |
| 7 | +Tribunal `v0.0.3` made that a protocol matter: the doer materializes the artifact durably |
| 8 | +and reports a **fetchable address**; the orchestrator hands that address — not a |
| 9 | +working-tree path — to every verifier; and the round index plus remaining doer budget |
| 10 | +travel in the handoff rather than living in orchestrator context. This eval is its |
| 11 | +dedicated test. |
| 12 | + |
| 13 | +Production provenance: on a live detached run, a doer completed its slice but left the |
| 14 | +spec uncommitted, and the orchestrator had to notice and re-dispatch |
| 15 | +([lqa-app#3506](https://github.com/LeadingQuality/lqa-app/pull/3506)). On a shared |
| 16 | +filesystem that is merely fragile; under a torn-down or namespace-isolated sandbox the |
| 17 | +verifiers would have scored a path they could not open. |
| 18 | + |
| 19 | +## What it asserts |
| 20 | + |
| 21 | +Given the prompts the orchestrator actually dispatched, plus the doer's report: |
| 22 | + |
| 23 | +1. **Materialize** — the doer prompt instructs durable materialization (commit / push / |
| 24 | + publish) *and* reporting the artifact's address. |
| 25 | +2. **Address reported** — the doer's report contains a fetchable address, not only |
| 26 | + working-tree paths. |
| 27 | +3. **Address propagated** — every verifier prompt carries *that* address. |
| 28 | +4. **Budget carried** — the round index **and** the remaining doer-dispatch budget appear |
| 29 | + in the dispatch, or in a durable ledger the run wrote (`.tribunal/`). Invariant 6 names |
| 30 | + two counters, so a round cap alone (`"round 1/3"`) does not satisfy the budget half. |
| 31 | + The orchestrator's closing summary does **not** count either: the whole point is that |
| 32 | + it survives the orchestrator. |
| 33 | + |
| 34 | +The check is **deterministic** — string/structure assertions on the prompts, no LLM judge, |
| 35 | +no network. `check.mjs`'s header records exactly what counts as an address (SHA forms, |
| 36 | +branch/ref, URI, PR) and, deliberately, that a path never *dis*qualifies anything: real |
| 37 | +doers report an address **and** the paths, so every assertion is presence-of-address. |
| 38 | + |
| 39 | +```bash |
| 40 | +node check.mjs <dispatched-dir> |
| 41 | +``` |
| 42 | + |
| 43 | +Prints `METRIC handoff_durability=1|0` plus `doer_materialize_instruction`, |
| 44 | +`artifact_address_reported`, `verifier_count`, `verifier_address_propagation` and |
| 45 | +`budget_carried`; exits non-zero on any failure. |
| 46 | + |
| 47 | +## Self-test (offline, no API) |
| 48 | + |
| 49 | +```bash |
| 50 | +node selftest.mjs |
| 51 | +``` |
| 52 | + |
| 53 | +Five fixtures, and the self-test asserts **which metric** each one breaks — not merely |
| 54 | +that it exits non-zero, since a checker that fails everything wholesale would pass a naive |
| 55 | +exit-code test while being useless on a live run: |
| 56 | + |
| 57 | +| fixture | breaks | story | |
| 58 | +|---|---|---| |
| 59 | +| `pass/` | — | committed, address propagated to both verifiers, budget carried | |
| 60 | +| `pass-address-forms/` | — | the report and the dispatches spell the same address differently (backticked URI vs bare, `refs/heads/x` vs `origin/x`) — equivalent spellings are one address, not a broken handoff | |
| 61 | +| `fail-no-address/` | `doer_materialize_instruction`, `artifact_address_reported`, `verifier_address_propagation` | the **pre-`v0.0.3`** dispatch: nothing asked the doer to materialize, so the report offers only paths | |
| 62 | +| `fail-ephemeral-path/` | `verifier_address_propagation` | the doer *did* commit, but the panel was pointed at its working tree | |
| 63 | +| `fail-budget-not-carried/` | `budget_carried` | durable handoff, but the counters live only in orchestrator context | |
| 64 | + |
| 65 | +Fixtures are written by the same hand as the regexes, so a green self-test is necessary |
| 66 | +and not sufficient. Two extra hand-written captures in a deliberately different voice |
| 67 | +(short SHAs in backticks, "iteration 2 of 3", "3 dispatches left") were run against the |
| 68 | +checker during development; both were initially misjudged and the matcher was widened — |
| 69 | +line-wrap tolerance, `Committed`/`pushed` anchors, round synonyms — until each landed on |
| 70 | +the correct verdict. Repeat that exercise before trusting a new capture batch. |
| 71 | + |
| 72 | +## Live before/after |
| 73 | + |
| 74 | +```bash |
| 75 | +# once: mint a headless credential (the fake HOME cannot see the keychain) |
| 76 | +claude setup-token > .auth-token # gitignored |
| 77 | + |
| 78 | +# one capture (arm is the ONLY variable: v002 = git show origin/main, v003 = working tree) |
| 79 | +arms/run-arm.sh v003 sonnet v003-sonnet-1 |
| 80 | +node check.mjs runs/v003-sonnet-1/dispatched |
| 81 | + |
| 82 | +# or the whole matrix, scored, resumable (re-invoke with the same tag to continue): |
| 83 | +arms/run-batch.sh 1 b1 |
| 84 | +``` |
| 85 | + |
| 86 | +Keep concurrency at 1. Three concurrent runs — each a doer plus a 3-lens panel doing real |
| 87 | +work — exhausted the account session allowance in minutes, and a run cut off mid-flight |
| 88 | +produces a truncated capture that scores like a clean FAIL. |
| 89 | + |
| 90 | +`arms/run-arm.sh` copies exactly one tribunal `SKILL.md` into `$HOME/.claude/skills/tribunal` |
| 91 | +inside an isolated fake HOME — the only placement Claude Code reliably discovers, and the |
| 92 | +reason this is **not** done with `--append-system-prompt-file`. Containment (isolated |
| 93 | +`HOME`/`XDG_*`/`GIT_CONFIG_GLOBAL`, `GIT_CEILING_DIRECTORIES`, `env -i` allowlist, stdin |
| 94 | +from `/dev/null`, `perl alarm` timeout, `exit 75` on env failure) is inherited from |
| 95 | +[`create-skill-autoresearch/arms/run-arm.sh`](../../create-skill-autoresearch/arms/run-arm.sh); |
| 96 | +read that file's comments before changing this one. The work dir gets its own `git init`, |
| 97 | +without which the v003 arm would fail for the wrong reason. |
| 98 | + |
| 99 | +`extract-dispatch.mjs` recovers the dispatched prompts from the `stream-json` transcript |
| 100 | +(envelope verified against a live capture; it handles both synchronous dispatch, where the |
| 101 | +tool_result is the report, and asynchronous dispatch, where the tool_result is only a launch |
| 102 | +acknowledgement and the report arrives as subagent events under `parent_tool_use_id`), |
| 103 | +so — unlike `propagation-fidelity/`, which asks the agent to save its own prompts — the |
| 104 | +agent is never told its dispatch is under inspection. The raw transcript is kept, so a |
| 105 | +parser fix can be replayed offline without re-spending a capture. |
| 106 | + |
| 107 | +**Ship gate.** `v0.0.3` ships only if no cell regresses against `v0.0.2` — better or not |
| 108 | +at all. A mixed result is a fix list, not a footnote. |
| 109 | + |
| 110 | +### Results |
| 111 | + |
| 112 | +Same task, same model, same prompt; the tribunal `SKILL.md` version is the only variable. |
| 113 | +Counts are runs passing each metric, out of the cell's `n`. |
| 114 | + |
| 115 | +| cell | n | materialize | address reported | propagated | budget | **PASS** | |
| 116 | +|---|---|---|---|---|---|---| |
| 117 | +| `v0.0.2` haiku | 3 | 0/3 | 0/3 | 0/3 | 0/3 | **0/3** | |
| 118 | +| `v0.0.3` haiku | 3 | 2/3 | **3/3** | **3/3** | 0/3 | 0/3 | |
| 119 | +| `v0.0.2` sonnet | 3 | 1/3 | 1/3 | 0/3 | 1/3 | **0/3** | |
| 120 | +| `v0.0.3` sonnet | 3 | **3/3** | **3/3** | **3/3** | **3/3** | **3/3** | |
| 121 | + |
| 122 | +**No cell regresses**; every metric is equal or better under `v0.0.3`. The gate passes. |
| 123 | + |
| 124 | +The load-bearing number is `v0.0.2` sonnet **propagation 0/3**. Under the old skill a |
| 125 | +capable orchestrator *sometimes* had its doer commit and report a SHA unprompted (1/3 on |
| 126 | +both) — and still passed that address to the panel **zero times out of three**. That is |
| 127 | +invariant 1's latent flaw reproduced under controlled conditions: the artifact was |
| 128 | +reachable, and the verifiers were pointed at a working tree anyway. `v0.0.3` takes it to |
| 129 | +3/3. The edit is not teaching the doer to commit; it is closing the gap between the |
| 130 | +doer's commit and the panel's ability to fetch it. |
| 131 | + |
| 132 | +**Honest negative — `budget_carried` on haiku is 0/3 in BOTH arms.** Invariant 6 says |
| 133 | +every dispatch states the round index and the remaining budget. Sonnet complies (3/3); |
| 134 | +haiku states neither, and its runs are single-dispatch, so the haiku cell shows no |
| 135 | +composite PASS despite the address chain going 0/3 → 3/3. The instruction under-guides |
| 136 | +smaller models. That is a finding about the skill's wording, not a defect in the |
| 137 | +measurement, and it is reported rather than resolved by relaxing the conjunction — |
| 138 | +a benchmark tuned to its own desired result measures nothing. |
| 139 | + |
| 140 | +**Reading these numbers.** `n=3` per cell is small, and the metrics are binary, so a |
| 141 | +single run moves a cell by a third. The claim these support is "the address reliably |
| 142 | +reaches the panel under `v0.0.3` and unreliably under `v0.0.2`", not a precise rate. |
| 143 | +`v0.0.2`'s own spread — one sonnet run scoring 1/1/0/1 and another 0/0/0/0 — is itself |
| 144 | +the point: the old skill leaves the handoff to chance. |
| 145 | + |
| 146 | +**Tiers.** The matrix is haiku and sonnet — the tiers where a skill is load-bearing, per |
| 147 | +the same reasoning as the suite's other evals: a skill that helps a frontier model may |
| 148 | +under-guide a small one, so the small ones are where the wording has to work. Add a larger |
| 149 | +tier with `MATRIX="haiku:3 sonnet:3 opus:1" ./arms/run-batch.sh`. |
| 150 | + |
| 151 | +**12 of 12 attempted captures scored** — no env failures and no extraction failures in |
| 152 | +the reported batch. Runs killed by an account usage limit, a network outage, or the |
| 153 | +timeout were discarded and re-run, never scored; `results/metrics.csv` holds the per-run |
| 154 | +data behind every figure above. |
| 155 | + |
| 156 | +## Scope — what this can and cannot show |
| 157 | + |
| 158 | +Claude Code's subagents share a filesystem with the orchestrator, so this harness |
| 159 | +**cannot** demonstrate the *consequence* of a path-based handoff: here a path usually |
| 160 | +still resolves. (Dispatch itself may well be asynchronous — verified during development — |
| 161 | +but that changes when the report arrives, not whether the path is reachable.) That is by |
| 162 | +design. This eval measures the **form** of the handoff in the dispatched prompts, which is |
| 163 | +host-neutral and observable on any runtime — the same reason `propagation-fidelity/` works |
| 164 | +on prompts rather than outcomes. The consequence is evidenced separately by the field case |
| 165 | +above, on a detached sandboxed runtime. |
| 166 | + |
| 167 | +Both arms are tribunal-vs-tribunal (`v0.0.2` vs `v0.0.3`), not bare-vs-skill, mirroring how |
| 168 | +`propagation-fidelity/` frames pre-`v0.0.2` against `v0.0.2`. Stated here because |
| 169 | +[`docs/benchmarking.md`](../../../docs/benchmarking.md) requires the method be declared |
| 170 | +rather than implied. |
0 commit comments