Spike: build the calibration-corpus runner (eval/calibration has no executor)

[tylerwillis]

Spike. eval/calibration/ contains 14 cases and a README describing a regression ratchet — and no runner exists anywhere in src/ or scripts/. Nothing executes it.

Why this is the prerequisite, not a nice-to-have

REQ-003.8.2 forbids prompt-template changes that lose a known catch. That obligation is currently unenforceable: every prompt-template and hashing change shipped so far has been an unmeasured quality gamble.

This matters immediately, because the two highest-value pending proposals both touch exactly the machinery the corpus is supposed to protect:

• adding falsifiability/quantifier probes to renderInstructions (changes reviewer behaviour on every requirement)
• semantics-preserving hash normalization, so comment- and whitespace-only edits stop invalidating verdicts (changes what counts as the same evidence)

Both are defensible on the evidence. Neither is safe to ship blind. The hashing one in particular carries real residual risk — a normalizer bug would silently mask a semantic edit, which is a quality regression disguised as an ergonomics win.

What the spike should answer

1. What does a runner need to be worth trusting? Presumably: for each case, dispatch the review with the current template, compare the verdict against the recorded expectation, and report per-case pass/fail plus an aggregate catch-rate.
2. Is 14 cases enough to detect a real regression, or does the corpus need expanding first? Both repos now have rich material — roughly 1,314 recorded verdicts, including ~35 substantive rejections in phoneopticon phrased as named surviving counterexamples, and the islice / inert-#86 / auth-bypass catches. Harvesting real historical catches into cases is likely higher value than authoring synthetic ones.
3. Non-determinism: LLM judgment varies run to run. Does the ratchet need N-of-M sampling, or a tolerance band, to avoid flapping? A corpus that fails randomly gets disabled, which is worse than no corpus.
4. Issue Mechanically validate calibration provenance anchors #20 notes provenance anchors can silently rot. Does the runner need to verify anchors before trusting a case?

Cost note

Running 14 judgment reviews per template change is not free, but it is trivially cheaper than the alternative currently in force, which is discovering a lost catch in production. Measure it as part of the spike — instrumented review cost is itself unmeasured today (the 7–8 min / 80–90k tokens figure is a single-session anecdote, and parallel dispatch has produced 6 verdicts in 5 minutes).

Priority

Quality infrastructure. Blocks nothing today, but gates anything that touches prompts or hashing — which is most of what would meaningfully improve speed and cost.