evals: offline regression harness over verified review misses

[heskew]

The lgtmaybe-inspired eval harness: replay verified production review misses against a prompt ref + model, judge whether the review now catches each known defect. Calibration PRs currently ship blind and wait a week for production verdicts; this answers "does the new text catch the misses that motivated it?" in minutes.

Shape

• evals/run-eval.sh — per fixture: standalone clone at the reviewed head (single-branch, ancestry-only — see below), production-composed layer scope at the ref under test (compose-review-scope.sh reused), claude -p agentic review with read-only tools, cheap-model judge (judge-prompt.md) scoring caught/partial/missed, TSV results + optional baseline regression gate.
• evals/repos.yml — per-repo layer sets mirroring the fleet callers.
• Fixtures are metadata-only YAML in ai-review-log (companion PR there): 17 verified misses pinned by base/reviewed SHAs — severity deflation (security + correctness), producing-side gaps, numeric coercion, teardown leaks, test validity.

Smoke-test findings (already fixed in this PR)

The first end-to-end run exposed an eval-design trap worth recording: a shared worktree leaks the future — git log --all sees post-review refs including the fix commit, and the reviewer treats the defect as already-handled. Checkouts are now single-branch clones of the reviewed commit's ancestry with the remote removed. With isolation, the smoke fixture (harper#1770 whitespace Retry-After) scores caught with the exact mechanism at the current ref — the numeric-coercion rule demonstrably working.

Fidelity caveats (v1, documented in the README)

No PR title/body/comments context; single claude -p pass rather than the full action harness; shared layers only (no caller repo-specific-checks). Good enough to regression-gate prompt edits; not a substitute for production calibration.

Next

• Full-corpus baseline run (in progress) — doubles as the offline measure of how many historical misses the post-calibration: week of 2026-06-29 #73/calibration: week of 2026-07-20 #80 rules already catch; committed as evals/baseline.tsv in a follow-up.
• Then the coverage-line ("Traced / not traced") prompt edit ships validated against these fixtures.

Companion: ai-review-log fixtures PR.

---

🤖 Generated with Claude Code

[github-actions]

Reviewed; no blockers found.

[gemini-code-assist]

Code Review

This pull request introduces an offline regression evaluation harness for review prompts, adding a test runner script (run-eval.sh), judging instructions, repository configurations, and documentation. The feedback focuses on improving the robustness of the shell script, specifically by using jq instead of sed for parsing JSON judge outputs, and using awk instead of grep for exact string matching of fixture IDs in the baseline file.

[heskew]

First full-corpus baseline is in (dc2f92d, evals/baseline.tsv): claude-sonnet-5 at current main catches 10/17, partial 4, missed 3.

Every fixture was a total miss at its original ref, so this doubles as the offline measure of the #70→#80 calibration arc: 59% of historical misses now caught outright, 82% at least surfaced. Highlights: the stale-context.user CRITICAL (harper#1535), the chain-abort leak, the nested-SQL authz gap, and all three producing-side oauth cases (#183/#184/#192 — #80's brand-new rule going 3-for-3) now score caught with exact mechanisms.

Survivors — the next calibration targets:

• harper-1858-mtls-required-mirror (config-mirroring gap between sibling servers)
• harper-1906-unawaited-puts-test-race (test-validity: unawaited async writes)
• oauth-170-http-issuer-jti-replay (cleartext issuer + non-atomic JTI check)

Flakiness note: harper-1770 and harper-1956 vote-split across repeat runs (caught vs partial/missed); recorded at their best-evidenced verdict. Single-run verdicts on marginal fixtures are noisy — a future improvement is N-vote judging on fixtures that have ever split.

Run conditions: reviewer claude-sonnet-5, judge claude-haiku-4-5, working tree ≈ 224c2ad, ~2-3 min/fixture sequential.

[heskew]

Review disposition (9175f61) — all five findings applied:

1. sed JSON parsing → jq (both threads): flat-object extraction + jq parse. This was demonstrably live — the empty evidence strings in the baseline runs were pretty-printed judge output defeating the sed regex.
2. Cache key collision: repo cache dirs now keyed by full org/name.
3. Clone cleanup: eval clones reclaimed on clean verdicts, kept for debugging on parse failures.
4. Baseline gate: awk exact-match on the id column instead of grep -E.

(Three inline threads could not take replies — their comment IDs were superseded when the dogfood review re-ran on the baseline push — so dispositions are recorded here; all threads resolved.)

[heskew]

Codex (OpenAI) review: four confirmed findings are posted inline. The existing checks are green, but the current Script tests do not exercise evals/run-eval.sh.