calibration: week of 2026-07-20

[github-actions]

Weekly calibration sweep for the PT week of 2026-07-20 (verdicts with closed_at in [2026-07-20, 2026-07-27)), synthesized from the raw verdict-labeled issues in HarperFast/ai-review-log.

The curated Calibration log (#1046) and False-negative log (#1045) exist for this week but contain only their boilerplate headers — no daily comments — so they contributed nothing. 45 verdict:pending issues remain open (13 a week ago): triage is falling behind ingest, so the mix below is a partial read.

Verdict mix

39 triaged: useful 22 · noise 7 · partial 10 (56% useful) — flat against last week's 53%. Verdicts arrived as one bulk batch (applied 2026-07-21/22, plus five late useful auto-applies on 07-24/25).

Model	Triaged	useful	noise	partial
claude-sonnet-5	23	16	4	3
claude-sonnet-4-6	10	4	0	6
gemini-3-flash-preview	6	2	3	1

Prompt ref	Triaged	useful	noise	partial
9cf49d2 (2026-06-29)	30	16	5	9
82c9484 (2026-07-21)	6	5	0	1
(none recorded — gemini)	3	1	2	0

Per repo: harper 24 (16/4/4) · oauth 10 (3/3/4) · harper-pro 5 (3/0/2).

The prompt-ref split is the load-bearing fact. 9cf49d2 predates #73 (merged as aa26ae2, 2026-07-17) and so carries neither the "Severity deflation on security / hardening paths" nor the "Fail-closed / validate-at-registration on new surfaces" bullet; 82c9484 carries both. Nine of the ten partials ran at 9cf49d2 — pre-text misses, not evidence the text failed. The post-text cell is 6 entries, far below the ~15 floor and unbalanced (5 of 6 are harper sonnet-5 clean runs from one day), so no before/after read is possible yet.

The sonnet-5 canary cell (23) clears the floor at 70% useful / 13% partial; sonnet-4-6 (10) does not, and reads 40% / 60%. As last week that gap is confounded, not a model-quality signal — sonnet-5 drew harper core PRs, sonnet-4-6 drew the oauth security work and harper-pro, where second-reviewer density is far higher.

Recurring NOISE patterns

All 7 are the same family — a "no blockers" run (three with full run-notes) on a diff with no correctness or security surface — and all were human-confirmed by heskew. Four are already covered by the "Mechanical, no-logic diffs" bullet: #1032 (workflow matrix conditional), #1024 (CI-lane sync), #1027 / #1028 (release bumps).

New corroborated sub-shape — type-only diffs with no runtime effect (2 points, both harper/sonnet-5, both heskew-confirmed): Promise<void> → Promise<unknown> widening to unblock tsc, which drew a four-bullet write-up over a behavior-neutral one-liner (#1040); Promise<void> → Promise<number | void> retype plus a removed redundant cast (#1011).

Watch-listed at 1 point each: reviewer-tool config as a diff surface (.gemini/config.yaml + styleguide, #1026); a release PR whose run re-traced its sibling PRs' fixes rather than its own +12/-3 diff (#1028).

Recurring PARTIAL / FALSE-NEGATIVE patterns

Every partial is the same shape: a clean run while a second reviewer — gemini inline, then a human (@kriszyp, heskew, cb1kenobi) who usually fixed it — caught a real defect. Corroboration is human/author-fixed rather than bot-only: seven of ten partials name the author commit that landed the fix.

1. Verified the enforcing side of a new guard, never the producing side — NEW (3 points, all oauth, 2 author-fixed).

• #1019 (oauth#183) — callback-side sessionId CSRF binding traced exhaustively and passed clean; the MCP initiation path never mints sessionId, so the new check silently no-ops for MCP flows. Author-fixed 300b06bed.
• #1021 (oauth#184) — dcrEnabled()'s cases enumerated and declared "all three cases correct"; a bare YAML dynamicClientRegistration: parses to null, which dcrConfig !== undefined treats as configured. Author-fixed 01a6f428b.
• #1042 (oauth#192) — run asserted an env-expanded "false" "cannot activate the gate"; @kriszyp shows an unexpanded ${VAR} placeholder survives coerceConfigBoolean as a truthy string and switches on a documented default-off gate. Unresolved.

Not covered by "Dispatch surfaces", "Fail-closed / validate-at-registration", or "Falsy vs nullish guards". → prompt edit.

2. Severity deflation on peer-flagged findings (5 points — but all pre-text). #987 (abort chain leaks native handles; @kriszyp-fixed 6be2c12d6/e0e78f003), #942 (Number(' ') → 0 on a function the run called "all correct"; heskew-fixed cb7de4825), #1035 (reprobe close-handling declared complete and "No gaps"; @kriszyp added the if (closed) return guard), #1009 (real catch on the untested generate: true branch, but generate+key called benign where gemini rated HIGH, and execFileSync event-loop block missed; both fixed in d30fb7f), #1017 (global prototype stub with no afterEach restore; author-fixed 8c622e764). All five at 9cf49d2, i.e. before #73's language shipped; #987 and #942 are also re-adjudicated carry-overs. → reinforce, no new text.

3. Concurrent peer review the run never saw (1 point, watch-listed). #1017 reported "no prior PR comments/threads to reconcile" while gemini had posted three findings — two high — 3.5 minutes earlier; the author confirmed both. A timing race against the existing "read the prior conversation" section, not a gap in it.

Covered / one-off: #978 (await Promise.all(checkpointWrites) outside the top try can bypass the park path — @kriszyp, unresolved; adjacent to "Setup/teardown that leaks on partial failure", watch-listed). #1020 is gemini severity inflation (doc-only drift filed as two blockers) — provider-harness output, out of scope; the underlying drift is real and shipped, and the Claude run on the same PR missed it too, a second point for a class the blocker list already names.

Changelist

• CALIBRATION.md — prepended the ## Week of 2026-07-20 entry.
• universal.md (Security) — new bullet "A new guard is only as good as whatever populates the value it reads." Enumerate every path that produces the value an enforcement check keys on, including paths that never set it and shapes the config layer can emit. Cites oauth#183 / #184 / #192.
• universal.md (What to ignore) — added "type-only edits with no runtime effect" to the "Mechanical, no-logic diffs" enumeration, with an explicit carve-out that a type change on an exported symbol is still an API-contract change and does need review.

No harper/*.md or repo-type/*.md edits — the oauth cluster is a plugin-agnostic guard-design shape.

No other calibration PR is open (only #18, an unrelated workflow change), so nothing here duplicates or contradicts in-flight work; #73, #77 and #78 have all merged.

🤖 Generated with Claude Code

[heskew]

1. Duplicate fixes are counted as independent calibration evidence

File: CALIBRATION.md:39

What: The two points cited for the new type-only pattern are not independent. ai-review-log #1011 and #1040 map to Harper PRs #1853 and #1890, both by the same author, both changing resources/DatabaseTransaction.ts::recordCommitLatency, and both fixing the same TS2345 regression introduced by Harper #1688. PR #1853 merged; the later #1890 closed unmerged.

Why it matters: This does not meet CALIBRATION-SWEEP.md's requirement of at least two independent data points before changing a prompt layer.

Suggested fix: Treat these as one underlying signal, keep the pattern watch-listed, and remove the universal.md edit until a genuinely independent occurrence appears.

2. “No runtime effect” can still hide build and declaration regressions

File: universal.md:94

What: The new rule ignores widened or narrowed annotations on internal functions based on runtime behavior. Such a change can leave emitted JavaScript unchanged while breaking tsc, weakening type checking, or altering declarations through an inferred exported type. The cited PRs themselves changed build status by repairing a failing TypeScript build.

Why it matters: universal.md:97 would mark these diffs as having nothing to verify and suppress their review log even though build and public type-contract behavior can change.

Suggested fix: Apply this exemption only after confirming type-check/build status, emitted output or metadata, and externally observable declarations are unchanged. Phrase the carve-out around the public declaration surface, not only directly exported symbols.

3. oauth#192 is incorrectly recorded as unresolved

Files: CALIBRATION.md:45, CALIBRATION.md:51, universal.md:50

What: OAuth commit 160992d4 fixed the unresolved ${FLAG} normalization gap on 2026-07-22, and oauth#192 merged on 2026-07-23, before this calibration PR was created. The calibration still calls it unresolved, and the prompt says only two of the three examples were author-fixed.

Why it matters: The calibration record understates its supporting evidence and leaves the aggregate fixed-partial count stale.

Suggested fix: Record oauth#192 as author-fixed by 160992d4, change “two author-fixed” to “all three author-fixed,” and update the fixed-partial count from seven to eight.

[heskew]

All three review findings verified against the sources and applied in 0caeb33:

1. Confirmed — one signal, not two. harper#1853 (merged) and #1890 (closed unmerged) are the same author fixing the same recordCommitLatency TS2345 regression. The universal.md type-only carve-out is withdrawn; the pattern is watch-listed in the entry with the improved framing for whenever an independent point appears.

2. Folded into the watch-list note — the future carve-out will be phrased around the public declaration surface and build status (type-check passes, emitted output and declarations unchanged), not just "no runtime effect".

3. Confirmed — oauth#192 was author-fixed in 160992d4 ("Make boolean config normalization total", 07-22) and merged 07-23, before this sweep ran. Entry and the new guard bullet now record all three producing-side points as author-fixed; the fixed-partial count is corrected to eight of ten. (Root cause: the sweep judges from pre-fetched verdict data — the triage verdict on ai-review-log#1042 was accurate when applied on 07-22 but stale by sweep time. Worth teaching the sweep to live-recheck "unresolved" claims it plans to cite; I'll fold that into CALIBRATION-SWEEP.md with the next workflow touch.)

Net: the PR now ships one prompt edit (the producing-side guard bullet, now 3-for-3 author-fixed) instead of two.