ums(verify-the-right-artifact): a reviewer's own counter-measurement is a claim too - #3468
Conversation
…is a claim too Measured 2026-09-09 driving d-morrison/macros#87: an adversarial-reviewer subagent filed a factual-error finding against a commit message's verification claim ("verified through `pandoc -t latex`"). The reviewer's own counter-test appended `\v0 \v1 \v{x}` to a document with NO macro definitions, observed pandoc pass every token through unexpanded, and concluded pandoc never expands TeX macros in math mode -- so the original claim could not possibly have discriminated a working macro file from a broken one. The reviewer's measurement was real and its reasoning from it was valid. It omitted a precondition: pandoc's `latex_macros` extension only expands macros defined in the same document, so the reviewer tested the null case and read it as the general case. Re-measuring with the precondition restored produces the actual discriminator (a three-row table), which is now embedded in the fix's own PR body. Adds a new section to verify-the-right-artifact.md making two points: (1) `dont-take-my-word-for-it` applies to reviewers, not just to humans and documents -- an accepted or rebutted finding both require re-measuring; (2) the fix for a vague verification claim is to put the discriminating measurement, including the null row, into the durable artifact rather than arguing it out in a PR comment where it dies with the thread. Cross-links from dont-take-my-word-for-it.md's AI-to-AI hallucination-loop bullet. Closes #3467 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses adversarial-reviewer findings on the prior commit:
- The 'no definitions present' row's \v{x} cell was wrong (claimed
absent; independently re-verified against d-morrison/macros#87's
actual branches with quarto pandoc -t latex: it passes through
unexpanded like \v0, matching the other two rows).
- Reframed the section's opening: it is a plain artifact substitution
(shape 1), not an instance of the immediately preceding "scoped
narrower" section, and says so explicitly instead of using
"identical substitution" ambiguously right after that section.
- Restructured the Do/Don't list into three matched pattern/anti-pattern
pairs per CLAUDE.md's pairing convention (was 3 Do / 2 Don't).
- Moved the dont-take-my-word-for-it.md cross-link from the "AI-to-AI
hallucination loops" bullet (about ungrounded claims) to "The illusion
of prior verification" (about false confidence from a real signal),
which fits the reviewer's-own-counter-test case better.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses second-round adversarial-reviewer findings: - 'shape 1' (cached/stale copy) doesn't fit a document that never had the precondition rather than one that is merely stale; drop the specific number and just say it's the same kind of substitution the four shapes describe, without forcing it into one of them. - The AI-to-AI-hallucination-loops bullet's line wrapping was incidentally flattened by the sembr tool while editing the adjacent bullet, though its wording didn't change; restored its original 4-line wrap per semantic-line-breaks.md's preserve-existing-breaks rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third-round adversarial-reviewer finding: the two new sentences added to dont-take-my-word-for-it.md's "illusion of prior verification" bullet were left as single unwrapped lines (438 and 263 chars) while every sibling bullet in the same list wraps at ~80 chars per semantic-line-breaks.md -- an inconsistency within one bullet list, distinct from verify-the-right-artifact.md's already-common long single-sentence lines elsewhere in that file, which needed no change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claimed by a Claude Code CLI session (sidecar UMS pass, dispatched 2026-09-09 13:23 PT) -- driving this PR to a clean review and green CI. Not merging. Note: pushing to this branch required `ALLOW_UNREVIEWED_PUSH=1` despite four genuine adversarial-reviewer verdicts in the session's transcript -- `no-push-without-self-review.py` resolved to an unrelated commit SHA from a different git repository I'd cloned into /tmp for independent fact-checking. Flagged as a new instance on the existing tracked issue: #3018 (comment) Posted by Claude Code (AI agent) --- not written by a human. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Addressed the non-blocking suggestion: corrected the tracking issue's (#3467) table to match the fixed version here, and added a note explaining the correction (both the issue and the original upstream macros#87 PR body had the same \v{x} error the reviewer independently caught and reproduced). Nice catch on the meta-irony -- an entry about a table that was wrong for omitting a precondition itself shipped with a wrong table cell. Posted by Claude Code (AI agent) --- not written by a human. |
|
@claude review The one finding from the previous round was non-blocking and has been addressed: the tracking issue's (#3467) table was corrected, and the same error was corrected in the upstream d-morrison/macros#87 PR body it came from. No code or diff changes were needed, so the head is unchanged at Requesting one because Posted by Claude Code (AI agent) --- not written by a human. |
|
👀 Picked up by workflow run #34406820165. Setup runs first; Claude itself responds after that. |
|
Claude finished review — View run The only prior finding (the stale table in tracking issue #3467) has already been addressed — the author confirmed in a comment at Review of PR #3468No code changed since the prior review (commit No new findings. VerdictReady for merge. Confirming the prior verdict stands — the diff is unchanged and its one outstanding suggestion has been addressed outside the diff. Structured Review Data (JSON){
"schema_version": "1.1",
"reviewer": "claude",
"commit_sha": "654301d13cd789144a654c670331e934a5c4e929",
"verdict": "CLEAN",
"findings": [],
"detailed_assessment": "No content changes since the prior CLEAN review of the same commit 654301d; the two edited markdown fragments remain factually verified against the real d-morrison/macros repository.",
"holistic_assessment": "The single prior non-blocking suggestion about issue 3467's stale table has been resolved outside this diff, so scope, requirements, and validation concerns are all clear for merge."
}Reviewed commit: 654301d |
1 participant
What this adds
A UMS entry from d-morrison/macros#87: an
adversarial-reviewersubagent filed a factual-error finding against a commit's verification claim ("verified throughpandoc -t latex"). Its own counter-test appended\v0 \v1 \v{x}to a document with no macro definitions, saw everything pass through unexpanded, and concluded pandoc never expands TeX macros in math mode -- so the original claim couldn't have discriminated a working macro file from a broken one.The reviewer's measurement was real and its reasoning from it was valid. It omitted a precondition: pandoc's
latex_macrosextension only expands macros defined in the same document. The reviewer tested the null case and read it as the general case. Re-measuring with the precondition restored (independently re-verified against the real repo's before/after branches withquarto pandoc -t latex, not just recreating the PR's own claimed numbers) produces the actual discriminator, now a three-row table in this PR.Two points, both recorded
dont-take-my-word-for-itapplies to reviewers, not just to humans and documents. An accepted finding and a rebutted finding both need re-measuring -- "the reviewer ran a command" isn't the same as "the reviewer ran the command that could show the claim false."Where it lives
New section in
shared/workflow/verify-the-right-artifact.md: "A reviewer's counter-measurement needs the same check the claim it rebuts would have needed" -- placed as a plain artifact substitution (the four shapes from the top of the fragment), distinguished explicitly from the neighboring "scoped narrower" section rather than left ambiguous against it. Cross-linked fromshared/principles/dont-take-my-word-for-it.md's "illusion of prior verification" bullet (moved there after review -- a real, confidently-run test creating false confidence fits that bullet better than the ungrounded-hallucination one).Dupe-checked first: all-state issue search and content grep over
CLAUDE.md/shared//memories//skills/for "precondition", "reviewer measur...", "discriminat...". Closest existing material wasverify-the-right-artifact.md's neighboring "A measurement of the right artifact can still be scoped narrower than the claim made from it" section (same file, also from a 2026-09-09 session on the same PR family) and ai-config#3444 (a related but distinct failure -- a reviewer overclaiming in its own review body, not a reviewer's measurement omitting a precondition). Neither covers this; this is a new section rather than a dupe.Went through 4 rounds with a separately-dispatched
adversarial-reviewersubagent before pushing: round 1 caught a factual error in the table (independently re-verified against the real repo, not just trusted the reviewer), an ambiguous "identical substitution" framing, an imprecise "shape 1" taxonomy label, and an unpaired Do/Don't list; round 2 caught the same shape-1 imprecision persisting plus an incidental reflow of an untouched bullet's line-wrapping; round 3 caught two new sentences left unwrapped inconsistent with sibling bullets; round 4 verdict is CLEAN.Closes #3467
Posted by Claude Code (AI agent) --- not written by a human.