The learning
Measured 2026-09-09 while driving d-morrison/macros#87 (fixing \v0/\v1 \def collisions in macros.qmd).
A commit message said the fix was "verified through pandoc -t latex". An adversarial-reviewer subagent dispatched to check the PR ran its own counter-test: appended $\v0$ $\v1$ $\v{x}$ to a document with no macro definitions, ran quarto pandoc -t latex, and observed every token pass through unexpanded. It concluded pandoc never expands TeX macros in math mode at all, so the stated verification could not possibly have discriminated a working macro file from a broken one, and filed the claim as unsubstantiated.
The reviewer's measurement was real and its reasoning from it was valid. It simply omitted a precondition: pandoc's latex_macros extension expands macros defined in the same document. The reviewer tested the null case (no definitions) and read that as the general case.
Re-measuring with the precondition restored produces the actual discriminator (independently re-verified against the real d-morrison/macros before/after branches, and again by the @claude reviewer on the resulting PR against three separate pandoc builds):
|
\v0 |
\v1 |
\v{x} |
| no definitions present |
\v0 |
\v1 |
\v{x} |
macros.qmd before |
\v0 |
\tilde{1} |
\v{x} |
macros.qmd after |
\tilde{0} |
\tilde{1} |
\tilde{x} |
The reviewer measured the top row. The middle row is the bug's signature (only \v1 expands, from a surviving delimited \def\v1). The bottom row is the fix.
(Correction 2026-09-09: this issue originally showed \v{x} as absent in the top row, matching the original PR body's table. Both were wrong -- \v{x} passes through unexpanded, identically to \v0, exactly the same shape of mistake this entry itself is about. Caught by the @claude reviewer on #3468, who independently reproduced all three rows against the real repository across three pandoc builds.)
The transferable point
- A reviewer's finding can be backed by a real command and still be wrong, because the command omitted a precondition the original claim depended on.
dont-take-my-word-for-it applies to reviewers too, not just to humans and documents -- an accepted or rebutted finding both require re-measuring; "the reviewer ran a command" is not the same as "the reviewer ran the command that could show the claim false".
- The fix for a vague verification claim is not to win the argument in a PR comment where it dies with the thread. It is to put the discriminating measurement -- including the null row that shows what a non-discriminating test looks like -- into the durable artifact (the commit message, the PR body). The original message ("verified through
pandoc -t latex") was true but unfalsifiable-looking, which is what invited a plausible wrong finding; the amended message with the three-row table both rebutted the finding and left the next reader unable to repeat the reviewer's mistake.
Where this belongs
Composes with shared/principles/dont-take-my-word-for-it.md, shared/workflow/address-every-comment.md (the Rebut disposition), shared/workflow/quotable-findings.md, and shared/workflow/verify-the-right-artifact.md (the reviewer verified an adjacent artifact -- a document without the definitions -- rather than the one the claim was about).
Filed as the tracking issue for a same-day UMS PR (#3468) adding a new section to verify-the-right-artifact.md ("A reviewer's counter-measurement needs the same check the claim it rebuts would have needed"), rather than a new fragment -- that file already carries the closest sibling section ("A measurement of the right artifact can still be scoped narrower than the claim made from it", also from a 2026-09-09 session on the same PR family, d-morrison/rme#1138).
Posted by Claude Code (AI agent) --- not written by a human.
The learning
Measured 2026-09-09 while driving d-morrison/macros#87 (fixing
\v0/\v1\defcollisions inmacros.qmd).A commit message said the fix was "verified through
pandoc -t latex". Anadversarial-reviewersubagent dispatched to check the PR ran its own counter-test: appended$\v0$ $\v1$ $\v{x}$to a document with no macro definitions, ranquarto pandoc -t latex, and observed every token pass through unexpanded. It concluded pandoc never expands TeX macros in math mode at all, so the stated verification could not possibly have discriminated a working macro file from a broken one, and filed the claim as unsubstantiated.The reviewer's measurement was real and its reasoning from it was valid. It simply omitted a precondition: pandoc's
latex_macrosextension expands macros defined in the same document. The reviewer tested the null case (no definitions) and read that as the general case.Re-measuring with the precondition restored produces the actual discriminator (independently re-verified against the real
d-morrison/macrosbefore/after branches, and again by the@claudereviewer on the resulting PR against three separate pandoc builds):\v0\v1\v{x}\v0\v1\v{x}macros.qmdbefore\v0\tilde{1}\v{x}macros.qmdafter\tilde{0}\tilde{1}\tilde{x}The reviewer measured the top row. The middle row is the bug's signature (only
\v1expands, from a surviving delimited\def\v1). The bottom row is the fix.(Correction 2026-09-09: this issue originally showed
\v{x}as absent in the top row, matching the original PR body's table. Both were wrong --\v{x}passes through unexpanded, identically to\v0, exactly the same shape of mistake this entry itself is about. Caught by the@claudereviewer on #3468, who independently reproduced all three rows against the real repository across three pandoc builds.)The transferable point
dont-take-my-word-for-itapplies to reviewers too, not just to humans and documents -- an accepted or rebutted finding both require re-measuring; "the reviewer ran a command" is not the same as "the reviewer ran the command that could show the claim false".pandoc -t latex") was true but unfalsifiable-looking, which is what invited a plausible wrong finding; the amended message with the three-row table both rebutted the finding and left the next reader unable to repeat the reviewer's mistake.Where this belongs
Composes with
shared/principles/dont-take-my-word-for-it.md,shared/workflow/address-every-comment.md(the Rebut disposition),shared/workflow/quotable-findings.md, andshared/workflow/verify-the-right-artifact.md(the reviewer verified an adjacent artifact -- a document without the definitions -- rather than the one the claim was about).Filed as the tracking issue for a same-day UMS PR (#3468) adding a new section to
verify-the-right-artifact.md("A reviewer's counter-measurement needs the same check the claim it rebuts would have needed"), rather than a new fragment -- that file already carries the closest sibling section ("A measurement of the right artifact can still be scoped narrower than the claim made from it", also from a 2026-09-09 session on the same PR family, d-morrison/rme#1138).Posted by Claude Code (AI agent) --- not written by a human.