Skip to content

Recurrence of #710: a review that appends a self-correction posts only the correction, and #808's last-heading rule is the likely cause #850

Description

@d-morrison

Symptom, which is #710's verbatim

UCD-SERG/serocalculator#685, run 34292812731, reviewed commit afb0502.

The posted review comment is 1694 characters and consists of two sections: ### Correction on pipeline status and ### Verdict. It twice cites analysis that is not in it:

Code content: verified clean via independent line-by-line checks (detailed in my review above) — no defects found in the diff itself.

Ready for merge (content assessment of the diff itself, based on the extensive verification detailed above).

There is nothing above it. I enumerated every comment surface on that PR:

Surface Result
Issue comments from the review bot one, the 1694-char comment itself
Formal reviews (/pulls/685/reviews) 0
Inline review comments (/pulls/685/comments) 0

So the verdict rests on content no reader can see. That is #710's exact failure mode, and #710 is closed.

Why the review body probably existed

Artifact sizes from the same run:

claude-review-payload-34292812731-1              1,400 bytes
claude-review-execution-34292812731-1-attempt1 143,111 bytes

A 1.4 KB payload against a 143 KB execution transcript. The payload is about the size of what got posted; the transcript is not. That is consistent with a substantial review having been produced and then dropped during extraction — though I cannot confirm it, because I have no way to download either artifact from outside the run. Treat this paragraph as a strong hypothesis, not a finding.

The likely mechanism, and why the earlier fixes point at it

CLAUDE.md records the two rules that govern the posted span:

A reviewer that writes a full review with a ### Verdict, then appends a short correction that also carries a ### Verdict, is textually indistinguishable from a redraft. #808's rule then starts the span at the correction and drops the review — which is #710's outcome, reintroduced by #710's own follow-up fix.

This run's shape fits that exactly: the surviving text is a correction about pipeline status plus a verdict, i.e. the appended tail, and it refers back to the review that a redraft-assuming rule would have discarded.

The distinguishing test, for whoever picks this up: read claude-review-execution-...-attempt1 and check whether an earlier assistant block carries an authored ### Verdict heading. If it does, this is #808 over-applying and the fix is to tell a correction from a redraft — a redraft restates the whole review, a correction is short and refers back to it. If it does not, the model genuinely wrote only this and the "detailed above" citations are fabricated, which is a different bug in the brief.

I have not run that test and cannot from here, so the two branches are both live.

Interaction with #849

The same run's require-clean-verdict failed, which I filed separately as #849 (a verdict that disclaims being a fully-clean claim scores needs-more-work). These are independent — #849 is the classifier reading a negated positive, this is the review body going missing — but they compound: a consumer sees a red gate and an approving one-paragraph verdict, and neither surface says the review is absent.

Notably review / require-review passed on this run, so the delivery gate believes a review was delivered. If the body is being dropped, that gate is measuring the wrong thing.

Impact

Every consumer of claude-code-review.yml. A reviewer that self-corrects — which the brief's own verdict-semantics instruction encourages, per #849 — loses its review. The failure is quiet: the comment reads as a deliberate short approval rather than as a truncation, and only a reader who notices "detailed above" has no referent will catch it.


Filed by Claude Code (AI agent) --- not written by a human. Found while reading the verdict on a PR of my own; I would have accepted "Ready for merge" had the dangling reference not been checked.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions