fix(nc-review): make the verdict file the deliverable, retry once - #1211
Conversation
Run 6 produced a thorough, accurate analysis and threw it away. The agent narrated nineteen numbered findings to stdout — including a correct walk through umask interaction with mode 0600 — then finished without ever calling write_file. It treated the chat as the output. The instruction to write the file was buried mid-prompt behind the reading and analysis instructions. It is now last, stated as the whole deliverable, and says plainly that chat output is discarded and never reaches a human. The rubric's Output section says the same, because that is the one instruction whose failure costs an entire run. Also adds a single retry, scoped to exactly this failure: if verdict.json is missing or empty after the first attempt, run once more with a terse write-the-file prompt. A genuine agent error still falls through to the safe comment rather than being retried blindly. Roughly three minutes. Separately, a failed run no longer leaves a stale label. Applying none on failure was right, but #1184 has carried agent:clean through two failed re-reviews, which reads as "an agent looked at this and was happy" when nothing of the kind happened. The failure path now clears all three agent:* labels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PZY52ePXLjwG9TaQgq2cHT
No changeset foundThis PR does not add a changeset, so it will not appear in the changelog or trigger a release. If the change is user-facing, add one: pnpm changesetPick a bump (patch / minor / major) and write the changelog entry in our usual voice ("Added X... Thanks to @you. Closes #123."), then commit the generated If this PR is docs-only or a chore that needs no release note, you can ignore this - or run |
nc-review: comments — 2 nitsThe PR addresses a demonstrated failure mode of the review agent (running analysis in chat instead of writing the verdict file) by reordering the prompt to put the write instruction last, and adds a one-shot retry plus a stale-label cleanup in the same failure path. Both code changes are correct and tight; the prompt and rubric edits match the surrounding tone. One small inconsistency between the description and the code is worth flagging — see finding.
🔴 blocking · 🟠 a reviewer would ask for a change · ⚪ optional Automated code review — correctness, security, design, tests, plus duplicates and scope. A human still decides; this is not a substitute for review and is not exhaustive. The required status checks separately cover lint, formatting, types, unused dependencies, the test suite and the build. This bot never merges. Maintainers can rerun with |
Run 6 produced a thorough and accurate analysis, then threw it away.
The agent worked through nineteen numbered findings on stdout — including a correct walk through how
umaskinteracts withmode: 0o600— and then finished without ever callingwrite_file. It treated the chat as the output. Zero-byte verdict, safe fallback comment, run wasted.Cause
The instruction to write the file was buried mid-prompt, behind the reading and analysis instructions. With the deeper rubric there is now a lot of analysis to do, so by the time the model finished thinking, "write the file" was a long way back.
Fix
The write instruction is now last and framed as the entire deliverable:
The rubric's Output section says the same thing, since this is the one instruction whose failure costs a whole run.
Retry
Adds a single retry, scoped to exactly this failure: if
verdict.jsonis missing or empty after the first attempt, run once more with a terse write-the-file prompt. Verified the guard fires on missing and empty, and not on valid JSON.Deliberately not a general retry — a genuine agent error still falls through to the safe comment rather than burning another three minutes.
Stale labels
Unrelated bug spotted while reading the output. Applying no label on failure was the right call, but leaving the previous run's label was not: #1184 has carried
agent:cleanthrough two failed re-reviews, which reads as "an agent looked at this and was happy" when nothing of the kind happened.The failure path now clears all three
agent:*labels.