What happened
On PR #2853, the review agent posted a REQUEST_CHANGES review at 16:17 UTC with 3 medium findings including [submodule-trust] — a legitimate security concern about submodules: true cloning a third-party repo in CI. The author pushed 3 follow-up commits (16:55–17:18), which caused the REQUEST_CHANGES to be dismissed. The final review run (run 28535929661) completed successfully at 17:46 UTC but posted only a status comment ('Finished Review · Success') with no review body or findings. When rh-hemartin approved on Jul 2, the only agent review visible was the dismissed one — effectively invisible in GitHub's review UI. The [submodule-trust] finding was never addressed.
What could go better
The final review run appears to have suppressed its findings, likely because they matched the previous run's findings (deduplication logic). However, since the previous review was DISMISSED, those findings were no longer visible to human reviewers. This created a gap: the review agent correctly identified a security concern, ran successfully on the final commit, but produced no visible output for the human to act on. Confidence: medium-high — I cannot confirm the suppression reason without reading the agent's JSONL trace, but the observable behavior (successful review with no posted body after a dismissed prior review) is clear from the PR timeline. Existing issues #2816 (persistent finding cache) and #2677 (surface unresolved count in status comment) are related but neither addresses this specific interaction: re-review should detect that its prior formal review was DISMISSED and re-post persistent findings rather than treating them as already-visible.
Proposed change
In the review agent's post-processing logic (the post-script or review skill that decides whether to post findings), add a check: before suppressing findings as duplicates of a prior run, verify the prior review's GitHub review state. If the prior review was DISMISSED or if no active (non-dismissed) review from the bot exists on the PR, post findings even if they match a previous run. This ensures findings remain visible when GitHub's review dismissal mechanism removes the original review. The check could use gh api repos/{owner}/{repo}/pulls/{pr}/reviews filtered to the bot user, looking for any review in CHANGES_REQUESTED state that is not DISMISSED. If none exist and the current run has findings, post them.
Validation criteria
On the next 5 PRs where: (a) the review agent posts REQUEST_CHANGES, (b) the author pushes new commits causing the review to be dismissed, and (c) the re-review finds persistent findings — the re-review should post a new review body with findings rather than only a status comment. Verify by checking that no PR reaches human approval with zero visible (non-dismissed) agent reviews when the agent has active findings.
Generated by retro agent from #2853
What happened
On PR #2853, the review agent posted a
REQUEST_CHANGESreview at 16:17 UTC with 3 medium findings including[submodule-trust]— a legitimate security concern aboutsubmodules: truecloning a third-party repo in CI. The author pushed 3 follow-up commits (16:55–17:18), which caused theREQUEST_CHANGESto be dismissed. The final review run (run 28535929661) completed successfully at 17:46 UTC but posted only a status comment ('Finished Review · Success') with no review body or findings. Whenrh-hemartinapproved on Jul 2, the only agent review visible was the dismissed one — effectively invisible in GitHub's review UI. The[submodule-trust]finding was never addressed.What could go better
The final review run appears to have suppressed its findings, likely because they matched the previous run's findings (deduplication logic). However, since the previous review was DISMISSED, those findings were no longer visible to human reviewers. This created a gap: the review agent correctly identified a security concern, ran successfully on the final commit, but produced no visible output for the human to act on. Confidence: medium-high — I cannot confirm the suppression reason without reading the agent's JSONL trace, but the observable behavior (successful review with no posted body after a dismissed prior review) is clear from the PR timeline. Existing issues #2816 (persistent finding cache) and #2677 (surface unresolved count in status comment) are related but neither addresses this specific interaction: re-review should detect that its prior formal review was DISMISSED and re-post persistent findings rather than treating them as already-visible.
Proposed change
In the review agent's post-processing logic (the post-script or review skill that decides whether to post findings), add a check: before suppressing findings as duplicates of a prior run, verify the prior review's GitHub review state. If the prior review was DISMISSED or if no active (non-dismissed) review from the bot exists on the PR, post findings even if they match a previous run. This ensures findings remain visible when GitHub's review dismissal mechanism removes the original review. The check could use
gh api repos/{owner}/{repo}/pulls/{pr}/reviewsfiltered to the bot user, looking for any review inCHANGES_REQUESTEDstate that is notDISMISSED. If none exist and the current run has findings, post them.Validation criteria
On the next 5 PRs where: (a) the review agent posts REQUEST_CHANGES, (b) the author pushes new commits causing the review to be dismissed, and (c) the re-review finds persistent findings — the re-review should post a new review body with findings rather than only a status comment. Verify by checking that no PR reaches human approval with zero visible (non-dismissed) agent reviews when the agent has active findings.
Generated by retro agent from #2853