fix(code-reviewer): require tool verification before asserting facts about external systems#614
Conversation
There was a problem hiding this comment.
Pull request overview
Adds explicit guidance to the code-reviewer prompt to prevent unverified assertions about external systems (especially version existence/currentness), reducing regressions caused by relying on training data.
Changes:
- Introduces a “Verification Before Assertion” section requiring tool-based verification for external-system/version claims.
- Adds a small lookup table mapping common claim types (GitHub Actions/PyPI/npm/nonexistence claims) to suggested verification commands.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| | Claim type | Required verification | | ||
| |---|---| | ||
| | GitHub Actions version | `gh release list --repo <owner>/<action> --limit 1` | |
There was a problem hiding this comment.
gh release list --repo <owner>/<action> --limit 1 is not a reliable way to determine the latest available GitHub Action version: many action repos don’t publish GitHub Releases (only tags), --limit 1 returns the most recently created release (not necessarily highest semver), and it may pick a prerelease. Consider updating the guidance to verify via tags (e.g., gh api repos/<owner>/<repo>/tags --paginate / git ls-remote --tags) and/or the action’s Marketplace page, and clarify how to choose a stable major tag (vN) vs a pinned patch (vN.N.N).
| | GitHub Actions version | `gh release list --repo <owner>/<action> --limit 1` | | |
| | GitHub Actions version | Use tags: `gh api repos/<owner>/<action>/tags --paginate` or `git ls-remote --tags https://github.com/<owner>/<action>.git`, and/or check the Action’s Marketplace page. Choose an appropriate tag: stable major (`vN`) or pinned patch (`vN.N.N`). | |
Training data is a source of questions to ask, not answers to assert. The code-reviewer was asserting GitHub Actions version numbers from training data without verifying them, causing correct versions to be downgraded. Adds a Verification Before Assertion section requiring a tool call before any claim about versions or external system state.