Skip to content

fix(code-reviewer): require tool verification before asserting facts about external systems#614

Open
Jamie-BitFlight wants to merge 1 commit intoobra:mainfrom
Jamie-BitFlight:fix/require-verification-before-assertion
Open

fix(code-reviewer): require tool verification before asserting facts about external systems#614
Jamie-BitFlight wants to merge 1 commit intoobra:mainfrom
Jamie-BitFlight:fix/require-verification-before-assertion

Conversation

@Jamie-BitFlight
Copy link

Training data is a source of questions to ask, not answers to assert. The code-reviewer was asserting GitHub Actions version numbers from training data without verifying them, causing correct versions to be downgraded. Adds a Verification Before Assertion section requiring a tool call before any claim about versions or external system state.

Copilot AI review requested due to automatic review settings March 4, 2026 16:03
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds explicit guidance to the code-reviewer prompt to prevent unverified assertions about external systems (especially version existence/currentness), reducing regressions caused by relying on training data.

Changes:

  • Introduces a “Verification Before Assertion” section requiring tool-based verification for external-system/version claims.
  • Adds a small lookup table mapping common claim types (GitHub Actions/PyPI/npm/nonexistence claims) to suggested verification commands.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.


| Claim type | Required verification |
|---|---|
| GitHub Actions version | `gh release list --repo <owner>/<action> --limit 1` |
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gh release list --repo <owner>/<action> --limit 1 is not a reliable way to determine the latest available GitHub Action version: many action repos don’t publish GitHub Releases (only tags), --limit 1 returns the most recently created release (not necessarily highest semver), and it may pick a prerelease. Consider updating the guidance to verify via tags (e.g., gh api repos/<owner>/<repo>/tags --paginate / git ls-remote --tags) and/or the action’s Marketplace page, and clarify how to choose a stable major tag (vN) vs a pinned patch (vN.N.N).

Suggested change
| GitHub Actions version | `gh release list --repo <owner>/<action> --limit 1` |
| GitHub Actions version | Use tags: `gh api repos/<owner>/<action>/tags --paginate` or `git ls-remote --tags https://github.com/<owner>/<action>.git`, and/or check the Action’s Marketplace page. Choose an appropriate tag: stable major (`vN`) or pinned patch (`vN.N.N`). |

Copilot uses AI. Check for mistakes.
@obra obra added bug Something isn't working claude-code Claude Code (Anthropic CLI) issues labels Mar 4, 2026
IgorTavcar added a commit to IgorTavcar/superpowers that referenced this pull request Mar 9, 2026
…bra#614)

Adds verification lookup table for version-specific claims that require
tool checks before assertion.

Upstream PR: obra#614

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working claude-code Claude Code (Anthropic CLI) issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants