Skip to content

README.md

Latest commit

 

History

History
286 lines (225 loc) · 10.2 KB

README.md

File metadata and controls

286 lines (225 loc) · 10.2 KB
Raw

AI Code Review for Pull Requests

A relay plan template that reviews pull requests, posts GitHub reviews with line-level inline comments, auto-fixes findings, and pushes verified commits. Two LLM calls — all GitHub interaction is handled by bash scripts (zero tokens on mechanical work).

What it does

prepare → review (LLM) → post findings → fix (LLM) → push → verify → post summary
                       ↘ approve → post approval
  1. Gathers context — PR title, description, file summary, and diff.
  2. Reviews — One LLM call produces structured findings with severity, category, file, line, description, and suggestion.
  3. Dismisses stale reviews — Previous AI reviews are dismissed before posting a new one.
  4. Posts a GitHub review — APPROVE if clean, REQUEST_CHANGES if issues found. Each finding with a valid line number becomes a line-level inline comment on the diff. Findings with invalid or missing lines are included in the review body only.
  5. Fixes — A worker LLM addresses error and warning findings. It self-verifies by running the project's verification command.
  6. Pushes — Commits and pushes using GITHUB_TOKEN (no loop — pushes via GITHUB_TOKEN do not trigger new workflow runs).
  7. Verifies — Runs the verification command after push.
  8. Posts a summary — A PR comment reporting what was found and fixed. Dismisses the REQUEST_CHANGES review so it no longer blocks merge.

File structure

bundled/ci/
  pr-review.md              Plan template — topology and artifact contracts
  README.md                 This file
  scripts/
    lib.sh                  Shared functions (dismiss reviews, derive base branch)
    prepare.sh              Gather PR context into the context artifact
    post-approval.sh        Dismiss stale reviews, post APPROVE review
    post-findings.sh        Dismiss stale reviews, post REQUEST_CHANGES with inline comments
    push-fixes.sh           Commit and push fix changes
    post-summary.sh         Post success comment, dismiss stale reviews
    post-failure.sh         Post failure comment

The template is a pure topology definition — each command step is a one-liner that calls a script. The scripts are independently testable, syntax-highlighted, and lintable.

Setup

1. Copy the workflow

Copy .github/workflows/pr-review.yml into your repository:

name: PR Review

on:
  pull_request:
    types: [opened, synchronize, reopened]
    branches: [main]

permissions:
  contents: write
  pull-requests: write

jobs:
  review:
    name: Relay PR Review
    runs-on: ubuntu-latest
    if: github.event.pull_request.head.repo.full_name == github.repository
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install pi-relay
        id: relay
        run: |
          git clone --depth=1 https://github.com/benaiad/pi-relay.git /tmp/pi-relay
          cd /tmp/pi-relay && npm ci && npm run build
          echo "root=/tmp/pi-relay" >> "$GITHUB_OUTPUT"

      - name: Run PR review
        run: |
          node ${{ steps.relay.outputs.root }}/dist/cli/main.js \
            bundled/ci/pr-review.md \
            -e pr_number="${{ github.event.pull_request.number }}" \
            --model "$RELAY_MODEL" \
            --thinking "${RELAY_THINKING:-off}"
        env:
          RELAY_MODEL: ${{ secrets.RELAY_MODEL }}
          RELAY_THINKING: ${{ secrets.RELAY_THINKING }}
          RELAY_PLAN_DIR: ${{ steps.relay.outputs.root }}/bundled/ci
          GH_TOKEN: ${{ github.token }}
          # Add your provider's API key secret below. Name depends on provider:
          #   zai → ZAI_API_KEY, anthropic → ANTHROPIC_API_KEY, openai → OPENAI_API_KEY
          ZAI_API_KEY: ${{ secrets.ZAI_API_KEY }}

2. Configure secrets

Go to Settings > Secrets and variables > Actions in your repository and add these secrets:

Secret Required Example Purpose
RELAY_MODEL Yes zai/glm-5.1 Model for the LLM. Format: provider/model-name.
RELAY_THINKING No medium Thinking level: off, low, medium, high. Defaults to off.
Provider API key Yes (varies) API key for your model provider. Name depends on provider — see below.

Provider API key names:

Provider Secret name
zai ZAI_API_KEY
anthropic ANTHROPIC_API_KEY
openai OPENAI_API_KEY
google GOOGLE_API_KEY
deepseek DEEPSEEK_API_KEY

Add the provider API key secret, then replace ZAI_API_KEY in the workflow's env block with your provider's key name:

        env:
          RELAY_MODEL: ${{ secrets.RELAY_MODEL }}
          RELAY_THINKING: ${{ secrets.RELAY_THINKING }}
          RELAY_PLAN_DIR: ${{ steps.relay.outputs.root }}/bundled/ci
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # your provider
          GH_TOKEN: ${{ github.token }}

The model name and provider are stored as secrets so they are not visible in the workflow file, logs, or the repository.

3. Customize the verification command

The default verification command is npm run check. To change it, add the -e verify= parameter:

          node ${{ steps.relay.outputs.root }}/dist/cli/main.js \
            bundled/ci/pr-review.md \
            -e pr_number="${{ github.event.pull_request.number }}" \
            -e verify="cargo test && cargo clippy -- -D warnings" \
            --model "$RELAY_MODEL" \
            --thinking "${RELAY_THINKING:-off}"

Parameters

Parameter Default Description
pr_number (required) Pull request number. Passed from the workflow.
verify npm run check Shell command that must exit 0. Used for verification after fixes and by the worker for self-testing.
max_diff_lines 10000 Maximum diff lines in the context artifact. Larger values give the reviewer more context but use more tokens.
base_branch (auto-derived) Base branch the PR targets. Derived from the PR via gh pr view when empty. Set explicitly for local testing.

Environment variables

The scripts expect these environment variables at runtime:

Variable Set by Purpose
RELAY_PLAN_DIR Workflow Path to bundled/ci/. Scripts resolve siblings via $RELAY_PLAN_DIR/scripts/.
RELAY_INPUT Relay runtime Directory containing artifact files for reading.
RELAY_OUTPUT Relay runtime Directory for writing artifact files.
GITHUB_REPOSITORY GitHub Actions owner/repo — used in API calls.
GITHUB_SERVER_URL GitHub Actions Base URL — used in "View full run" links.
GITHUB_RUN_ID GitHub Actions Run ID — used in "View full run" links.
GH_TOKEN Workflow Authentication for gh CLI.

Triggers

The workflow triggers on:

  • opened — First review when the PR is created.
  • synchronize — Re-review on every new push to the PR branch.
  • reopened — Re-review when a closed PR is reopened.

Only PRs targeting main are reviewed. To review PRs targeting other branches, change the branches filter:

on:
  pull_request:
    types: [opened, synchronize, reopened]
    branches: [main, develop]

Fork PRs are skipped (they lack access to secrets).

How it works

The plan has 13 steps but only 2 LLM calls. Everything else is a script.

Approve path (1 LLM call):

prepare → review → post_approval → approved

Fix path (2 LLM calls):

prepare → review → post_findings → fix → push_fixes → verify → post_summary → fixed

Line-level inline comments

The reviewer outputs a line field for each finding. The post-findings script validates line numbers against the actual diff:

  1. Parses git diff -U0 with POSIX awk to extract valid file:line pairs (changed lines only).
  2. For each finding: if file:line is valid, it becomes a line-level inline comment. If the line is invalid, empty, or the file isn't in the diff, the finding is skipped as an inline comment — it remains visible in the review body.

The review always posts successfully regardless of line accuracy.

Stale review dismissal

Stale reviews are dismissed at two points:

  1. Before posting a new review — post-approval.sh and post-findings.sh dismiss all previous reviews by github-actions[bot] before posting. This prevents stale findings from coexisting with the fresh review.
  2. After successful fix + verify — post-summary.sh dismisses the REQUEST_CHANGES review it posted earlier, since the findings have been addressed. This unblocks merge.

Dismissal failures are silently ignored (|| true).

Loop prevention

Fix commits are pushed using GITHUB_TOKEN. GitHub's built-in protection prevents GITHUB_TOKEN-triggered events from starting new workflow runs. No infinite loops.

Never use a personal access token (PAT) for the push step. PATs bypass this protection.

Local testing

Run the plan locally against an existing PR:

RELAY_PLAN_DIR=./bundled/ci \
  node dist/cli/main.js \
  bundled/ci/pr-review.md \
  -e pr_number=42 \
  -e base_branch=main \
  --model zai/glm-5.1 \
  --thinking medium

Dry-run (no LLM calls, no API key needed):

node dist/cli/main.js \
  bundled/ci/pr-review.md \
  -e pr_number=1 \
  --model fake/model \
  --dry-run

Troubleshooting

Problem Cause Fix
$RELAY_PLAN_DIR/scripts/prepare.sh: not found RELAY_PLAN_DIR not set Add it to the workflow env (see setup step 1)
Model not found error RELAY_MODEL secret missing or misspelled Check Settings > Secrets
No API key error Provider API key secret not set Add the provider-specific secret
git push fails Missing contents: write permission Add the permission to the workflow
Review not posted Missing pull-requests: write permission Add the permission to the workflow
gh api 403 error GH_TOKEN not set in env Add GH_TOKEN: ${{ github.token }} to the env block