GitHub PR Triage Agent

An eve agent that triages pull requests. When a PR is opened it reads the diff, applies labels, and posts one structured triage comment (summary, risk, review focus, labels, and suggested reviewers), with labels, risk signals, and reviewer routing driven by a single customizable ruleset, triage.yml.

Vercel runs its own agents on eve in production. This template packages that same pattern as a PR-triage agent you can deploy in your repo in minutes.

It runs as a GitHub App webhook: GitHub delivers pull_request events to the deployed agent, which verifies the signature, mints an installation token, reads the auto-injected diff, and replies on the PR.

---

Local development

The project is built with the eve CLI (the eve binary, a dev dependency). Before deploying, verify it builds locally from a fresh clone:

npm install     # install dependencies
eve dev         # run the agent locally (dev server + terminal UI)
eve build       # compile the agent and host output, to verify a clean build

eve dev is handy for iterating; eve build is the verification step. You don't strictly need either to ship: vercel deploy builds remotely on Vercel, so a local build is just for confirming things compile before you push.

---

Deploy

This button does the initial deploy only (the one-click equivalent of vercel link + vercel deploy in step 1); the steps below are still required: disable Deployment Protection, create the GitHub App, set the env vars, and install the App.

The steps below are the verified happy path. There's a deliberate ordering wrinkle: the GitHub App's webhook URL needs the deployment URL, but the deployment doesn't need the App's credentials to exist first. So deploy first, then create the App, then set the credentials and redeploy.

Prerequisites

• A Vercel account and the Vercel CLI (npm i -g vercel), or use npx vercel.
• A model credential. Easiest is the Vercel AI Gateway: once the project is linked to Vercel, gateway model ids (e.g. anthropic/claude-sonnet-4.6) authenticate via OIDC with no key to manage. (To call a provider directly instead, set e.g. ANTHROPIC_API_KEY.)

1. Deploy to get a public URL

Use the Deploy with Vercel button at the top of this section, or do it manually:

vercel link        # link this directory to a Vercel project
vercel deploy --prod

Note the production URL it prints (e.g. https://<your-app>.vercel.app). It has no GitHub credentials yet; that's expected for this first deploy.

Host where you can make the deployment public. A GitHub webhook is unauthenticated from Vercel's point of view, so if the project sits behind Vercel Deployment Protection (Vercel Authentication / SSO), every delivery is rejected with 401 before it reaches the agent, and nothing happens. Pick a team/project where disabling protection is acceptable (for an OSS template, that means a public-by-default team; some teams enforce protection org-wide and you'll need an admin or a different team).

2. Make the deployment public

Disable Deployment Protection for the project:

• Dashboard: Project → Settings → Deployment Protection → Vercel Authentication → Off (or "Only Preview Deployments" to keep previews protected while production stays public).

Verify it's public. Health should return 200 with no SSO redirect:

curl -s -o /dev/null -w '%{http_code}\n' https://<your-app>.vercel.app/eve/v1/health   # -> 200

A quick way to confirm the webhook route is wired (and that the app, not the platform, is in charge of auth): an unsigned POST should get the app's own 401 unauthorized (plain text), not a Vercel SSO HTML page:

curl -s -X POST https://<your-app>.vercel.app/eve/v1/github -d '{}'   # -> "unauthorized"

That 401 is the agent's HMAC signature check (x-hub-signature-256, verified with your webhook secret) doing its job. Every real delivery from GitHub is signed and will pass.

3. Create the GitHub App

GitHub → Settings → Developer settings → GitHub Apps → New GitHub App:

• Webhook URL: https://<your-app>.vercel.app/eve/v1/github
• Webhook secret: generate one and keep it. You'll set the same value as the GITHUB_WEBHOOK_SECRET env var in step 5. The two must be identical, or HMAC verification fails and every delivery is rejected.

  openssl rand -hex 32

• Repository permissions:
  • Pull requests: Read & write: reads the diff, lists labels, applies labels, and posts the comment. On a PR, GitHub accepts this one permission for all of those.
  • Metadata: Read-only: the mandatory baseline (auto-selected).
  • (You do not need Issues or Contents. Contents: Read only becomes relevant if you later add sandbox-based features.)
• Subscribe to events: Pull request.

Create the App, then note the App ID and the App slug (from its URL, github.com/apps/<slug>).

4. Generate the private key

On the App's page, Generate a private key. A .pem downloads. Keep it; it's only shown once.

5. Set the env vars and redeploy

Set these on the Vercel project (Production): three required vars plus an optional slug. The names must match exactly, since they're what the channel reads:

Env var	Required?	Value
GITHUB_APP_ID	yes	the App ID (a number)
GITHUB_APP_PRIVATE_KEY	yes	the .pem contents
GITHUB_WEBHOOK_SECRET	yes	the webhook secret from step 3, which must be byte-for-byte identical to the value in the App's Webhook secret field, or HMAC verification fails and every delivery is rejected
GITHUB_APP_SLUG	optional	the App slug, only used for @mention/comment-driven turns, not the pull-request-opened trigger this template ships with, so you can skip it for v1

Set them like this. The webhook secret uses the interactive prompt so it never lands in your shell history; the private key is piped from the file so its newlines survive:

# App ID (not secret), inline is fine:
printf '%s' '<APP_ID>' | vercel env add GITHUB_APP_ID production

# Webhook secret: use the prompt so it never hits shell history:
vercel env add GITHUB_WEBHOOK_SECRET production
# paste the secret when prompted

# Private key: pipe from the file so newlines survive:
vercel env add GITHUB_APP_PRIVATE_KEY production < /path/to/your-app.private-key.pem

# Optional (only for @mention/comment turns):
# vercel env add GITHUB_APP_SLUG production

The agent normalizes the key whether the newlines arrive real or escaped (\n), so either storage form works.

Redeploy so the variables take effect:

vercel deploy --prod

6. Install the App and create labels

• Install the App on the target repo: https://github.com/apps/<slug>/installations/new → select the repo.
• Create the labels the ruleset uses. triage.yml leads with GitHub's built-in defaults (bug, documentation, enhancement), which every repo already has, so labeling works on the first PR with no setup. The two extra labels (dependencies, refactor) are not GitHub defaults; create them if you want them applied (until then apply_labels simply skips them):

  gh label create dependencies -R <owner>/<repo> --color 0366d6 --description "Adds, removes, or upgrades dependencies"
  gh label create refactor     -R <owner>/<repo> --color fbca04 --description "Restructures code without changing behavior"

7. Open a PR

Open a pull request with a real change. Within a few seconds you should see:

• A triage comment with: Summary, (Key changes, omitted for trivial PRs), Risk, Review focus, Labels, Suggested reviewers.
• The fitting labels applied (only labels that both fit the change and exist in the repo; a PR that fits no label gets none, which is fine because the comment still carries the triage).

The agent triages on PR open and does not re-run on later pushes to the same PR; re-triage on new commits is a v2 item.

What you will not see: an "eyes" reaction. The channel only reacts to a triggering comment, and an opened PR has none, so on PR-open the first visible signal is the comment itself, not a reaction. (The reaction does appear if you later wire up @mention/comment-triggered turns.)

Troubleshooting: if the first PR gets a short error comment (with an error id) instead of a triage, the agent likely couldn't reach a model. Confirm the Vercel AI Gateway is enabled for the project, or set a provider key (e.g. ANTHROPIC_API_KEY) as an env var and redeploy.

Trace: in Vercel → Observability → Agent Runs, the PR-open turn runs with no sandbox provisioned (verified): turn.started → model step → load_skill → apply_labels → message.completed, with no sandbox/checkout span. This is the turn.started override at work: a diff-only triage never pays for a sandbox.

---

Customize the triage rules

triage.yml is the one file you edit: the label catalog (name + when-to-apply), risk signals, and path→reviewer routing.

triage.yml is read at build time, not at runtime. Editing it has no effect on a running deployment until you rebuild and redeploy (vercel deploy). If you change a rule and see no change in behavior, that's why.