Skip to content

feat: gh-aw intake automation — PR review, triage, and digest workflows (phase 1)#18737

Merged
pettinarip merged 3 commits into
devfrom
feat/gh-aw-intake-automation
Jul 14, 2026
Merged

feat: gh-aw intake automation — PR review, triage, and digest workflows (phase 1)#18737
pettinarip merged 3 commits into
devfrom
feat/gh-aw-intake-automation

Conversation

@pettinarip

@pettinarip pettinarip commented Jul 9, 2026

Copy link
Copy Markdown
Member

Description

Phase 1 of the gh-aw intake automation plan: four agentic workflows (plus gh-aw's generated maintenance workflow) to improve PR review throughput — the baseline to beat is time-to-first-review P75 = 6.32 days (measured over the last 12 months; the measurement script stays local and will re-measure after rollout).

Workflows

Workflow Trigger Output Rollout state
pr-reviewer same-repo non-draft PR opened/ready lane-aware verdict-first comment + routing labels live
backlog-sweeper weekdays ~06:00 first-pass reviews of the 5 oldest unreviewed PRs (including fork PRs) + evidence-based recommend close labels live, never closes
issue-triager issue opened/reopened + 6-hourly backfill labels only — never comments live
team-digest weekdays ~07:00 Discord digest: merge-ready / review queue / SLA breaches / close queue / unanswered external issues live

Fork (external) PRs are handled by the backlog sweeper, not an event-driven reviewer: fork-triggered pull_request runs never receive repository secrets on public repos, so an agentic workflow can't start its engine there. The sweeper runs on a schedule (full secrets) and sweeps the whole open queue, forks included.

All agents run read-only (Claude engine, strict: true); every write goes through whitelisted safe-outputs with per-run caps (labels from fixed allowlists, Discord posts with mentions suppressed). Actions and containers are SHA-pinned.

Validated with a dry-run simulation

All four prompts were exercised read-only against the live repo queue (real PRs and issues, nothing posted) and tuned from the findings:

  • team-digest: explicit team-handle allowlist for the external-issues section, comments-field clarification, tighter section caps so the digest fits Discord comfortably
  • backlog-sweeper: skips PRs a maintainer commented on within the last 14 days, so it never follows on the heels of a manual sweep
  • issue-triager: may strip template-applied product labels (wallet 👛, product 🧰, event 📅, quiz 📚) from clearly-junk submissions

Before merge

  • Create the recommend close label
  • Add DISCORD_INTAKE_WEBHOOK_URL repo secret
  • ANTHROPIC_API_KEY repo secret (already present)
  • Team review of the workflow prompts

After merge (workflows only fire from the default branch)

  • Watch the reviewer and sweeper comments for ~2 weeks, tune prompts

@netlify

netlify Bot commented Jul 9, 2026

Copy link
Copy Markdown

Deploy Preview for ethereumorg ready!

Name Link
🔨 Latest commit 33cb320
🔍 Latest deploy log https://app.netlify.com/projects/ethereumorg/deploys/6a55fe07e313c20008986755
😎 Deploy Preview https://deploy-preview-18737.ethereum.it
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
7 paths audited
Performance: 60 (🟢 up 4 from production)
Accessibility: 95 (no change from production)
Best Practices: 100 (no change from production)
SEO: 98 (no change from production)
PWA: 60 (no change from production)
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions github-actions Bot added dependencies 📦 Changes related to project dependencies tooling 🔧 Changes related to tooling of the project labels Jul 9, 2026
@pettinarip
pettinarip force-pushed the feat/gh-aw-intake-automation branch from d7097c3 to c62b16c Compare July 10, 2026 17:38
@pettinarip
pettinarip marked this pull request as ready for review July 10, 2026 17:39
@pettinarip
pettinarip requested a review from wackerow as a code owner July 10, 2026 17:39
…ws (phase 1)

Four agentic workflows (plus gh-aw's generated maintenance workflow) to
improve intake throughput; baseline to beat is time-to-first-review
P75 = 6.32 days.

- pr-reviewer: lane-aware, verdict-first first-pass review of same-repo
  PRs with routing labels
- backlog-sweeper: weekday sweep of the 8 oldest unreviewed PRs (forks
  included) with evidence-based recommend-close labels; never closes
- issue-triager: labels-only classification of new issues; never comments
- team-digest: weekday Discord digest of the PR/issue queue

All agents run read-only (strict: true); writes go through allowlisted
safe-outputs with per-run caps. Actions and containers are SHA-pinned.
Prompts tuned via a read-only dry-run simulation against the live queue.
@pettinarip
pettinarip force-pushed the feat/gh-aw-intake-automation branch from c62b16c to 5fa168a Compare July 10, 2026 17:53
@pettinarip
pettinarip marked this pull request as draft July 10, 2026 17:56
@pettinarip
pettinarip marked this pull request as ready for review July 10, 2026 17:56
@github-actions

Copy link
Copy Markdown
Contributor

🔎 First-pass review — ✅ Looks mergeable

Lane: tooling. Phase 1 of the gh-aw intake automation: four agentic workflow prompts (pr-reviewer, backlog-sweeper, issue-triager, team-digest) plus a shared review core and gh-aw's generated maintenance/lock files. All engines run read-only (strict: true, network: defaults), every write goes through capped safe-outputs, and the Discord job suppresses mentions (allowed_mentions.parse: []). Generated *.lock.yml and agentics-maintenance.yml are SHA-pinned and correctly marked linguist-generated merge=ours in .gitattributes. No blocking issues — routing to human review of the prompts, which is the remaining pre-merge checklist item.

Analysis

Checked

  • Label allowlists resolve — the main gh-aw failure mode is an add-labels allowed: entry that doesn't exist in the repo (gh-aw applies labels but never creates them). Sampled across all four allowlists — recommend close, Status: Stale, dev required, design, config ⚙️, needs review 👀, needs triage 📥, duplicate 🐥🐥, a11y ♿️, proposal 🤔, awaiting changes, wallet 👛 — all exist. ✅
  • Untrusted-data guardrails present in each prompt; safe-outputs caps sane (reviewer add-comment max 1 + hide-older-comments; sweeper max 8; triager labels-only, never comments).
  • First-pass review marker in the shared core matches the sweeper's dedupe check.
  • CI: no checks have run yet (status pending, 0 checks) — nothing to diagnose.

Non-blocking observations

  • Sweeper selection ("skip PRs already carrying a First-pass review comment" / maintainer-commented-in-14-days) and the digest's Ready to merge / Recommend close sections depend on per-PR comment lookups, but the pre-fetched open-prs.json only includes ...reviewDecision,headRefName — no comments. The agent can still fetch them via the GitHub toolset, so this works, just at extra API cost per run. Worth confirming that's acceptable against the 15-min sweeper timeout with a full 8-PR sweep.
  • This PR carries dependencies 📦, but it touches no package.json/pnpm-lock.yaml — only gh-aw's actions-lock.json. Not really a deps change; tooling 🔧 alone fits.
  • .gitattributes ends without a trailing newline (diff shows \ No newline at end of file). Cosmetic.

Generated by PR Reviewer (team) for #18737 · 108.2 AIC · ⌖ 28.5 AIC · ⊞ 5K ·

- backlog-sweeper: 8 -> 5 PRs per run (comment cap 5, label cap 15) to
  fit the 15-minute timeout with per-PR comment/diff lookups
- pr-reviewer + sweeper: scoped remove-labels safe output (type labels
  only) so stale path-labeler labels that survive force-pushes get
  corrected; routing/status labels stay human-only
- .gitattributes: add trailing newline

@myelinated-wackerow myelinated-wackerow left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pettinarip Nice work — the security posture is tight: SHA-pinned actions and containers, read-only per-job permissions, firewall egress allowlist, ANTHROPIC_API_KEY/MCP token excluded from the agent container and secrets redacted from safe-outputs, and fork PRs correctly routed to the scheduled sweeper rather than pull_request_target. The compiled pr-reviewer gate is even stricter than the source (checks head.repo.id == repository_id alongside full_name). Approving.

Three non-blocking clean-up items for the tuning window:

  1. team-digest.md is missing the "treat as untrusted data" guardrail that pr-review-core.md and issue-triager.md both carry. It's the one prompt that ingests attacker-controllable text (PR/issue titles + bodies) and emits to an external sink (Discord). Mentions are already suppressed (allowed_mentions: {parse: []}) and it's an internal channel, so low severity — but worth adding the same one-liner for consistency.

  2. The PR description says the sweeper reviews "the 8 oldest", but the prompt caps at 5 (add-comment max: 5, "5 oldest by createdAt", "Do not exceed 5 reviewed PRs per run"). Just a stale number in the description.

  3. .gitattributes merge=ours on *.lock.yml is a no-op unless a [merge "ours"] driver = true is registered in git config (it isn't a built-in driver). Harmless — conflicts just fall back to manual — but the intended auto-keep-ours won't happen without the driver.

None of these block merge.

- Claude Opus / @wackerow

@pettinarip

Copy link
Copy Markdown
Member Author

Addressed the review nits (thanks @wackerow):

  1. Added the untrusted-data guardrail to team-digest.md (33cb320) — same wording as the other prompts, lock file recompiled.
  2. Fixed the stale "8 oldest" → "5 oldest" in the PR description.
  3. On merge=ours: you're right that it's inert without a registered driver, but gh aw compile writes that .gitattributes line itself (I removed it and the next compile re-added it), so I'm leaving it as-is — conflicts on lock files just resolve manually, and the correct resolution is always "recompile" anyway.

@pettinarip
pettinarip merged commit 77418ff into dev Jul 14, 2026
15 checks passed
@pettinarip
pettinarip deleted the feat/gh-aw-intake-automation branch July 14, 2026 10:50
@claude claude Bot mentioned this pull request Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies 📦 Changes related to project dependencies needs review 👀 tooling 🔧 Changes related to tooling of the project

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants