feat(ci): add PR readiness helper bot#16231
Draft
Joibel wants to merge 6 commits into
Draft
Conversation
Adds a standalone workflow_run-triggered bot that maintains a single sticky comment on PRs guiding contributors through contributor-fixable CI failures (lint, codegen, UI, build, docs, title, feature files, DCO) and PR template compliance (judged via GitHub Models, advisory and fail-closed). Blocking issues also move the PR to draft; the bot never undrafts. PRs from OWNERS members and bots are skipped, and unit/E2E test results are deliberately ignored as too flaky to be a readiness signal. Runs entirely from the default branch with minimal permissions, never executes PR-head code, and treats PR-controlled strings as data only. Ships with DRY_RUN=true: decisions render to the job step summary until the comment behaviour has been validated on real PRs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alan Clucas <alan@clucas.org>
Step summaries are only visible in the Actions UI; emit the signal states, AI verdict and comment/draft decision (and the rendered comment when applicable) to the job log as well so runs can be inspected via the API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alan Clucas <alan@clucas.org>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alan Clucas <alan@clucas.org>
Joibel
changed the title
Joibel
added
the
area/contributing
Fork test flight showed gpt-4o-mini hallucinating template violations on compliant PRs: claiming "#TODO" was present in descriptions with no Fixes line at all, and rejecting AI sections that genuinely declared tool use because they did not say "None". Since the verdict gates draft conversion, false non-compliance is the expensive direction. Rewrite the prompt to be compliant-by-default, require verbatim evidence before flagging, and spell out the Fixes-line-absent and AI-declared-use cases explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alan Clucas <alan@clucas.org>
All bot PRs (dependabot, renovate, the cherry-pick app) bypass the readiness helper entirely — the author gate runs before any GitHub Models call, so bot PRs never spend inference quota. Add the actual bot logins to the exemption test so this cannot regress silently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Alan Clucas <alan@clucas.org>
Live fork testing confirmed the default Actions token cannot run
convertPullRequestToDraft ("Resource not accessible by integration").
Mint an app installation token (same create-github-app-token pattern
as the cherry-pick bot) when PR_READINESS_APP_ID /
PR_READINESS_APP_PRIVATE_KEY secrets are configured, and use it only
for the draft mutation. Without the secrets the bot still comments and
logs a warning instead of drafting. The app needs only Pull requests:
Read & write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Alan Clucas <alan@clucas.org>
Member
Author
|
I should convert this to ts rather than js |
MasonM
reviewed
Jun 9, 2026
|
|
||
| - name: Resolve PR and classify checks | ||
| id: prepare | ||
| uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 |
Member
There was a problem hiding this comment.
FYI: You can shell out to node to run TypeScript, since it natively supports running TypeScript via type stripping now: https://nodejs.org/learn/typescript/run-natively
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Maintainer burden is high partly because PRs sit in a not-ready-for-review state for reasons the contributor could fix themselves — lint/codegen/docs failures, non-conventional titles, missing DCO sign-off, missing
.features/files, and unfilled PR descriptions. Unit/E2E tests are too flaky to be a readiness signal, but the other checks are deterministic and have documented fixes. This bot turns those signals into one actionable comment for the contributor, so maintainers only see PRs once they're genuinely reviewable.Modifications
Adds a standalone
workflow_run-triggered workflow (pr-readiness.yaml) that fires when CI / Docs / PR Title Check / PR Feature Check complete on a PR, plus its logic under.github/pr-readiness/:make pre-commit -Betc.) and a log link; edits to a ✅ all-clear when resolved; never posts on PRs that never had an issue.actions/ai-inference,models: read) — advisory, schema-validated, fail-closed on errors/rate limits, output sanitised (mentions neutralised, issue-closing keywords redacted).OWNERSand from bots. Ignores unit/E2E results entirely.permissions: {}with minimal job grants, PR-controlled strings handled as data, actions SHA-pinned. Fork PRs are matched by head SHA (workflow_run.pull_requestsis empty for forks).DRY_RUN: "true"— decisions render to the job step summary only, for a observation period before going live.Maintainers tune guidance/coverage in
checks.config.json(pure data); design and rollout notes in.github/pr-readiness/README.md.Verification
node --test '.github/pr-readiness/test/*.test.js'actionlintclean; all actions pinned to full commit SHAs.Documentation
.github/pr-readiness/README.mddocuments behaviour, the security model, dry-run rollout, and maintenance (check-name drift detection, workflow-name coupling). Not user-facing, so nodocs/changes.AI
This PR was developed with Claude Code (design research, implementation, tests, and this description), reviewed and directed by a human maintainer.