Add review-community-package skill#11057
Draft
fnune wants to merge 2 commits into
Draft
Conversation
A maintainer-facing skill for reviewing PRs that add an entry to community-packages/package-list.json. Codifies the deep review that historically lives in maintainer heads: schema URL resolves, schema name matches the pulumi-<name> repo convention, docs files are present upstream, SDKs are published, package is a general-purpose abstraction rather than a one-product deployment, etc. Six-phase workflow with cost-tiered short-circuiting: triage, mechanical checks, judgement, heavy verification (regenerate metadata locally; install in a throwaway container per the contributor's own installation-configuration.md), report, act. Phases 1-3 always run; heavy phases auto-skip when earlier ones surface a Fail. Three outcomes: close-and-continue (matches the maintainer-finishing pattern from past chains like #9743 to #10010; required when CI cannot run on a fork), ask-for-submitter-changes, close-with-reasoning. Hard safety rules: never run the provider install outside a container, never perform a state-mutating gh or git action without explicit user confirmation of the full command and message body, bail out for internal-contributor PRs and defer to /pr-review. Writes a structured markdown artifact per PR-sha to \${XDG_CACHE_HOME:-\$HOME/.cache}/pulumi-registry-reviews/pr-<num>/ with YAML frontmatter for machine-readability and a clickable-link body for human re-reading. ## Test plan - Trial-ran against #11056 (atensecurity/pulumi-thoth). Skill caught that docs/_index.md and docs/installation-configuration.md are missing from the upstream provider repo (at v0.1.11 and on main), and that TS/Python/Go SDKs are not published. Recommended ask-for-submitter-changes with concrete asks. Artifacts at ~/.cache/pulumi-registry-reviews/pr-11056/ - The skill never executed any state-mutating gh or git action during the trial runs, as required.
Three changes after a real run against #11056 surfaced friction: 1. Comment-style brevity rules for all generated PR comments, review bodies, and close messages: under 150 words, no em-dashes, no opening preamble or closing flourish, one-line asks with the why in a parenthetical or link. The previous default produced ~400-word walls of text with em-dashes throughout, against the global style the maintainer has set. 2. SDK presence probe is now driven by the schema's language map, not an assumption that all four (TS/Python/Go/C#) are required. A declared language with no published SDK is Fail; a published SDK that lags the provider version by more than one minor is also Fail (the schema promises an API the SDK does not expose); undeclared languages are not probed. 3. Obvious packaging-config bugs in published SDK names (doubled tokens like Foo.Pulumi.Bar.Bar from a misconfigured tfgen rootNamespace) are now an explicit Warn pattern, not just "best effort, varies." ## Test plan - Re-ran the comment redraft from the #11056 review with the new brevity rules: 120 words vs the previous 400, no em-dashes, identical action items. - Confirmed against atensecurity/pulumi-thoth that its schema declares all four languages, so the SDK probe still flags the missing npm publish and the lagging PyPI publish; behaviour on that PR is unchanged but the skill no longer over-asks for providers that intentionally ship fewer SDKs.
Contributor
|
Your site preview for commit e2cec08 is ready! 🎉 http://registry--origin-pr-11057-e2cec08a.s3-website.us-west-2.amazonaws.com/registry. |
Contributor
|
Your site preview for commit 0171abd is ready! 🎉 http://registry--origin-pr-11057-0171abd0.s3-website.us-west-2.amazonaws.com/registry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A maintainer-facing skill for reviewing PRs that add an entry to
community-packages/package-list.json. Codifies the deep review that
historically lives in maintainer heads: schema URL resolves, schema
name matches the pulumi- repo convention, docs files are present
upstream, SDKs are published, package is a general-purpose
abstraction rather than a one-product deployment, etc.
Six-phase workflow with cost-tiered short-circuiting: triage,
mechanical checks, judgement, heavy verification (regenerate metadata
locally; install in a throwaway container per the contributor's own
installation-configuration.md), report, act. Phases 1-3 always run;
heavy phases auto-skip when earlier ones surface a Fail.
Three outcomes: close-and-continue (matches the maintainer-finishing
pattern from past chains like #9743 to #10010; required when CI cannot
run on a fork), ask-for-submitter-changes, close-with-reasoning.
Hard safety rules: never run the provider install outside a container,
never perform a state-mutating gh or git action without explicit user
confirmation of the full command and message body, bail out for
internal-contributor PRs and defer to /pr-review.
Writes a structured markdown artifact per PR-sha to
${XDG_CACHE_HOME:-$HOME/.cache}/pulumi-registry-reviews/pr-/
with YAML frontmatter for machine-readability and a clickable-link
body for human re-reading.
Test plan
that docs/_index.md and docs/installation-configuration.md are
missing from the upstream provider repo (at v0.1.11 and on main),
and that TS/Python/Go SDKs are not published. Recommended
ask-for-submitter-changes with concrete asks. Artifacts at
~/.cache/pulumi-registry-reviews/pr-11056/
the trial runs, as required.