Skip to content

Commit 7232c85

Browse files
danielgtaylordanielgtaylor
authored
Merge pull request #364 from rest-sh/codex/release-qa-skill
Add release QA skill
2 parents 346bad6 + 76bc9f9 commit 7232c85

5 files changed

Lines changed: 593 additions & 0 deletions

File tree

‎.agents/skills/rsh-release-qa/SKILL.md‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
---
2+
name: rsh-release-qa
3+
description: Run Restish release readiness QA from main using recent commits, built-in historical risk patterns, docs, and full release gates; report reproducible issues without patching product code
4+
---
5+
6+
# Restish Release QA
7+
8+
Run an evidence-first release readiness pass for Restish. The goal is high
9+
confidence and clear reproducible findings, not implementation work.
10+
11+
## Use This Skill For
12+
13+
- Preparing a Restish release or release candidate from `main`.
14+
- Turning recent commits, docs changes, and historical Restish QA risk patterns
15+
into a concrete release strategy.
16+
- Running the full pre-release gate, targeted regression probes, built-binary
17+
smoke checks, and docs validation.
18+
- Reporting blockers, non-blocking risks, unrun checks, and reproduction steps.
19+
20+
## Do Not Use This Skill For
21+
22+
- Patching release-blocking bugs.
23+
- Refactoring, test authoring, or docs edits during QA.
24+
- Making release tags, publishing artifacts, or changing release workflows
25+
unless the user explicitly asks for those separate actions.
26+
27+
If a bug is found, document it thoroughly and suggest the smallest credible fix.
28+
Do not edit product code. Only write `TODO.md` when the user has explicitly
29+
allowed local follow-up notes for the current run. `TODO.md` is local-only and
30+
must never be staged or committed.
31+
32+
## Core Workflow
33+
34+
1. Confirm the exact release candidate:
35+
- Check `git status --short`, current branch, and `git rev-parse HEAD`.
36+
- Fetch `origin` when currentness matters.
37+
- Prefer testing the fetched `origin/main` or an exact candidate commit. If
38+
the user's working branch should not move, create a temporary worktree.
39+
2. Determine the diff surface:
40+
- Find the last release tag or user-provided baseline.
41+
- Read recent first-parent merges and non-merge commits.
42+
- Classify touched areas: generated OpenAPI commands, auth/config,
43+
request pipeline, pagination, plugins, output, docs, packaging, CI.
44+
3. Apply the built-in risk map:
45+
- Read `references/release-risk-map.md` for the distilled release matrix.
46+
- Read `references/findings-mining.md` for historical finding classes and
47+
safe repro patterns.
48+
- Match recent commits to the risk areas and choose targeted probes.
49+
4. Build a run-specific strategy:
50+
- Always include the full release gate unless the user explicitly narrows it.
51+
- Add targeted probes for recent commits and relevant findings.
52+
- Include user-facing docs checks for docs or command-surface changes.
53+
5. Run the checks:
54+
- Create a fresh temporary run directory and use isolated config/cache paths
55+
for built-binary smokes.
56+
- Prefer harmless local servers, `--rsh-print`, `--rsh-server` overrides,
57+
localhost failure targets, and read-only public specs.
58+
- Never use real user credentials or destructive provider calls.
59+
6. Report release readiness:
60+
- Lead with blockers or "no release blockers found".
61+
- Include exact candidate commit, commands run, failures, environment issues,
62+
targeted probes, and residual risk.
63+
- For each issue, include severity, expected behavior, actual behavior,
64+
reproduction commands, impact, and a suggested fix direction.
65+
66+
## Full Gate
67+
68+
Use the release gate reference for the canonical command set:
69+
`references/release-gates.md`.
70+
71+
Default full gate:
72+
73+
- `go test ./...`
74+
- `go test -tags=integration ./...`
75+
- Targeted `go test -race` over risk-bearing packages.
76+
- Build `restish` and first-party plugin binaries.
77+
- `go run ./cmd/restish-docgen --check`
78+
- `npm --prefix site ci` with a disposable npm cache when needed.
79+
- `npm --prefix site run build` for the docs site.
80+
- Built-binary smoke checks for help, version, redaction, and at least one
81+
generated OpenAPI command surface when the release touches CLI/OpenAPI/auth.
82+
83+
Use per-run `GOCACHE` and `GOPATH` paths under the temporary run directory when
84+
the default Go caches are not writable in the sandbox. Report any resulting
85+
network, toolchain-download, or module-download limitation instead of changing
86+
dependencies during release QA.
87+
88+
## Findings Mining
89+
90+
Use `references/findings-mining.md` for distilled historical issue classes,
91+
triage rules, and repro selection.
92+
93+
High or security findings with open or uncertain status are release blockers
94+
until verified fixed or explicitly deferred by the user. Medium findings are
95+
blockers when they affect common workflows, release packaging, data integrity,
96+
credential leakage, or newly changed code.
97+
98+
## Targeted Probe Rules
99+
100+
- Generated OpenAPI changes: connect or load a small public/local spec, inspect
101+
help, run `doctor api`, and test request construction without provider writes.
102+
- Auth/config changes: check noninteractive setup, missing env diagnostics,
103+
redaction, profile precedence, and trusted project config boundaries.
104+
- Request pipeline changes: check content negotiation, redirects, retries,
105+
TLS flags, server overrides, and network-error diagnostics.
106+
- Pagination changes: check page parameters, Link headers, metadata filters,
107+
max page/item limits, and strict `items_path` behavior.
108+
- Plugin changes: build first-party plugins and run relevant integration/race
109+
tests; check subprocess lifecycle risks.
110+
- Docs/command-surface changes: run docgen drift and Hugo build; smoke the
111+
affected help output from a built binary.
112+
- Security/redaction changes: test verbose and non-verbose errors with URL
113+
userinfo, credential-looking query params, headers, JSON bodies, forms, and
114+
generated credentials.
115+
116+
## Reporting Template
117+
118+
Use this shape unless the user asks for something else:
119+
120+
```text
121+
Release candidate: <commit> on <branch/ref>
122+
Baseline: <tag or commit>
123+
124+
Readiness: ready / not ready / ready with noted risks
125+
126+
Findings:
127+
- P1/P2/... <title>
128+
Impact:
129+
Repro:
130+
Expected:
131+
Actual:
132+
Suggested fix direction:
133+
134+
Checks run:
135+
- <command> - passed/failed/skipped
136+
137+
Targeted probes:
138+
- <scenario> - passed/failed
139+
140+
Residual risk:
141+
- <unrun check, environmental limitation, or deferred finding>
142+
```
143+
144+
If every required check passes and no blocking issue is found, say so clearly.
145+
Do not bury skipped checks or environmental limitations.

‎.agents/skills/rsh-release-qa/agents/openai.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
interface:
2+
display_name: "Restish Release QA"
3+
short_description: "Run evidence-first Restish release checks"
4+
default_prompt: "Use $rsh-release-qa to assess Restish release readiness from main with full tests, docs validation, historical-risk probes, and reproducible findings."

‎.agents/skills/rsh-release-qa/references/findings-mining.md‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
# Historical Finding Classes
2+
3+
This reference distills prior Restish release findings into reusable QA
4+
heuristics. Use it to choose targeted probes without depending on local
5+
planning files.
6+
7+
## Triage Rules
8+
9+
- Credential leakage, data loss, corrupted config/cache state, or common
10+
generated-command breakage is release-blocking until verified fixed or
11+
explicitly deferred by the user.
12+
- Medium issues are blockers when they affect common workflows, credentials,
13+
generated commands, config/cache integrity, release packaging, or code changed
14+
since the last release.
15+
- Low issues become targeted smoke checks when a nearby subsystem changed.
16+
- Help/example bugs are release risks when they teach users to send the wrong
17+
value, leak secret-looking values, or make common copy/paste examples fail.
18+
- Rejected historical findings are still useful for documentation and
19+
"working as intended" checks.
20+
21+
## Historical Blocker And Near-Blocker Classes
22+
23+
### Auth And Redaction
24+
25+
- Optional anonymous OpenAPI security alternatives (`{}` plus a credential)
26+
must remain callable when optional env-backed credentials are unresolved.
27+
Ready credentials should still be preferred; required-auth operations should
28+
still fail on missing env.
29+
- OpenAPI `mutualTLS` should be satisfied by resolved TLS client cert/key or TLS
30+
signer transport settings, then fail later with normal TLS diagnostics if the
31+
files/plugins are bad.
32+
- Noninteractive `api connect` auth setup should reject unknown credential IDs,
33+
wrong-case credential IDs, unknown fields, and unused credential-scoped
34+
prompts.
35+
- Verbose body redaction must catch `password`, `password2`,
36+
`confirm_password`, `client_secret`, `secret`, `token`, and similar fields,
37+
while preserving ordinary numeric fields such as `max_tokens` and
38+
`token_budget`.
39+
- Network-error diagnostics must redact URL userinfo, credential-looking query
40+
params from the original URL, secrets added by `--rsh-query`/env/defaults,
41+
and generated query apiKey credentials in both the top-level error prefix and
42+
nested transport URL.
43+
- Verbose request traces should redact configured credentials in headers,
44+
query, cookies, JSON bodies, form bodies, response headers, plugin stderr, and
45+
redirect diagnostics.
46+
47+
### Generated OpenAPI Commands
48+
49+
- Duplicate path/operation header parameters with the same HTTP header name,
50+
including case variants, should merge before generated flag creation and
51+
should not skip the whole operation.
52+
- Generated fallback command names should remain short, stable, and traceable
53+
when `operationId` is absent or when `operation_base` is configured.
54+
- Specs with `description` fields that look like `$ref` objects must not be
55+
treated as schema refs.
56+
- OpenAPI extension effects (`x-cli-*`) should be visible and should not report
57+
nested effects for ignored entries.
58+
- Stale local spec files should refresh raw spec and generated operation caches
59+
before generated command registration or execution.
60+
- Concurrent `api connect` runs should preserve config entries and generated
61+
spec/operation caches for every successful connect.
62+
63+
### Schema Help And Examples
64+
65+
- OpenAPI 3.1 conditionals inside `allOf` should not hide direct object
66+
properties in generated help. Conditional required branches should render as
67+
useful constraints.
68+
- Schema help should keep effective flag types consistent after invalid
69+
enum/type fallback, such as string enum values declared under an integer
70+
schema.
71+
- Generated body examples should normalize string-valued boolean/integer/number
72+
defaults/examples to real JSON scalar values when unambiguous.
73+
- JSON nulls in `enum`, `default`, `example`, and `const` should render as
74+
`null`, not Go-ish `<nil>`, and generated examples should avoid contradicting
75+
null-only constraints.
76+
- Mixed-type root `enum`/`const` values should not make generated body examples
77+
contradict an object schema.
78+
- Generated examples should redact real secret-like explicit examples but avoid
79+
over-redacting ordinary non-secret token counters or logprob fields.
80+
- Generated shorthand examples should not turn URI/URL/URN strings into `@file`
81+
control syntax.
82+
- Generated shorthand examples for arrays of objects should be shell-safe and
83+
structurally copyable; otherwise prefer JSON file input.
84+
- Generated root help should handle absent, short, multiline Markdown, and
85+
README-length `info.description` values without overwhelming command help.
86+
87+
### Query, Path, And Pagination
88+
89+
- `deepObject` and `form` object query help should show syntax Restish accepts,
90+
and runtime serialization should match each OpenAPI style.
91+
- JSON-content query parameters should encode one JSON query value matching the
92+
schema. Whole-array parent values should work; repeated array-of-object flags
93+
must not silently become arrays of JSON strings.
94+
- String-or-array query flags should support multiple values or clearly point
95+
users to an accepted `--rsh-query` workaround.
96+
- Placeholder/free-form query parameter names should either be modeled
97+
deliberately or documented as requiring `--rsh-query`.
98+
- Path parameters with slashes, commas, percent signs, spaces, question marks,
99+
and already-escaped input should encode consistently. Pathful
100+
`--rsh-server` overrides are a known place to check for double encoding.
101+
- Page-param pagination should honor the effective query parameter, preserve
102+
strict `items_path` behavior, respect max-page/max-item limits, and handle
103+
metadata filters.
104+
- Link-header pagination should handle relative next URLs, quoted parameters
105+
with commas, malformed targets, unsupported schemes, and standalone `links`
106+
output.
107+
108+
### Media Types And Bodies
109+
110+
- Generated XML request-body `@file` input should send raw XML bytes with the
111+
declared XML media type.
112+
- Generated NDJSON request-body `@file` input should send raw newline-delimited
113+
bytes with the declared NDJSON media type, not a JSON-encoded string.
114+
- Wildcard or protocol-specific raw binary media types such as `*/*`,
115+
`application/*`, and `application/offset+octet-stream` should send raw bytes
116+
for `string format: binary` bodies while preserving the declared
117+
`Content-Type`.
118+
- Multipart examples should be usable or should explain `@file` syntax clearly.
119+
Check missing files, unreadable files, literal `@`, repeated file parts,
120+
binary parts, and per-part encoding metadata.
121+
- Operations advertising multiple request or response `content` entries should
122+
choose sensible defaults and respect `--rsh-content-type`, explicit `Accept`,
123+
vendor `+json`, XML/text alternatives, and strict provider expectations.
124+
125+
### Built-Ins, Plugins, And Docs
126+
127+
- Built-in commands should reject unsupported output formats clearly and should
128+
avoid advertising inherited output/filter flags they do not support.
129+
- Unknown subcommands under command groups such as `cache` and `plugin` should
130+
fail, not silently exit 0.
131+
- Explicit `--rsh-header` and `--rsh-query` should have documented precedence
132+
against env/default values and should not unexpectedly drop unrelated entries
133+
unless that is intentional.
134+
- First-party command plugins should expose long help consistently for root
135+
`--help`, `-h`, `help`, and subcommand `--help` forms.
136+
- Plugin protocol changes should remain additive or bump/check compatibility.
137+
Subprocesses that start must be waited on; timeout/error paths must close
138+
pipes and avoid goroutine/process leaks.
139+
- Generated docs regions, README install guidance, plugin docs, release
140+
packaging docs, and Hugo site build should stay in sync with command behavior.
141+
142+
## Safe Repro Patterns
143+
144+
Use these shapes when turning a historical class into a release probe.
145+
146+
- Local fixture: write a small OpenAPI file under `t.TempDir()` in tests or
147+
under `${TMPDIR:-/tmp}` for manual QA; connect it with isolated config/cache.
148+
- Public spec: connect into isolated config/cache with fake credentials, then
149+
inspect help, `doctor api`, `api auth inspect`, `--rsh-generate-body`, or
150+
request construction only.
151+
- Request construction: use `--rsh-print H`, `--rsh-print B`, verbose output,
152+
`--rsh-server https://httpbin.org/anything`, or a localhost failure target.
153+
- Redaction: intentionally use fake secrets with unique strings and grep the
154+
resulting stderr/stdout for leaks.
155+
- Network diagnostics: use `127.0.0.1:9`, `--rsh-timeout 200ms`, and
156+
`--rsh-retry 0` so failures are quick and local.
157+
- Provider drift: compare generated command output against generic requests or
158+
curl only with safe read-only endpoints.
159+
160+
## Issue Report Shape
161+
162+
For every new problem, capture:
163+
164+
- Candidate commit and platform.
165+
- Exact commands and environment variables, with fake secrets only.
166+
- Expected behavior.
167+
- Actual behavior, including relevant stderr/stdout excerpts.
168+
- Whether the issue is a release blocker.
169+
- Severity and impact.
170+
- Related historical class from this reference.
171+
- Suggested fix direction, without patching code during QA.
172+
173+
## Residual Risk Notes
174+
175+
Call out:
176+
177+
- Checks skipped because a tool was unavailable, such as local GoReleaser.
178+
- Network-dependent probes that were not run or were inconclusive.
179+
- Public APIs that failed for provider/network reasons rather than Restish
180+
behavior.
181+
- Dirty worktrees, non-candidate branches, or tests run against a commit that
182+
differs from the requested release candidate.

0 commit comments

Comments
 (0)