You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix(skills-contrib): remove hardcoded projects/ paths from QA skill docs
Replace literal `projects/<project-name>/...` path patterns with
project-relative descriptions. Durable skill docs should not encode
the transient project-workspace path convention; practitioners follow
the repo's project layout, so the skills need only say "the project
directory" and "a sibling manual-qa-reports/ directory".
Signed-off-by: Will Madden <madden@prisma.io>
Copy file name to clipboardExpand all lines: skills-contrib/drive-qa-plan/SKILL.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -216,7 +216,7 @@ The coverage map mirrors the TOC's AC mapping verbatim. Every AC appears exactly
216
216
217
217
### 7. Place the script in the canonical location
218
218
219
-
By convention this repo's project artefacts live at `projects/<project-name>/`. Manual-QA scripts go at `projects/<project-name>/manual-qa.md`. If the project has multiple milestones with materially different QA shape, prefer `projects/<project-name>/manual-qa/<milestone>.md` and link them from a slim index. Reports (one per QA pass) accumulate at `projects/<project-name>/manual-qa-reports/<YYYY-MM-DD>-<runner>.md` — owned by `drive-qa-run`.
219
+
Place the script in the project directory alongside the project spec, following the repo's project-workspace convention. The script is a project artefact that ships with the PR. If the project has multiple milestones with materially different QA shape, prefer a per-milestone file (e.g. `manual-qa/<milestone>.md`) and link them from a slim index. Reports (one per QA pass) accumulate in a sibling `manual-qa-reports/` directory — owned by `drive-qa-run`.
220
220
221
221
## Common Pitfalls
222
222
@@ -231,7 +231,7 @@ By convention this repo's project artefacts live at `projects/<project-name>/`.
231
231
9.**TOC written after the scenarios.** Symptom: the TOC row summaries don't match the scenario bodies, or rows are missing. Cause: TOC was bolted on at the end. Fix: write TOC first; scenarios are the expansion.
232
232
10.**No exploratory scenario.** Symptom: the script enumerates everything the author thought of; nothing invites the runner to look beyond. Fix: add at least one charter scenario with a time budget. Unknown unknowns are the whole point of charters.
233
233
11.**Pre-classifying severity in failure modes.** Symptom: "Red flag (blocker): the original bug returns". Fix: the script-author doesn't know the runtime context (release stage, blast radius, surrounding changes) at execution time. Enumerate the failure category; let the runner classify in the report.
234
-
12.**Using `wip/` for the script.** Symptom: the file lands at `wip/manual-qa.md`. Cause: the author treated it as scratch. Fix: manual-QA scripts are durable artefacts that ship alongside the spec — they belong in `projects/<project>/` (tracked) and travel with the PR. `wip/` is gitignored and the script would vanish.
234
+
12.**Using `wip/` for the script.** Symptom: the file lands at `wip/manual-qa.md`. Cause: the author treated it as scratch. Fix: manual-QA scripts are durable artefacts that ship alongside the spec — they belong in the project directory (tracked) and travel with the PR. `wip/` is gitignored and the script would vanish.
235
235
13.**Missing isolation tags.** Symptom: scenarios have no Isolation field; the runner falls back to running them serially. Fix: every scripted scenario carries one of `tmpdir` / `workspace` / `read-only` / `external`. Exploratory charters too. The runner can't parallelise what the script doesn't tag.
236
236
14.**Over-tagging isolation.** Symptom: every scenario tagged `workspace` "to be safe". Fix: the runner allocates a fresh `git worktree` per `workspace` scenario; tagging a tmpdir-only scenario as `workspace` doubles its setup cost for no benefit. Pick the strictest tag the scenario *actually* needs.
237
237
15.**Phantom dependencies.** Symptom: scenarios list "Preconditions: scenario 1 completed" defensively, with no actual data flow. Cause: the author confused "the script lists scenario 1 first" with "scenario N consumes scenario 1's output". Fix: only declare a dependency if the scenario reads an artefact the prerequisite produced. False dependencies serialise the DAG for no reason.
@@ -251,7 +251,7 @@ By convention this repo's project artefacts live at `projects/<project-name>/`.
251
251
252
252
## Checklist
253
253
254
-
-[ ] Script lives at `projects/<project-name>/manual-qa.md` (or per-milestone equivalent), not `wip/`.
254
+
-[ ] Script lives in the project directory (not `wip/`).
255
255
-[ ] First ~50 lines give a reader the full scope: frame paragraph + "Out of scope of this script" + spec/plan/PR links + TOC table.
256
256
-[ ] TOC table has one row per scenario with: number, verb-phrase title, one-line claim, **Isolation tag**, AC IDs covered.
257
257
-[ ] Every scripted scenario has all required sections: What you're proving, Covers, **Isolation**, **Oracle**, **Preconditions**, Steps, What you should see, Failure modes (+ Restore if the scenario mutates state outside its isolation context).
Copy file name to clipboardExpand all lines: skills-contrib/drive-qa-run/SKILL.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -54,12 +54,12 @@ Two cross-cutting rules:
54
54
55
55
## The report skeleton
56
56
57
-
Reports live at `projects/<project-name>/manual-qa-reports/<YYYY-MM-DD>-<runner-handle>.md`. One report per run. Reports accumulate (never overwrite) so the QA history is auditable.
57
+
Reports live in a `manual-qa-reports/` directory in the project workspace, named `<YYYY-MM-DD>-<runner-handle>.md`. One report per run. Reports accumulate (never overwrite) so the QA history is auditable.
> **Script:**`projects/<project>/manual-qa.md` (commit `<sha>` at run time)
62
+
> **Script:**`manual-qa.md` (commit `<sha>` at run time)
63
63
> **Runner:** <nameorLLMsessionid>
64
64
> **Environment:** <OS, Node version, branch + commit, any other relevant facts>
65
65
> **Started / finished:** <ISOtimestamps>
@@ -241,7 +241,7 @@ When you observe a failure-mode match or a mismatch with "What you should see":
241
241
5. Fill in the **Coverage outcome** table by walking the script's "Sign-off coverage map" row-by-row. Each AC inherits its result from the worst-severity finding in any of the scenarios that covered it (Blocker > High > Follow-up > Pass). N/A rows stay N/A.
242
242
6. Write **Suggested follow-ups** that cross-reference the disposition map. Each bullet should name the finding ID and its disposition (so a reader can see at a glance whether the bullet is "file this ticket" or "verify this post-merge").
243
243
7. Set the **finished** timestamp.
244
-
8. Save the report to `projects/<project-name>/manual-qa-reports/<YYYY-MM-DD>-<runner>.md`. Add the artefacts directory at the sibling `manual-qa-reports/artefacts/F-N/` paths.
244
+
8. Save the report to the project's `manual-qa-reports/<YYYY-MM-DD>-<runner>.md`. Add the artefacts directory at the sibling `manual-qa-reports/artefacts/F-N/` paths.
245
245
246
246
### 6. Hand off
247
247
@@ -268,7 +268,7 @@ Surface the report to the orchestrator / implementer / spec-author. The hand-off
268
268
8.**Letting the exploratory scenario blow its budget.** Symptom: runner is on minute 90 of a 30-minute charter, has filed 5 follow-ups, hasn't started the final scripted scenario. Fix: respect the time budget; remaining ideas go in "Suggested follow-ups".
269
269
9.**Filing findings against the script when they belong to the system.** Symptom: "F-3 — script said `migration plan` is a no-op but actually proposed an operation". Cause: runner attributed an observed-vs-expected mismatch to script staleness instead of a behavioural regression. Fix: only when you've verified the script *is* stale (e.g. the spec changed) does the finding belong to the script. If the script reflects current spec and reality diverges, the finding belongs to the system.
270
270
10.**Confabulating cleanup.** Symptom: report says "restored to clean state" but `git status` was never actually run. Fix: paste the literal `git status` output (or its short form) into the report at the end of each state-mutating scenario, even when clean. Evidence over claims.
271
-
11.**Using `wip/` for the report.** Symptom: report lands at `wip/qa-report.md`. Cause: runner treated it as ephemeral. Fix: reports are durable audit trail. They belong at `projects/<project>/manual-qa-reports/`. `wip/` is gitignored and the report would vanish.
271
+
11.**Using `wip/` for the report.** Symptom: report lands at `wip/qa-report.md`. Cause: runner treated it as ephemeral. Fix: reports are durable audit trail. They belong in the project's `manual-qa-reports/` directory. `wip/` is gitignored and the report would vanish.
272
272
12.**Serialising scenarios out of caution.** Symptom: runner dispatches scenarios one at a time even though they're tagged `tmpdir` or `read-only`. Cause: runner ignored the isolation tags. Fix: trust the tags. The cost of a mis-trusted tag is one false positive in the next QA round; the cost of pessimistic serialisation is every QA round taking 5x longer than it should. If the script genuinely under-tags, file that as a 📝 Follow-up against `drive-qa-plan` and run with the cautious schedule *this round only*.
273
273
13.**Running `workspace` scenarios in the live workspace.** Symptom: runner skips the `git worktree add` step and runs the planted-file scenario directly in the project root. Cause: runner conflated "the scenario tagged `workspace`" with "execute against the user's checkout". Fix: `workspace` means "needs its own working tree", which the runner manufactures via `git worktree`. The user's actual checkout stays clean throughout the run.
274
274
14.**Confusing isolation contexts in artefact paths.** Symptom: `manual-qa-reports/artefacts/F-3/` references a path under a worktree that's been removed by the time someone reads the report. Fix: copy artefacts into the durable `artefacts/F-N/` directory *before* tearing down the source worktree or tmpdir. Worktree paths are ephemeral; artefact directories are durable.
@@ -310,6 +310,6 @@ Surface the report to the orchestrator / implementer / spec-author. The hand-off
310
310
-[ ] Coverage outcome table walks every AC the script's coverage map enumerates; each AC inherits the worst-severity finding from its covering scenarios.
311
311
-[ ] Verdict in header keys off the disposition map: ✅ Pass only when every finding is ✅ resolved, 🔍 Triage required when any finding awaits orchestrator confirmation, ❌ Fail when any finding has a 🔧 fix-in-PR disposition or the runner observed an in-flight blocker.
312
312
-[ ] "✅ Pass-with-follow-ups" never appears as a verdict — that wording was retired (see Common Pitfalls #17).
313
-
-[ ] Report saved to `projects/<project>/manual-qa-reports/<YYYY-MM-DD>-<runner>.md`, artefacts under sibling `artefacts/F-N/`.
313
+
-[ ] Report saved to the project's `manual-qa-reports/<YYYY-MM-DD>-<runner>.md`, artefacts under sibling `artefacts/F-N/`.
314
314
-[ ] Worktrees torn down with `git worktree remove --force`; tmpdirs cleaned up; user's checkout left clean.
315
315
-[ ] Did NOT edit the script in place; script-quality issues filed as 📝 Follow-up findings instead.
0 commit comments