Skip to content

Commit 5c4d981

Browse files
committed
fix(skills-contrib): remove hardcoded projects/ paths from QA skill docs
Replace literal `projects/<project-name>/...` path patterns with project-relative descriptions. Durable skill docs should not encode the transient project-workspace path convention; practitioners follow the repo's project layout, so the skills need only say "the project directory" and "a sibling manual-qa-reports/ directory". Signed-off-by: Will Madden <madden@prisma.io>
1 parent 7221730 commit 5c4d981

2 files changed

Lines changed: 8 additions & 8 deletions

File tree

skills-contrib/drive-qa-plan/SKILL.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -216,7 +216,7 @@ The coverage map mirrors the TOC's AC mapping verbatim. Every AC appears exactly
216216

217217
### 7. Place the script in the canonical location
218218

219-
By convention this repo's project artefacts live at `projects/<project-name>/`. Manual-QA scripts go at `projects/<project-name>/manual-qa.md`. If the project has multiple milestones with materially different QA shape, prefer `projects/<project-name>/manual-qa/<milestone>.md` and link them from a slim index. Reports (one per QA pass) accumulate at `projects/<project-name>/manual-qa-reports/<YYYY-MM-DD>-<runner>.md` — owned by `drive-qa-run`.
219+
Place the script in the project directory alongside the project spec, following the repo's project-workspace convention. The script is a project artefact that ships with the PR. If the project has multiple milestones with materially different QA shape, prefer a per-milestone file (e.g. `manual-qa/<milestone>.md`) and link them from a slim index. Reports (one per QA pass) accumulate in a sibling `manual-qa-reports/` directory — owned by `drive-qa-run`.
220220

221221
## Common Pitfalls
222222

@@ -231,7 +231,7 @@ By convention this repo's project artefacts live at `projects/<project-name>/`.
231231
9. **TOC written after the scenarios.** Symptom: the TOC row summaries don't match the scenario bodies, or rows are missing. Cause: TOC was bolted on at the end. Fix: write TOC first; scenarios are the expansion.
232232
10. **No exploratory scenario.** Symptom: the script enumerates everything the author thought of; nothing invites the runner to look beyond. Fix: add at least one charter scenario with a time budget. Unknown unknowns are the whole point of charters.
233233
11. **Pre-classifying severity in failure modes.** Symptom: "Red flag (blocker): the original bug returns". Fix: the script-author doesn't know the runtime context (release stage, blast radius, surrounding changes) at execution time. Enumerate the failure category; let the runner classify in the report.
234-
12. **Using `wip/` for the script.** Symptom: the file lands at `wip/manual-qa.md`. Cause: the author treated it as scratch. Fix: manual-QA scripts are durable artefacts that ship alongside the spec — they belong in `projects/<project>/` (tracked) and travel with the PR. `wip/` is gitignored and the script would vanish.
234+
12. **Using `wip/` for the script.** Symptom: the file lands at `wip/manual-qa.md`. Cause: the author treated it as scratch. Fix: manual-QA scripts are durable artefacts that ship alongside the spec — they belong in the project directory (tracked) and travel with the PR. `wip/` is gitignored and the script would vanish.
235235
13. **Missing isolation tags.** Symptom: scenarios have no Isolation field; the runner falls back to running them serially. Fix: every scripted scenario carries one of `tmpdir` / `workspace` / `read-only` / `external`. Exploratory charters too. The runner can't parallelise what the script doesn't tag.
236236
14. **Over-tagging isolation.** Symptom: every scenario tagged `workspace` "to be safe". Fix: the runner allocates a fresh `git worktree` per `workspace` scenario; tagging a tmpdir-only scenario as `workspace` doubles its setup cost for no benefit. Pick the strictest tag the scenario *actually* needs.
237237
15. **Phantom dependencies.** Symptom: scenarios list "Preconditions: scenario 1 completed" defensively, with no actual data flow. Cause: the author confused "the script lists scenario 1 first" with "scenario N consumes scenario 1's output". Fix: only declare a dependency if the scenario reads an artefact the prerequisite produced. False dependencies serialise the DAG for no reason.
@@ -251,7 +251,7 @@ By convention this repo's project artefacts live at `projects/<project-name>/`.
251251

252252
## Checklist
253253

254-
- [ ] Script lives at `projects/<project-name>/manual-qa.md` (or per-milestone equivalent), not `wip/`.
254+
- [ ] Script lives in the project directory (not `wip/`).
255255
- [ ] First ~50 lines give a reader the full scope: frame paragraph + "Out of scope of this script" + spec/plan/PR links + TOC table.
256256
- [ ] TOC table has one row per scenario with: number, verb-phrase title, one-line claim, **Isolation tag**, AC IDs covered.
257257
- [ ] Every scripted scenario has all required sections: What you're proving, Covers, **Isolation**, **Oracle**, **Preconditions**, Steps, What you should see, Failure modes (+ Restore if the scenario mutates state outside its isolation context).

skills-contrib/drive-qa-run/SKILL.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -54,12 +54,12 @@ Two cross-cutting rules:
5454

5555
## The report skeleton
5656

57-
Reports live at `projects/<project-name>/manual-qa-reports/<YYYY-MM-DD>-<runner-handle>.md`. One report per run. Reports accumulate (never overwrite) so the QA history is auditable.
57+
Reports live in a `manual-qa-reports/` directory in the project workspace, named `<YYYY-MM-DD>-<runner-handle>.md`. One report per run. Reports accumulate (never overwrite) so the QA history is auditable.
5858

5959
```markdown
6060
# Manual QA report — <TICKET-ID> (<short title>) — <YYYY-MM-DD>
6161

62-
> **Script:** `projects/<project>/manual-qa.md` (commit `<sha>` at run time)
62+
> **Script:** `manual-qa.md` (commit `<sha>` at run time)
6363
> **Runner:** <name or LLM session id>
6464
> **Environment:** <OS, Node version, branch + commit, any other relevant facts>
6565
> **Started / finished:** <ISO timestamps>
@@ -241,7 +241,7 @@ When you observe a failure-mode match or a mismatch with "What you should see":
241241
5. Fill in the **Coverage outcome** table by walking the script's "Sign-off coverage map" row-by-row. Each AC inherits its result from the worst-severity finding in any of the scenarios that covered it (Blocker > High > Follow-up > Pass). N/A rows stay N/A.
242242
6. Write **Suggested follow-ups** that cross-reference the disposition map. Each bullet should name the finding ID and its disposition (so a reader can see at a glance whether the bullet is "file this ticket" or "verify this post-merge").
243243
7. Set the **finished** timestamp.
244-
8. Save the report to `projects/<project-name>/manual-qa-reports/<YYYY-MM-DD>-<runner>.md`. Add the artefacts directory at the sibling `manual-qa-reports/artefacts/F-N/` paths.
244+
8. Save the report to the project's `manual-qa-reports/<YYYY-MM-DD>-<runner>.md`. Add the artefacts directory at the sibling `manual-qa-reports/artefacts/F-N/` paths.
245245

246246
### 6. Hand off
247247

@@ -268,7 +268,7 @@ Surface the report to the orchestrator / implementer / spec-author. The hand-off
268268
8. **Letting the exploratory scenario blow its budget.** Symptom: runner is on minute 90 of a 30-minute charter, has filed 5 follow-ups, hasn't started the final scripted scenario. Fix: respect the time budget; remaining ideas go in "Suggested follow-ups".
269269
9. **Filing findings against the script when they belong to the system.** Symptom: "F-3 — script said `migration plan` is a no-op but actually proposed an operation". Cause: runner attributed an observed-vs-expected mismatch to script staleness instead of a behavioural regression. Fix: only when you've verified the script *is* stale (e.g. the spec changed) does the finding belong to the script. If the script reflects current spec and reality diverges, the finding belongs to the system.
270270
10. **Confabulating cleanup.** Symptom: report says "restored to clean state" but `git status` was never actually run. Fix: paste the literal `git status` output (or its short form) into the report at the end of each state-mutating scenario, even when clean. Evidence over claims.
271-
11. **Using `wip/` for the report.** Symptom: report lands at `wip/qa-report.md`. Cause: runner treated it as ephemeral. Fix: reports are durable audit trail. They belong at `projects/<project>/manual-qa-reports/`. `wip/` is gitignored and the report would vanish.
271+
11. **Using `wip/` for the report.** Symptom: report lands at `wip/qa-report.md`. Cause: runner treated it as ephemeral. Fix: reports are durable audit trail. They belong in the project's `manual-qa-reports/` directory. `wip/` is gitignored and the report would vanish.
272272
12. **Serialising scenarios out of caution.** Symptom: runner dispatches scenarios one at a time even though they're tagged `tmpdir` or `read-only`. Cause: runner ignored the isolation tags. Fix: trust the tags. The cost of a mis-trusted tag is one false positive in the next QA round; the cost of pessimistic serialisation is every QA round taking 5x longer than it should. If the script genuinely under-tags, file that as a 📝 Follow-up against `drive-qa-plan` and run with the cautious schedule *this round only*.
273273
13. **Running `workspace` scenarios in the live workspace.** Symptom: runner skips the `git worktree add` step and runs the planted-file scenario directly in the project root. Cause: runner conflated "the scenario tagged `workspace`" with "execute against the user's checkout". Fix: `workspace` means "needs its own working tree", which the runner manufactures via `git worktree`. The user's actual checkout stays clean throughout the run.
274274
14. **Confusing isolation contexts in artefact paths.** Symptom: `manual-qa-reports/artefacts/F-3/` references a path under a worktree that's been removed by the time someone reads the report. Fix: copy artefacts into the durable `artefacts/F-N/` directory *before* tearing down the source worktree or tmpdir. Worktree paths are ephemeral; artefact directories are durable.
@@ -310,6 +310,6 @@ Surface the report to the orchestrator / implementer / spec-author. The hand-off
310310
- [ ] Coverage outcome table walks every AC the script's coverage map enumerates; each AC inherits the worst-severity finding from its covering scenarios.
311311
- [ ] Verdict in header keys off the disposition map: ✅ Pass only when every finding is ✅ resolved, 🔍 Triage required when any finding awaits orchestrator confirmation, ❌ Fail when any finding has a 🔧 fix-in-PR disposition or the runner observed an in-flight blocker.
312312
- [ ] "✅ Pass-with-follow-ups" never appears as a verdict — that wording was retired (see Common Pitfalls #17).
313-
- [ ] Report saved to `projects/<project>/manual-qa-reports/<YYYY-MM-DD>-<runner>.md`, artefacts under sibling `artefacts/F-N/`.
313+
- [ ] Report saved to the project's `manual-qa-reports/<YYYY-MM-DD>-<runner>.md`, artefacts under sibling `artefacts/F-N/`.
314314
- [ ] Worktrees torn down with `git worktree remove --force`; tmpdirs cleaned up; user's checkout left clean.
315315
- [ ] Did NOT edit the script in place; script-quality issues filed as 📝 Follow-up findings instead.

0 commit comments

Comments
 (0)