docs(roadmap): capture six next-generation QA roadmaps

lore-proofkeeper/roadmaps/criteria-scenarios.md

@@ -0,0 +1,74 @@
+---
+schema_version: 1
+id: PK-KWGJQ7WACJYP
+type: roadmap
+---
+# Acceptance-Criteria Scenarios
+
+## Outcomes
+
+- "Verified" deepens from *one test exists for the capability* to *every
+  stated acceptance criterion is exercised and asserted* — including negative
+  paths ("checkout refuses an expired card"), which single-goal drives rarely
+  attempt.
+- Test plans stop being model-invented and become requirement-derived: the
+  scenarios a drive attempts are auditable against the artifact a human
+  already reviewed, where prompt-driven QA agents invent coverage the reviewer
+  must take on faith.
+- Coverage reporting tells a maintainer *which criteria* of a capability are
+  unverified, not merely that the capability has a test.
+
+## Initiatives
+
+### Criteria extraction
+
+Read a requirement's `## Acceptance Criteria` (and `## Requirements` outcome
+statements) through the published contract and derive one drive goal per
+criterion — phrased as the observable outcome to assert, with negative-path
+criteria flagged so the drive knows it is proving a refusal.
+
+### Per-criterion drives
+
+Run one drive per criterion through the existing loop (bounded by the scoped
+pool's concurrency), each compiling to its own spec with a criterion-tagged
+title, each gated on fidelity independently. A criterion whose drive gives up
+or asserts nothing is unverified with its own reason — one criterion's failure
+never hides another's success.
+
+### Scenario-level coverage
+
+Extend the coverage read-model and PR comment: a capability shows N/M criteria
+verified, with per-criterion `## Verified By` entries written back through the
+existing merge path (the dedup already keys on the spec reference).
+
+## Success Measures
+
+- A requirement with three acceptance criteria yields three goals, three
+  drives, and up to three verifying specs — visible as 3/3 (or 2/3 with the
+  gap named) in `proofkeeper coverage`.
+- A negative-path criterion produces a spec asserting the refusal behavior,
+  green under the fidelity gate.
+- The scoped PR comment shows criteria-level verification for touched
+  capabilities.
+
+## Assumptions
+
+- Acceptance criteria are written as observable outcomes (the corpus template
+  already prompts for this); prose criteria that name no observable outcome
+  are surfaced as "not driveable" rather than silently skipped.
+- The graph contract exposes enough of the requirement body to extract
+  criteria without parsing engine internals.
+
+## Risks
+
+- Criterion decomposition multiplies drive cost by M. Mitigation: scoped runs
+  only re-verify criteria the change touches, and the visible-cost roadmap
+  bounds spend.
+- Over-granular criteria produce trivial specs. Mitigation: the
+  assertion-required rule already refuses outcome-free sessions; the coverage
+  report keeps humans in the loop on criterion quality.
+
+## Related Roadmaps
+
+- autonomous-qa-enhancements
+- visible-cost

lore-proofkeeper/roadmaps/fidelity-matrix.md

@@ -0,0 +1,69 @@
+---
+schema_version: 1
+id: PK-KWGJQB3VB72P
+type: roadmap
+---
+# Cross-Target Fidelity
+
+## Outcomes
+
+- The word "stable" gains statistical weight: a committed test has survived
+  re-runs across more than one browser/target and cold conditions, not N
+  back-to-back runs in one process — closing the gap between "passed the gate
+  locally" and "stays green in CI".
+- Flakiness is reported as a rate with a quarantine band, not a boolean: a
+  reviewer sees 9/10 with the failing condition named, and near-misses are
+  quarantined with evidence instead of silently committed or discarded.
+
+## Initiatives
+
+### Attempt matrix
+
+Extend the fidelity gate to distribute its N attempts across a configured
+matrix — browsers (chromium first, then webkit/firefox where installed) and
+targets (the environments the config already declares) — reusing the runner's
+existing per-target loop and the verdict's per-attempt `runs`/`errors`
+plumbing.
+
+### Pass-rate policy
+
+Make acceptance a policy, not a constant: `stable` remains all-green by
+default, but the verdict reports the pass rate per matrix cell, and a
+configurable quarantine band (e.g. ≥80% but not 100%) marks a candidate
+"flaky — quarantined" with per-cell evidence, distinct from "failed".
+
+### Cold-condition attempts
+
+At least one attempt per assessment runs cold (fresh browser process, fresh
+context) so recording-time warmth cannot mask timing dependence — the failure
+mode most likely to surface later in CI.
+
+## Success Measures
+
+- A test that passes chromium but fails webkit is quarantined with the cell
+  named, never committed.
+- A ~10–20% flaky test (simulated) is caught by the matrix at materially
+  higher probability than by N=3 same-process runs, demonstrated in the test
+  suite with a scripted runner.
+- Verdict output (CLI and PR comment) shows per-cell pass rates without
+  changing the exit-code contract.
+
+## Assumptions
+
+- The Runner interface's target list and the verdict's existing pass/error
+  fields are sufficient plumbing; this is gate policy, not a runner rewrite.
+- CI runners can provision the additional browsers where the matrix asks for
+  them; the matrix degrades gracefully to what is installed.
+
+## Risks
+
+- Matrix attempts multiply gate cost. Mitigation: the matrix is configuration
+  (default stays current behavior); scoped runs can use a narrow matrix and
+  releases a wide one.
+- Quarantine bands could normalize flakiness. Mitigation: quarantined is
+  still unverified — the band only changes the reported reason and evidence.
+
+## Related Roadmaps
+
+- autonomous-qa-enhancements
+- qa-benchmark

lore-proofkeeper/roadmaps/proofkeeper-action.md

@@ -0,0 +1,69 @@
+---
+schema_version: 1
+id: PK-KWGJQ8N8YK52
+type: roadmap
+---
+# Proofkeeper GitHub Action
+
+## Outcomes
+
+- Adopting Proofkeeper on a repository is one YAML block: a published
+  `itsthelore/proofkeeper-action@v1` runs scoped QA on pull requests and posts
+  the evidence comment — no bespoke workflow authoring.
+- The trust-boundary posture becomes the selling point it deserves to be: a
+  security team can approve the action by reading its documentation — shell
+  off by default, egress allowlisted to the product under test, observations
+  redacted, write-backs only ever as human-reviewed PRs.
+- Open-source distribution counters the closed-platform funnel: where DROID
+  adoption means adopting Factory, Proofkeeper adoption is `npm i` or one
+  Action reference, inspectable end to end.
+
+## Initiatives
+
+### Composite action
+
+A composite action wrapping the existing scoped entry point
+(`proofkeeper qa --config … --base-ref …`): pinned Node and package versions,
+Playwright browser provisioning, inputs mapping to the CLI's existing flags
+(config path, coverage source, concurrency, propose), and the model key
+supplied via the consumer's secrets.
+
+### Trust and permissions documentation
+
+A security README for the action: exactly which permissions the workflow
+needs (and that `contents: read` suffices without `--propose`), what egress
+occurs and to where, what reaches the model provider after redaction, and how
+`allowShell`/`allowedHosts` opt-ins behave in CI.
+
+### Marketplace listing
+
+Publish to the GitHub Marketplace with the Dogfood badge story: the action
+runs on Proofkeeper's own repository, verifying Proofkeeper's own corpus,
+before it is asked to verify anyone else's.
+
+## Success Measures
+
+- A fresh repository goes from no QA automation to a scoped-QA evidence
+  comment on its next pull request by adding one workflow block and one
+  secret.
+- The action's own CI (dogfooding it on this repository) is green and
+  required.
+- The permissions documentation answers a security review without reading
+  source.
+
+## Assumptions
+
+- The CLI's existing scoped mode and PR comment are the right surface to wrap;
+  the action adds packaging, not behavior.
+- Consumers accept bring-your-own-key economics (their secret, their spend).
+
+## Risks
+
+- Action version drift against the npm package. Mitigation: the action pins
+  the package version and releases in lockstep with CalVer cuts.
+- CI browser cold-start cost per run. Mitigation: document Playwright caching;
+  scoped mode already limits drives to touched capabilities.
+
+## Related Roadmaps
+
+- autonomous-qa-enhancements

lore-proofkeeper/roadmaps/qa-benchmark.md

@@ -0,0 +1,93 @@
+---
+schema_version: 1
+id: PK-KWGJQ9J6X473
+type: roadmap
+---
+# Public Deterministic QA Benchmark
+
+## Outcomes
+
+- "Better than DROID" becomes an argument anyone can re-run instead of a
+  claim: a public benchmark reporting verified-capability rate, token cost,
+  and wall-clock per capability across open sample applications with seeded
+  corpora.
+- Scoring is deterministic — a capability is verified iff its compiled spec
+  passes the fidelity gate against the sample app — no embeddings, no LLM
+  judge, matching the engine's recorded evaluation philosophy (ADR-066).
+- The benchmark doubles as the public launch artifact (Show HN) and as a
+  regression harness for Proofkeeper itself: a change that drops the
+  verified rate on the benchmark apps is visible before release.
+- The benchmark is **agent-agnostic** by construction: it is a property of the
+  corpora, sample apps, and deterministic scoring — runnable against any
+  autonomous-QA agent — with Proofkeeper as the reference agent. An industry
+  benchmark Proofkeeper wins carries more weight than a product's own
+  marketing page.
+
+## Initiatives
+
+### Benchmark harness
+
+A runner that takes (sample app, corpus, agent config), runs the agent —
+`proofkeeper qa` as the reference — per capability, and emits a
+machine-readable result: verified/unverified per capability, fidelity
+pass-rates, tokens in/out (already measured per drive), and wall-clock.
+Deterministic re-scoring from recorded results, so publishing never requires
+re-spending tokens.
+
+The harness, sample apps, corpora, and results page live in
+`itsthelore/rac-benchmarks` as a subdir member (`autonomousqa/`, alongside
+`decisiongrounding/`), per the engine's repository-topology decision
+(ADR-092: one repo per concern, subdir per member — benchmarks are the named
+example). The benchmark is a thin consumer of two published contracts — the
+Proofkeeper CLI (npm) and `rac export --graph` — so it belongs to the
+benchmarks concern, not to either product's source. Proofkeeper's own repo
+keeps only the release-gate hook that runs the pinned benchmark.
+
+### Sample apps and corpora
+
+Three to four small open apps spanning the drive modalities — a browser flow
+app, an API service, a CLI tool, and a browser extension — each with a seeded
+Lore corpus whose requirements carry acceptance criteria of graded difficulty,
+including negative paths and deliberately ambiguous cases that *should* score
+as honestly unverified.
+
+### Published results
+
+A results page (and per-release history) generated from the harness output:
+rates by app, by modality, and by model — the BYOK story made concrete by
+publishing the same benchmark across several providers.
+
+## Success Measures
+
+- A third party can clone the benchmark, set one API key, and reproduce the
+  published numbers within the run-to-run variance the page itself states.
+- Every Proofkeeper release links its benchmark results; a regression in
+  verified rate blocks the release the way Dogfood blocks a merge.
+- The benchmark distinguishes models measurably (same harness, different
+  verified rates/costs), demonstrating the routing thesis the visible-cost
+  roadmap builds on.
+
+## Assumptions
+
+- Small seeded apps are representative enough to be meaningful; the benchmark
+  measures the agent loop, not app complexity records.
+- Publishing honest unverified rates (not 100%) is a credibility feature, not
+  a marketing liability.
+- `rac-benchmarks` remains the benchmarks-concern home (ADR-092), and the
+  benchmark consumes Proofkeeper and rac only through their published
+  contracts — the npm package and the graph export — never repo internals.
+- Benchmark apps freeze on their own cadence, independent of Proofkeeper's
+  CalVer line; the Proofkeeper release gate pins a benchmark version.
+
+## Risks
+
+- Overfitting the drive to benchmark apps. Mitigation: benchmark apps are
+  frozen once published; new capabilities land as new apps, and the Dogfood
+  corpus remains an independent signal.
+- Cross-provider comparisons invite disputes. Mitigation: publish the exact
+  harness config per result; scoring stays deterministic and re-runnable.
+
+## Related Roadmaps
+
+- visible-cost
+- criteria-scenarios

lore-proofkeeper/roadmaps/regression-triage.md

@@ -0,0 +1,86 @@
+---
+schema_version: 1
+id: PK-KWGJQ71B3KEN
+type: roadmap
+---
+# Regression Triage — Stale Test or Broken Product
+
+## Outcomes
+
+- When a committed `## Verified By` test fails, the maintainer learns *which
+  thing broke*: the test (the UI changed but the requirement still holds) or
+  the product (the requirement itself is now violated) — with evidence, not a
+  red X to investigate by hand.
+- Verified coverage stays verified over time: stale tests are repaired through
+  the same human-reviewed PR path that created them, so the corpus's
+  `verified_by` edges never silently rot.
+- Proofkeeper's structural advantage over prompt-driven QA agents becomes a
+  shipped capability: a requirements corpus is ground truth that lets test
+  failure be *adjudicated*, not just reported. Agents without one (Factory
+  DROID's automated-qa, QA Wolf's maintained suites) can update a broken test
+  or escalate it, but cannot say whether the product still does what it is
+  supposed to do.
+
+## Initiatives
+
+### Triage command
+
+A `proofkeeper triage` entry point: given a failing verifying test (or a CI
+report of one), load the capability it verifies from the graph, re-run the
+committed spec to confirm the failure, and open the triage loop. Same exit-code
+discipline as the rest of the CLI: distinct codes for "test repaired",
+"product regression", and "inconclusive".
+
+### Re-drive verdict
+
+The adjudication core: re-drive the capability against the requirement text
+(the same drive → record → compile loop), then compare. A fresh drive that
+finishes and asserts the requirement's outcomes while the old spec fails means
+the test went stale; a fresh drive that cannot satisfy the requirement's
+outcomes means the product regressed. The verdict carries the evidence — the
+failing assertion, the fresh session's steps, and the requirement lines they
+map to.
+
+### Repair write-back
+
+For a stale-test verdict, compile the fresh session and propose the updated
+spec through the existing write-back path — a human-reviewed PR replacing the
+old verifier, never a silent self-heal (the trust boundary stays PR review).
+For a regression verdict, open an issue-shaped report instead: capability,
+requirement lines violated, trace, and the diff between recorded and observed
+behavior.
+
+### Failure-learning integration
+
+Triage verdicts feed the learning store: repaired-test causes steer future
+drives toward more resilient locators and assertions; regression reports
+accumulate per capability so flaky products are visible over time.
+
+## Success Measures
+
+- A deliberately introduced UI rename (requirement unchanged) yields a
+  stale-test verdict and a repair PR whose new spec passes the fidelity gate.
+- A deliberately introduced behavior break (requirement violated) yields a
+  regression verdict naming the violated requirement lines, with a trace.
+- No triage outcome ever writes to a base branch; every repair is a PR.
+
+## Assumptions
+
+- Requirements carry enough observable intent (acceptance criteria, outcome
+  statements) for a re-drive to adjudicate against; where they do not, the
+  verdict is honestly "inconclusive" rather than guessed.
+- The fidelity gate remains the acceptance bar for any repaired spec.
+
+## Risks
+
+- A model may "repair" a test into asserting the broken behavior. Mitigation:
+  the verdict is grounded in the requirement text, the repair PR shows the
+  old/new assertions side by side, and a human reviews it.
+- Re-drives spend tokens on every CI failure. Mitigation: triage runs on
+  demand (or on a verifying-test failure only), and the visible-cost roadmap
+  gives it a budget.
+
+## Related Roadmaps
+
+- autonomous-qa-enhancements
+- visible-cost