diff --git a/lore-proofkeeper/roadmaps/criteria-scenarios.md b/lore-proofkeeper/roadmaps/criteria-scenarios.md new file mode 100644 index 0000000..e7f4114 --- /dev/null +++ b/lore-proofkeeper/roadmaps/criteria-scenarios.md @@ -0,0 +1,74 @@ +--- +schema_version: 1 +id: PK-KWGJQ7WACJYP +type: roadmap +--- +# Acceptance-Criteria Scenarios + +## Outcomes + +- "Verified" deepens from *one test exists for the capability* to *every + stated acceptance criterion is exercised and asserted* — including negative + paths ("checkout refuses an expired card"), which single-goal drives rarely + attempt. +- Test plans stop being model-invented and become requirement-derived: the + scenarios a drive attempts are auditable against the artifact a human + already reviewed, where prompt-driven QA agents invent coverage the reviewer + must take on faith. +- Coverage reporting tells a maintainer *which criteria* of a capability are + unverified, not merely that the capability has a test. + +## Initiatives + +### Criteria extraction + +Read a requirement's `## Acceptance Criteria` (and `## Requirements` outcome +statements) through the published contract and derive one drive goal per +criterion — phrased as the observable outcome to assert, with negative-path +criteria flagged so the drive knows it is proving a refusal. + +### Per-criterion drives + +Run one drive per criterion through the existing loop (bounded by the scoped +pool's concurrency), each compiling to its own spec with a criterion-tagged +title, each gated on fidelity independently. A criterion whose drive gives up +or asserts nothing is unverified with its own reason — one criterion's failure +never hides another's success. + +### Scenario-level coverage + +Extend the coverage read-model and PR comment: a capability shows N/M criteria +verified, with per-criterion `## Verified By` entries written back through the +existing merge path (the dedup already keys on the spec reference). + +## Success Measures + +- A requirement with three acceptance criteria yields three goals, three + drives, and up to three verifying specs — visible as 3/3 (or 2/3 with the + gap named) in `proofkeeper coverage`. +- A negative-path criterion produces a spec asserting the refusal behavior, + green under the fidelity gate. +- The scoped PR comment shows criteria-level verification for touched + capabilities. + +## Assumptions + +- Acceptance criteria are written as observable outcomes (the corpus template + already prompts for this); prose criteria that name no observable outcome + are surfaced as "not driveable" rather than silently skipped. +- The graph contract exposes enough of the requirement body to extract + criteria without parsing engine internals. + +## Risks + +- Criterion decomposition multiplies drive cost by M. Mitigation: scoped runs + only re-verify criteria the change touches, and the visible-cost roadmap + bounds spend. +- Over-granular criteria produce trivial specs. Mitigation: the + assertion-required rule already refuses outcome-free sessions; the coverage + report keeps humans in the loop on criterion quality. + +## Related Roadmaps + +- autonomous-qa-enhancements +- visible-cost diff --git a/lore-proofkeeper/roadmaps/fidelity-matrix.md b/lore-proofkeeper/roadmaps/fidelity-matrix.md new file mode 100644 index 0000000..30e5fa0 --- /dev/null +++ b/lore-proofkeeper/roadmaps/fidelity-matrix.md @@ -0,0 +1,69 @@ +--- +schema_version: 1 +id: PK-KWGJQB3VB72P +type: roadmap +--- +# Cross-Target Fidelity + +## Outcomes + +- The word "stable" gains statistical weight: a committed test has survived + re-runs across more than one browser/target and cold conditions, not N + back-to-back runs in one process — closing the gap between "passed the gate + locally" and "stays green in CI". +- Flakiness is reported as a rate with a quarantine band, not a boolean: a + reviewer sees 9/10 with the failing condition named, and near-misses are + quarantined with evidence instead of silently committed or discarded. + +## Initiatives + +### Attempt matrix + +Extend the fidelity gate to distribute its N attempts across a configured +matrix — browsers (chromium first, then webkit/firefox where installed) and +targets (the environments the config already declares) — reusing the runner's +existing per-target loop and the verdict's per-attempt `runs`/`errors` +plumbing. + +### Pass-rate policy + +Make acceptance a policy, not a constant: `stable` remains all-green by +default, but the verdict reports the pass rate per matrix cell, and a +configurable quarantine band (e.g. ≥80% but not 100%) marks a candidate +"flaky — quarantined" with per-cell evidence, distinct from "failed". + +### Cold-condition attempts + +At least one attempt per assessment runs cold (fresh browser process, fresh +context) so recording-time warmth cannot mask timing dependence — the failure +mode most likely to surface later in CI. + +## Success Measures + +- A test that passes chromium but fails webkit is quarantined with the cell + named, never committed. +- A ~10–20% flaky test (simulated) is caught by the matrix at materially + higher probability than by N=3 same-process runs, demonstrated in the test + suite with a scripted runner. +- Verdict output (CLI and PR comment) shows per-cell pass rates without + changing the exit-code contract. + +## Assumptions + +- The Runner interface's target list and the verdict's existing pass/error + fields are sufficient plumbing; this is gate policy, not a runner rewrite. +- CI runners can provision the additional browsers where the matrix asks for + them; the matrix degrades gracefully to what is installed. + +## Risks + +- Matrix attempts multiply gate cost. Mitigation: the matrix is configuration + (default stays current behavior); scoped runs can use a narrow matrix and + releases a wide one. +- Quarantine bands could normalize flakiness. Mitigation: quarantined is + still unverified — the band only changes the reported reason and evidence. + +## Related Roadmaps + +- autonomous-qa-enhancements +- qa-benchmark diff --git a/lore-proofkeeper/roadmaps/proofkeeper-action.md b/lore-proofkeeper/roadmaps/proofkeeper-action.md new file mode 100644 index 0000000..56ad058 --- /dev/null +++ b/lore-proofkeeper/roadmaps/proofkeeper-action.md @@ -0,0 +1,69 @@ +--- +schema_version: 1 +id: PK-KWGJQ8N8YK52 +type: roadmap +--- +# Proofkeeper GitHub Action + +## Outcomes + +- Adopting Proofkeeper on a repository is one YAML block: a published + `itsthelore/proofkeeper-action@v1` runs scoped QA on pull requests and posts + the evidence comment — no bespoke workflow authoring. +- The trust-boundary posture becomes the selling point it deserves to be: a + security team can approve the action by reading its documentation — shell + off by default, egress allowlisted to the product under test, observations + redacted, write-backs only ever as human-reviewed PRs. +- Open-source distribution counters the closed-platform funnel: where DROID + adoption means adopting Factory, Proofkeeper adoption is `npm i` or one + Action reference, inspectable end to end. + +## Initiatives + +### Composite action + +A composite action wrapping the existing scoped entry point +(`proofkeeper qa --config … --base-ref …`): pinned Node and package versions, +Playwright browser provisioning, inputs mapping to the CLI's existing flags +(config path, coverage source, concurrency, propose), and the model key +supplied via the consumer's secrets. + +### Trust and permissions documentation + +A security README for the action: exactly which permissions the workflow +needs (and that `contents: read` suffices without `--propose`), what egress +occurs and to where, what reaches the model provider after redaction, and how +`allowShell`/`allowedHosts` opt-ins behave in CI. + +### Marketplace listing + +Publish to the GitHub Marketplace with the Dogfood badge story: the action +runs on Proofkeeper's own repository, verifying Proofkeeper's own corpus, +before it is asked to verify anyone else's. + +## Success Measures + +- A fresh repository goes from no QA automation to a scoped-QA evidence + comment on its next pull request by adding one workflow block and one + secret. +- The action's own CI (dogfooding it on this repository) is green and + required. +- The permissions documentation answers a security review without reading + source. + +## Assumptions + +- The CLI's existing scoped mode and PR comment are the right surface to wrap; + the action adds packaging, not behavior. +- Consumers accept bring-your-own-key economics (their secret, their spend). + +## Risks + +- Action version drift against the npm package. Mitigation: the action pins + the package version and releases in lockstep with CalVer cuts. +- CI browser cold-start cost per run. Mitigation: document Playwright caching; + scoped mode already limits drives to touched capabilities. + +## Related Roadmaps + +- autonomous-qa-enhancements diff --git a/lore-proofkeeper/roadmaps/qa-benchmark.md b/lore-proofkeeper/roadmaps/qa-benchmark.md new file mode 100644 index 0000000..81ecd9b --- /dev/null +++ b/lore-proofkeeper/roadmaps/qa-benchmark.md @@ -0,0 +1,93 @@ +--- +schema_version: 1 +id: PK-KWGJQ9J6X473 +type: roadmap +--- +# Public Deterministic QA Benchmark + +## Outcomes + +- "Better than DROID" becomes an argument anyone can re-run instead of a + claim: a public benchmark reporting verified-capability rate, token cost, + and wall-clock per capability across open sample applications with seeded + corpora. +- Scoring is deterministic — a capability is verified iff its compiled spec + passes the fidelity gate against the sample app — no embeddings, no LLM + judge, matching the engine's recorded evaluation philosophy (ADR-066). +- The benchmark doubles as the public launch artifact (Show HN) and as a + regression harness for Proofkeeper itself: a change that drops the + verified rate on the benchmark apps is visible before release. +- The benchmark is **agent-agnostic** by construction: it is a property of the + corpora, sample apps, and deterministic scoring — runnable against any + autonomous-QA agent — with Proofkeeper as the reference agent. An industry + benchmark Proofkeeper wins carries more weight than a product's own + marketing page. + +## Initiatives + +### Benchmark harness + +A runner that takes (sample app, corpus, agent config), runs the agent — +`proofkeeper qa` as the reference — per capability, and emits a +machine-readable result: verified/unverified per capability, fidelity +pass-rates, tokens in/out (already measured per drive), and wall-clock. +Deterministic re-scoring from recorded results, so publishing never requires +re-spending tokens. + +The harness, sample apps, corpora, and results page live in +`itsthelore/rac-benchmarks` as a subdir member (`autonomousqa/`, alongside +`decisiongrounding/`), per the engine's repository-topology decision +(ADR-092: one repo per concern, subdir per member — benchmarks are the named +example). The benchmark is a thin consumer of two published contracts — the +Proofkeeper CLI (npm) and `rac export --graph` — so it belongs to the +benchmarks concern, not to either product's source. Proofkeeper's own repo +keeps only the release-gate hook that runs the pinned benchmark. + +### Sample apps and corpora + +Three to four small open apps spanning the drive modalities — a browser flow +app, an API service, a CLI tool, and a browser extension — each with a seeded +Lore corpus whose requirements carry acceptance criteria of graded difficulty, +including negative paths and deliberately ambiguous cases that *should* score +as honestly unverified. + +### Published results + +A results page (and per-release history) generated from the harness output: +rates by app, by modality, and by model — the BYOK story made concrete by +publishing the same benchmark across several providers. + +## Success Measures + +- A third party can clone the benchmark, set one API key, and reproduce the + published numbers within the run-to-run variance the page itself states. +- Every Proofkeeper release links its benchmark results; a regression in + verified rate blocks the release the way Dogfood blocks a merge. +- The benchmark distinguishes models measurably (same harness, different + verified rates/costs), demonstrating the routing thesis the visible-cost + roadmap builds on. + +## Assumptions + +- Small seeded apps are representative enough to be meaningful; the benchmark + measures the agent loop, not app complexity records. +- Publishing honest unverified rates (not 100%) is a credibility feature, not + a marketing liability. +- `rac-benchmarks` remains the benchmarks-concern home (ADR-092), and the + benchmark consumes Proofkeeper and rac only through their published + contracts — the npm package and the graph export — never repo internals. +- Benchmark apps freeze on their own cadence, independent of Proofkeeper's + CalVer line; the Proofkeeper release gate pins a benchmark version. + +## Risks + +- Overfitting the drive to benchmark apps. Mitigation: benchmark apps are + frozen once published; new capabilities land as new apps, and the Dogfood + corpus remains an independent signal. +- Cross-provider comparisons invite disputes. Mitigation: publish the exact + harness config per result; scoring stays deterministic and re-runnable. + +## Related Roadmaps + +- visible-cost +- criteria-scenarios diff --git a/lore-proofkeeper/roadmaps/regression-triage.md b/lore-proofkeeper/roadmaps/regression-triage.md new file mode 100644 index 0000000..90e1e6e --- /dev/null +++ b/lore-proofkeeper/roadmaps/regression-triage.md @@ -0,0 +1,86 @@ +--- +schema_version: 1 +id: PK-KWGJQ71B3KEN +type: roadmap +--- +# Regression Triage — Stale Test or Broken Product + +## Outcomes + +- When a committed `## Verified By` test fails, the maintainer learns *which + thing broke*: the test (the UI changed but the requirement still holds) or + the product (the requirement itself is now violated) — with evidence, not a + red X to investigate by hand. +- Verified coverage stays verified over time: stale tests are repaired through + the same human-reviewed PR path that created them, so the corpus's + `verified_by` edges never silently rot. +- Proofkeeper's structural advantage over prompt-driven QA agents becomes a + shipped capability: a requirements corpus is ground truth that lets test + failure be *adjudicated*, not just reported. Agents without one (Factory + DROID's automated-qa, QA Wolf's maintained suites) can update a broken test + or escalate it, but cannot say whether the product still does what it is + supposed to do. + +## Initiatives + +### Triage command + +A `proofkeeper triage` entry point: given a failing verifying test (or a CI +report of one), load the capability it verifies from the graph, re-run the +committed spec to confirm the failure, and open the triage loop. Same exit-code +discipline as the rest of the CLI: distinct codes for "test repaired", +"product regression", and "inconclusive". + +### Re-drive verdict + +The adjudication core: re-drive the capability against the requirement text +(the same drive → record → compile loop), then compare. A fresh drive that +finishes and asserts the requirement's outcomes while the old spec fails means +the test went stale; a fresh drive that cannot satisfy the requirement's +outcomes means the product regressed. The verdict carries the evidence — the +failing assertion, the fresh session's steps, and the requirement lines they +map to. + +### Repair write-back + +For a stale-test verdict, compile the fresh session and propose the updated +spec through the existing write-back path — a human-reviewed PR replacing the +old verifier, never a silent self-heal (the trust boundary stays PR review). +For a regression verdict, open an issue-shaped report instead: capability, +requirement lines violated, trace, and the diff between recorded and observed +behavior. + +### Failure-learning integration + +Triage verdicts feed the learning store: repaired-test causes steer future +drives toward more resilient locators and assertions; regression reports +accumulate per capability so flaky products are visible over time. + +## Success Measures + +- A deliberately introduced UI rename (requirement unchanged) yields a + stale-test verdict and a repair PR whose new spec passes the fidelity gate. +- A deliberately introduced behavior break (requirement violated) yields a + regression verdict naming the violated requirement lines, with a trace. +- No triage outcome ever writes to a base branch; every repair is a PR. + +## Assumptions + +- Requirements carry enough observable intent (acceptance criteria, outcome + statements) for a re-drive to adjudicate against; where they do not, the + verdict is honestly "inconclusive" rather than guessed. +- The fidelity gate remains the acceptance bar for any repaired spec. + +## Risks + +- A model may "repair" a test into asserting the broken behavior. Mitigation: + the verdict is grounded in the requirement text, the repair PR shows the + old/new assertions side by side, and a human reviews it. +- Re-drives spend tokens on every CI failure. Mitigation: triage runs on + demand (or on a verifying-test failure only), and the visible-cost roadmap + gives it a budget. + +## Related Roadmaps + +- autonomous-qa-enhancements +- visible-cost diff --git a/lore-proofkeeper/roadmaps/visible-cost.md b/lore-proofkeeper/roadmaps/visible-cost.md new file mode 100644 index 0000000..ded6434 --- /dev/null +++ b/lore-proofkeeper/roadmaps/visible-cost.md @@ -0,0 +1,69 @@ +--- +schema_version: 1 +id: PK-KWGJQABQ1SQQ +type: roadmap +--- +# Cost as a Feature + +## Outcomes + +- Autonomous QA at a price you can see: every drive's token spend is visible + where decisions are made (QA summary, scoped PR comment), a runaway drive + halts at a budget instead of surprising an invoice, and easy turns can be + routed to cheap models without changing Proofkeeper. +- Cost becomes a wedge against closed platforms: a BYOK consumer can measure, + bound, and optimize spend per verified capability — numbers a platform + black-box cannot show. + +## Initiatives + +### Budget enforcement + +Turn the measured usage (adapters already surface provider tokens per turn, +accumulated per drive) into a limit: a `--max-tokens` cap per drive/run that +stops the loop at the boundary with an honest "budget exhausted" unverified +reason — recorded to failure-learning like any other stop, never a crash. + +### Cost in the evidence + +Spend lines in the QA summary and the scoped PR comment: tokens in/out per +capability and per run, so the reviewer reading the verification evidence +also sees what it cost — and the per-capability history makes expensive +capabilities visible over time. + +### Wayfinder composition + +Route by complexity without embedding a router: Proofkeeper keeps exactly one +model seam (`OPENAI_BASE_URL`), and a Wayfinder endpoint behind it sends +trivial turns to a cheap model and hard turns to a strong one. Compose, don't +embed — routing stays a separate product (per the engine's recorded Wayfinder +boundary decisions), and Proofkeeper's only contribution is documenting the +composition and verifying the adapter behaves identically through it. + +## Success Measures + +- A drive with `--max-tokens` below its natural spend stops at the cap with + the budget named in its unverified reason; one with headroom is unaffected. +- The scoped PR comment shows per-capability token totals. +- The same corpus verified through a Wayfinder endpoint reports a materially + lower cost per verified capability than a single strong model, measured by + the qa-benchmark harness. + +## Assumptions + +- Provider usage fields remain the stable cost signal (already mapped by both + adapters). +- Wayfinder speaks the OpenAI-compatible surface, so composition needs no new + adapter. + +## Risks + +- A too-tight default budget would fail honest drives. Mitigation: no default + cap — budgets are explicit operator opt-ins, like the shell. +- Cost lines could drown the evidence comment. Mitigation: one line per + capability, totals in the summary row. + +## Related Roadmaps + +- autonomous-qa-enhancements +- qa-benchmark