Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/workflows/dogfood.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Dogfooding

# Proofkeeper verifies itself: run its own `coverage` command against its own
# Lore corpus (lore-proofkeeper/) and fail if any capability is unverified. This
# turns the dogfood signal into an enforced gate — the corpus can never drift
# un-green. It exercises the real CLI and the real `rac export --graph` contract
# path (ADR-063, ADR-083), not a bespoke checker.

on:
push:
branches: ["**"]
pull_request:

jobs:
coverage:
name: self-coverage (proofkeeper verifies proofkeeper)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: "20"
cache: npm
- run: npm ci
- run: npm run build

# The `rac` engine is a contract dependency, not a package dependency:
# install it as an external CLI from rac-core (its distribution is
# "requirements-as-code"; the console script is `rac`). Proofkeeper consumes
# only its published `rac export --graph` JSON output.
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install the rac engine (provides the `rac` CLI)
run: pip install "git+https://github.com/itsthelore/rac-core.git"

- name: Proofkeeper coverage over its own corpus (fails on any unverified)
run: node dist/cli.js coverage --corpus lore-proofkeeper/
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Banner: add docs/assets/proofkeeper-header-{dark,light}.png, then uncomment.

<p align="center">
<a href="https://github.com/itsthelore/proofkeeper/actions/workflows/ci.yml"><img src="https://github.com/itsthelore/proofkeeper/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
<a href="https://github.com/itsthelore/proofkeeper/actions/workflows/dogfood.yml"><img src="https://img.shields.io/github/actions/workflow/status/itsthelore/proofkeeper/dogfood.yml?branch=main&label=dogfooding&logo=githubactions&logoColor=white" alt="Dogfooding: Proofkeeper verifies its own corpus"></a>
<a href="https://www.npmjs.com/package/@itsthelore/proofkeeper"><img src="https://img.shields.io/npm/v/@itsthelore/proofkeeper" alt="npm"></a>
<img src="https://img.shields.io/badge/node-%E2%89%A520-blue" alt="Node >= 20">
<img src="https://img.shields.io/badge/types-TypeScript-blue.svg" alt="TypeScript">
Expand Down
63 changes: 63 additions & 0 deletions lore-proofkeeper/designs/design-dogfood-coverage-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
schema_version: 1
id: PK-KWBH4HMAVW38
type: design
---
# Self-Verification Coverage Gate

## Context

Proofkeeper records its own capabilities in `lore-proofkeeper/` and proves them with `## Verified By` links, but the dogfood report was never enforced. This design makes "Proofkeeper's corpus stays fully verified" a required CI gate, using Proofkeeper's own coverage command, and surfaces it as a README badge.

## User Need

A maintainer (and a reader evaluating the project) needs confidence that Proofkeeper actually practises what it sells — that every capability it claims is backed by a test — without manually running coverage and without trusting an advisory number that can silently drift.

## Design

- A dedicated workflow `.github/workflows/dogfood.yml`, named so its status badge reads as the dogfood signal. On push and pull request it:
- checks out the repo, sets up Node, `npm ci`, `npm run build`;
- sets up Python and installs the `rac` engine from the rac-core repository (`pip install "git+https://github.com/itsthelore/rac-core.git"`), which provides the `rac` command;
- runs `node dist/cli.js coverage --corpus lore-proofkeeper/`. The command exits non-zero if any capability is unverified, so the job fails on drift with no extra scripting.
- The README adds a badge pointing at this workflow's status on the default branch (a shields.io GitHub-Actions badge with a `dogfooding` label), next to the existing CI and npm badges.
- This requirement is itself verified by the workflow file, keeping the corpus self-consistent (every capability, including this one, has a `## Verified By`).

## Constraints

- Reuse the published behaviour: the gate runs the real `coverage` CLI and the real `rac export` contract path, not a bespoke checker (ADR-063, ADR-083).
- No engine import: `rac` is installed as an external CLI; Proofkeeper consumes its JSON output only.
- Determinism over snapshots: export the corpus live rather than committing a graph file that could drift from the artifacts.

## Rationale

Enforcing the existing coverage command is the smallest change that turns an advisory signal into a guarantee, and it dogfoods the product end to end — Proofkeeper verifies itself exactly as a consumer would. A workflow-status badge is honest because it reflects a real run, not a hand-set value.

## Alternatives

- **Commit a `rac export --graph` snapshot and check against it.** Rejected: the snapshot drifts from the artifacts and would need its own freshness gate.
- **A custom script that parses the corpus directly.** Rejected: it would bypass the very CLI and contract path the product ships, weakening the dogfood.
- **A manually maintained "passing" badge.** Rejected: not tied to a real result; can lie.

## Accessibility

The badge carries descriptive alt text; status is conveyed by text ("passing"/"failing"), not colour alone.

## Style Guidance

Match the existing workflow conventions in `.github/workflows/ci.yml` (Node 20, `npm ci`, named jobs) and place the badge alongside the current badge row.

## Open Questions

- Whether to pin the rac-core install to a released tag for extra determinism once a stable cadence exists. Deferred; track via the schema_version guard for now.

## Related Requirements

- req-dogfood-coverage-gate

## Related Roadmaps

- autonomous-qa-enhancements

## Status

Accepted
40 changes: 40 additions & 0 deletions lore-proofkeeper/requirements/req-dogfood-coverage-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
schema_version: 1
id: PK-KWBH4H0WB6VH
type: requirement
---
# Self-Verification Coverage Gate

## Problem

Proofkeeper dogfoods itself: its own capabilities live as requirements in `lore-proofkeeper/`, and `proofkeeper coverage --corpus lore-proofkeeper/` reports which have a verifying test. But nothing enforced that report — a capability could be added without a `## Verified By` link and the corpus would quietly drift un-green, undermining the product's own promise. The dogfood signal was advisory, not a gate.

## Requirements

- [REQ-001] Continuous integration runs `proofkeeper coverage` against Proofkeeper's own corpus on every push and pull request, and fails the build when any capability is unverified.
- [REQ-002] The gate uses Proofkeeper's own published CLI behaviour (coverage exits non-zero when a capability has no verifying test), not a bespoke script, so Proofkeeper verifies itself the same way it verifies any consumer.
- [REQ-003] The gate obtains the graph through the supported contract path — the `rac` CLI exporting the corpus — rather than a committed snapshot that could drift.
- [REQ-004] The self-verification status is visible from the README as a badge reflecting the gate's latest result on the default branch.

## Success Metrics

- A pull request that adds a capability without a verifying test fails CI on the coverage gate.
- The README badge shows the dogfood gate passing on the default branch when the corpus is fully verified.

## Risks

- The gate depends on installing the `rac` engine in CI; an upstream rac-core breakage could fail the job for reasons unrelated to Proofkeeper. Mitigation: install the engine from its repository as a contract dependency, and rely on the graph schema_version guard to distinguish a genuine contract break from noise.
- A flaky external install could make the gate noisy. Mitigation: keep the job minimal and cache where possible; the gate's only product step is the deterministic coverage command.

## Assumptions

- The `rac` engine is installable from the rac-core repository and exposes `rac export <dir> --graph` (the published contract).
- `proofkeeper coverage` continues to exit non-zero when any capability is unverified.

## Related Roadmaps

- autonomous-qa-enhancements

## Verified By

- `.github/workflows/dogfood.yml`
6 changes: 6 additions & 0 deletions lore-proofkeeper/roadmaps/autonomous-qa-enhancements.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ rediscover its runtime ID, and emit a compiled test that re-loads the extension
so an extension's popup/options and page effects are verified end-to-end, not just
ordinary web apps. Serves the faithful-tests outcome on a new surface.

### Self-verification coverage gate

A required CI job that runs `proofkeeper coverage` against Proofkeeper's own
corpus and fails on any unverified capability, with a README badge — so the
dogfood signal becomes an enforced guarantee that can never drift un-green.

## Success Measures

- A pull request shows exactly one Proofkeeper QA comment regardless of how many
Expand Down
Loading