Skip to content

FIXTURES.md

Latest commit

 

History

History
125 lines (89 loc) · 5.29 KB

FIXTURES.md

File metadata and controls

125 lines (89 loc) · 5.29 KB
Raw

FIXTURES

The harness is trustworthy only because it is differentially self-tested: each oracle carries a golden-good fixture it must accept and a golden-bad fixture it must reject. This document describes the shipped fixtures and how to add your own.

Why golden-good AND golden-bad

A validator that only ever runs against "good" inputs can silently rot into a rubber stamp that returns PASS for everything — the exact bluff this project exists to kill. Pairing a must-PASS fixture with a must-FAIL fixture proves the oracle discriminates:

  • golden-good must PASS → the oracle doesn't raise false alarms on correct output.
  • golden-bad must FAIL → the oracle actually detects the defects it claims to detect.

self-validate.sh in each harness asserts both and exits non-zero if either is wrong.

Visual fixtures (visual/fixtures/)

Both are self-contained HTML with inline CSS supporting light and dark (prefers-color-scheme + [data-theme="dark"]).

good.html — must PASS

A centered subscription-confirmation card: heading, paragraph, and two buttons (Confirm primary + Cancel). Everything sits inside the viewport, nothing overlaps, no control is oversized. Correct in both themes → verdict PASS.

bad.html — must FAIL

Deliberately seeds one instance of every defect class the oracle detects:

Defect How it's seeded Oracle finding
Overlap two <p> labels absolutely positioned at the same top/left overlap (100% of smaller box)
Off-screen / clipped a button at left:1160px; width:260px on a 1280px viewport → right edge at 1420px offscreen-clipped
Giant / unbounded a 2400px × 640px button giant-widget (>85%vw, >60%vh)

Running the oracle on it yields multiple hard error findings → verdict FAIL.

Export fixtures (export/fixtures/)

The PDFs are generated from Markdown via pandoc → weasyprint, with an embedded raster figure produced by Chromium. Sources are committed alongside the built PDFs so the fixtures are fully reproducible.

File Role
good.md source for the clean PDF
bad.md source for the broken PDF
figure.png raster chart embedded in the good PDF (so pdfimages finds an image)
golden-good.pdf built artifact, must PASS
golden-bad.pdf built artifact, must FAIL

golden-good.pdf — must PASS

Clean prose, a complete milestone table whose final row is Release Q4 … Shipped, and an embedded throughput figure. It satisfies every check: enough words, both --expect phrases (Shipped, Release Q4) present, no markup leak, ≥1 embedded image, and legible OCR.

golden-bad.pdf — must FAIL

Seeds the defects the validator hunts:

Defect How it's seeded Check that fails
Raw Mermaid leak a ```mermaid gantt block left unrendered as body text (gantt, dateFormat, section, :done,, :active,, :crit,) TEXTUAL/textual.no-leak
Truncated export the milestone table is cut off — Release Q4/Shipped rows removed CONTENT/content.faithful[...]
Missing figure no embedded raster image FULL-VISUAL/visual.images

Verdict FAIL.

Rebuilding the export fixtures

npm run build-fixtures      # == bash export/build-fixtures.sh

Requires pandoc, weasyprint, and the Playwright Chromium (for make-image.js). The build is deterministic from good.md / bad.md. self-validate.sh auto-builds the PDFs if they are missing.

Adding your own fixtures

You do not need to touch the shipped goldens to test your own product. Point the oracles at your artifacts (see USAGE.md). But if you want to extend the self-test — for example to pin a new defect class you care about — follow the golden-good/golden-bad pattern:

A new visual fixture

  1. Add visual/fixtures/my-good.html (renders cleanly) and visual/fixtures/my-bad.html (contains exactly the defect you want the oracle to catch).
  2. Sanity-check them directly: 
    node visual/visual-oracle.js --input visual/fixtures/my-good.html --out /tmp/v --name my-good   # expect rc 0
node visual/visual-oracle.js --input visual/fixtures/my-bad.html  --out /tmp/v --name my-bad    # expect rc 1
  3. To make it part of self-validation, add a matching pair of invocations + assertions to visual/self-validate.sh (mirror the existing golden-good / golden-bad blocks: run the oracle, capture the exit code, assert good rc==0 and bad rc==1).

A new export fixture

  1. Add export/fixtures/my-good.md and export/fixtures/my-bad.md; extend build-fixtures.sh to render them to my-good.pdf / my-bad.pdf.
  2. Verify directly: 
    node export/validate-pdf.js --pdf export/fixtures/my-good.pdf --out /tmp/e --name my-good --expect "…"   # rc 0
node export/validate-pdf.js --pdf export/fixtures/my-bad.pdf  --out /tmp/e --name my-bad  --expect "…"   # rc 1
  3. Add the pair + assertions to export/self-validate.sh.

Rule of thumb: every new fixture ships in a good/bad pair, and the bad one must fail for the specific reason you designed it to — check the verdict.json to confirm the right finding fired, not just that the exit code was non-zero. A red light for the wrong reason is still a bluff.