Self-validating web QA harness. Two independent oracles that catch the failures automated UI tests routinely miss, and that prove they still work every time you run them:
- Visual-proof (
visual/) — renders any HTML file or URL in Chromium in both light and dark themes, captures a full-page screenshot per theme, and runs a layout oracle that flags element overlap, off-screen / clipped controls, and unbounded "giant" widgets — plus an OCR legibility signal. Constitution §11.4.170. - Exported-document validator (
export/) — given a PDF, checks that the exported document has real content (text extraction + required phrases), is textually clean (no raw Markdown / Mermaid source leaking as body text), and is visually faithful (embedded images present, rendered pages OCR to legible text). Constitution §11.4.168.
Both ship with golden-good and golden-bad fixtures and a self-validate.sh that
asserts golden-good PASSes and golden-bad FAILs. If either assertion breaks, the harness
itself is broken — and it says so, loudly.
Most UI test suites bluff. They assert on values and tokens that are trivially satisfiable and tell you nothing about whether a human can actually use the screen:
expect(element).toHaveText("Confirm")passes even if the button is rendered 2400px wide, off-screen, or stacked on top of another control.expect(pdf).toContain("Release Q4")— wait, most suites don't even check the exported artifact; they check the DOM that produced it, then trust the exporter.- A snapshot of a JSON blob or a class list is green while the page is visually broken.
Those tests are token-equality theater. They match strings, not reality. A change that
destroys the layout, clips the primary action, or leaks raw \``mermaid gantt```` source
into a PDF sails through — because nothing looked at the rendered pixels or the
exported bytes.
This harness refuses that game. Its rules:
- UI proof must come from the rendered artifact, never from a value or token. The visual oracle inspects the geometry of the live rendered page (bounding boxes, viewport containment, overlap area) and the actual screenshot (OCR). The export validator inspects the actual PDF bytes (pdftotext / pdfimages / rasterize→OCR).
- Value-equality and token-equality assertions are forbidden as UI proof. Checking that a string equals a string, or that an element "has" some text/class, is not evidence that the UI works. It is explicitly not what this harness does, and it is the failure mode this harness exists to replace.
- The harness proves itself. Every run is guarded by golden fixtures: a
known-good input that must PASS and a known-broken input that must FAIL. A harness
that can only ever say PASS is a bluff too.
self-validate.shmakes the harness put up or shut up. - Missing tools SKIP honestly — they never fake a PASS. If
tesseractorpdfimagesis absent, the relevant check reportsSKIPwith a reason. It is never silently upgraded to green.
Each oracle carries two fixtures:
| Fixture | Visual harness | Export harness | Must yield |
|---|---|---|---|
| golden-good | a clean card UI, correct in light & dark | clean PDF: full table, embedded figure, no markup leak | PASS |
| golden-bad | overlapping labels + clipped button + 2400px giant button | PDF with raw Mermaid gantt source leaked, truncated table, no figure | FAIL |
self-validate.sh runs the oracle against both and asserts good → rc 0 and
bad → rc 1. This is a differential test: it proves the oracle can both accept
correct output and reject broken output. An oracle that always passes (the classic
bluff) fails self-validation because golden-bad would wrongly PASS.
Captured on macOS (Apple Silicon), Node v22, with all system tools present. Full logs and
verdict JSON are committed under evidence/self-validation/.
§11.4.170 VISUAL-PROOF HARNESS — SELF-VALIDATION
>>> GOLDEN-GOOD (expect PASS)
- light : PASS (errors=0 warnings=0) png=golden-good.light.png
- dark : PASS (errors=0 warnings=0) png=golden-good.dark.png
>>> GOLDEN-BAD (expect FAIL)
- light : FAIL (errors=5) ... giant-widget, offscreen-clipped x2, overlap x2
- dark : FAIL (errors=5) ... giant-widget, offscreen-clipped x2, overlap x2
ASSERTIONS
PASS: golden-good verdict=PASS (rc=0)
PASS: golden-bad verdict=FAIL (rc=1)
SELF-VALIDATION RESULT: PASS (good=PASS, bad=FAIL as required)
§11.4.168 EXPORTED-DOCUMENT VALIDATOR — SELF-VALIDATION
>>> GOLDEN-GOOD (expect PASS)
[PASS] CONTENT/content.nonempty — extracted 106 words (min 40)
[PASS] CONTENT/content.faithful[Shipped]
[PASS] CONTENT/content.faithful[Release Q4]
[PASS] TEXTUAL/textual.no-leak — no raw markup / mermaid source in body text
[PASS] FULL-VISUAL/visual.images — pdfimages found 1 embedded image(s) (min 1)
[PASS] FULL-VISUAL/visual.ocr — OCR of 2 pages recovered 109 legible words (min 15)
>>> GOLDEN-BAD (expect FAIL)
[FAIL] CONTENT/content.faithful[Shipped] — required phrase MISSING (truncated)
[FAIL] CONTENT/content.faithful[Release Q4] — required phrase MISSING (truncated)
[FAIL] TEXTUAL/textual.no-leak — mermaid:gantt, mermaid:dateFormat, :done, :active, :crit leaked
[FAIL] FULL-VISUAL/visual.images — pdfimages found 0 embedded image(s) (min 1)
ASSERTIONS
PASS: golden-good verdict=PASS (rc=0)
PASS: golden-bad verdict=FAIL (rc=1)
SELF-VALIDATION RESULT: PASS (good=PASS, bad=FAIL as required)
Honesty note on SKIPs: on a host missing a system tool (e.g.
tesseract), the corresponding check printsSKIPwith a reason rather than PASS. In the environment that produced the evidence above, every tool was present, so there were no SKIPs. Reproduce it yourself withnpm run self-validate.
npm install # pulls @playwright/test + @axe-core/playwright and, via postinstall,
# downloads the Playwright-managed ChromiumThen install the system tools the export validator and OCR need — see tool dependencies below.
# prove the harness works (golden-good PASS, golden-bad FAIL)
npm run self-validate
# run the visual oracle on your own page
node visual/visual-oracle.js --input ./my-page.html --out ./out --name my-page \
--viewport 1280x800 --themes light,dark
# run the PDF validator on your own document
node export/validate-pdf.js --pdf ./my-doc.pdf --out ./out --name my-doc \
--min-words 40 --min-ocr-words 15 --min-images 1 --expect "Total" --expect "Signed"Full options, the long-page caveat, and CI wiring are in USAGE.md. The golden fixtures and how to add your own are in FIXTURES.md.
anti-bluff-web-harness/
├── visual/ §11.4.170 host-rendered visual-proof
│ ├── visual-oracle.js renders light+dark, detects overlap/clip/giant + OCR
│ ├── self-validate.sh golden-good PASS + golden-bad FAIL assertions
│ └── fixtures/
│ ├── good.html clean card UI (must PASS)
│ └── bad.html overlap + clipped + giant defects (must FAIL)
├── export/ §11.4.168 exported-document validator
│ ├── validate-pdf.js content + textual + full-visual checks
│ ├── build-fixtures.sh pandoc -> weasyprint golden PDFs
│ ├── make-image.js raster figure for the good PDF (Chromium)
│ ├── self-validate.sh golden-good PASS + golden-bad FAIL assertions
│ └── fixtures/
│ ├── good.md / bad.md fixture sources
│ ├── figure.png embedded raster
│ └── golden-good.pdf / golden-bad.pdf
├── evidence/self-validation/ committed proof (logs, verdict JSON, screenshots)
├── package.json
├── USAGE.md FIXTURES.md LICENSE
The Node deps (@playwright/test, @axe-core/playwright) install via npm install, which
also downloads a self-contained Chromium. The following system tools are required for
the full feature set; missing ones cause honest SKIPs, not fake passes.
| Tool | Package | Used by | If missing |
|---|---|---|---|
| Chromium | bundled by Playwright (npm install) |
visual oracle, make-image.js |
visual oracle cannot run |
poppler (pdftotext, pdfimages, pdftoppm) |
poppler / poppler-utils |
export validator | content/textual/visual checks SKIP |
| tesseract | tesseract / tesseract-ocr |
OCR in both harnesses | OCR checks SKIP |
| pandoc | pandoc |
rebuilding export fixtures | build-fixtures.sh cannot run |
| weasyprint | weasyprint |
rebuilding export fixtures (PDF engine) | build-fixtures.sh cannot run |
Install examples:
# macOS (Homebrew)
brew install poppler tesseract pandoc weasyprint
# Debian / Ubuntu
sudo apt-get install -y poppler-utils tesseract-ocr pandoc weasyprintVerified tool versions in the evidence run: Node v22.22.3, poppler/pdftotext 26.06.0, tesseract 5.5.2, pandoc 3.9.0.2, WeasyPrint 66.0.
MIT.