Skip to content

README.md

Latest commit

 

History

History
77 lines (55 loc) · 3.33 KB

README.md

File metadata and controls

77 lines (55 loc) · 3.33 KB
Raw

world-builder-audit

Standalone audit toolkit for procedural world-builder gyms.

📄 Specification document: World-builder gyms & their audits.

The single organizing lens is the audits rubric (audit_rubric.yaml): a set of quality audits — each a concrete question the audit asks about a target gym — grouped into categories. An interactive viewer browses the rubric (and, when present, overlays a session's per-audit verdicts).

Quick start

# 1. Install
uv sync

# 2. Provide credentials for the semantic mechanism (one of):
export ANTHROPIC_API_KEY=sk-ant-…    # an API key, or
ant auth login                        # an OAuth profile the SDK reads automatically

# 3. Audit a gym, then serve the viewer — in one command.
#    Scores the gym against audit_rubric.yaml (writes a session under sessions/),
#    then starts the viewer. The ONLY prompt is "Press Enter to open in your browser".
uv run ./audit.py /path/to/some-gym       # http://127.0.0.1:8765

The gym can be a local path or a git URL. Omit it to reuse the last-audited gym (.last_target) with no prompt; on the very first run with none set, you're asked once.

The audit is semantic by default: Claude reads the target gym's code and judges each audit present / partial / absent with file:line evidence — the honest mechanism. It uses claude-opus-4-8 (override with --model or AUDIT_MODEL). Split the steps if you prefer: uv run ./audit.py audit [gym] (audit only) and uv run ./audit.py viewer (viewer only).

For a fast, LLM-free run (no key, byte-reproducible, over-crediting upper bound), pass --deterministic: uv run python scripts/run_audit.py --deterministic.

The audits rubric

audit_rubric.yaml is the source of truth:

The rubric is domain-agnostic — 32 audits across 6 categories + 2 meta groups.

  • categories[] — Placeholder & interpolation hygiene, Reward declaration shape validity, Reward naming, pairing & contradictions, Time-sensitivity, Scope & guard typing, Structural quality signals.
  • meta_audits[] — Suppression discipline, Linter infrastructure.
  • Each audit has: id, name, what (the property checked), why (why it matters), and baseline_impl (a pointer to the reference implementation in a baseline gym).

What's inside

File / dir Purpose
audit_rubric.yaml The rubric — categories + meta audits, the audit's only lens
viewer/ FastAPI server + single-page audits viewer
sessions/ Audit results land here, one folder per session (04-scored.json)
audit.py Launcher: sets the target gym and opens the viewer

Sessions

A session is a folder under sessions/<id>/ containing 04-scored.json. The viewer's session selector overlays per-audit verdicts when the file carries a audit_evaluations map (keyed by audit id, each { level, evidence[] }); otherwise the viewer shows the rubric alone.

Repo conventions

  • Audit results NEVER touch the target gym. Sessions live under sessions/ in THIS repo.
  • The rubric is authored to be gym-agnostic — audits are phrased as questions, and baseline_impl points at a reference gym's implementation for grounding.

License

Proprietary — © Patronus AI.