Skip to content

CLAUDE.md

Latest commit

 

History

History
387 lines (332 loc) · 19.1 KB

CLAUDE.md

File metadata and controls

387 lines (332 loc) · 19.1 KB
Raw

CLAUDE.md - Bench

What This Is

Bench is a constitutional governance layer for Claude Code. Every code change Claude proposes passes through an adversarial brigade of models that challenge, defend, and rule on it before a single line commits. Every verdict is hash-chained into an auditable ledger. This is not a code review tool. This is a judicial system for AI-generated code.

Bench governs itself. Every change to this codebase is subject to the same governance pipeline. This is non-negotiable.

Architecture

PreToolUse Hook -> Challenger (Sonnet) -> Defender (Sonnet) -> Oracle (Opus) -> Ledger
  • Hook intercepts Write/Edit/MultiEdit tool calls
  • Constitution snapshot loaded once per pipeline run (frozen within, hot-reload between)
  • The governed project's CLAUDE.md is read once per run alongside it and passed to all three stages as file_context, on every provider, so the judge's evidence does not vary by transport. It is framed as untrusted repository input that informs scope and cannot waive or amend a constraint. Content over pipeline/runner.py's _MAX_CONTEXT_CHARS (10,000) is truncated, and the framing is prepended after truncation so it always precedes the content.
  • Oracle verdict is PASS or VETO. VETO is binding.
  • Every verdict hashed and chained into bench-ledger.json
  • Ledger destination is project-scoped via ledger.chain.resolve_ledger_path(): every governed project, Bench included, uses <project>/.bench/bench-ledger.json, and BENCH_LEDGER_PATH overrides it. Bench has no exemption. An operational ledger records the full diff body of every change it governs, so publishing one publishes every change it ever saw; it is therefore never a tracked artifact of the repository it governs, and dogfooding does not earn an exception. .bench/ is gitignored. A fresh clone starts with no chain, and its first governed edit opens a new one at GENESIS.
  • Bench's own chain was migrated from ledger/ to .bench/ by copying the legacy segment, ledger-meta.json, and every entry file unchanged, then confirming python -m cli verify reported VALID at the new location with an unchanged genesis hash. No entry was modified, reordered, or removed, so this is a relocation rather than a retirement, which matters because a storage-location change is not a permitted C-008 retirement trigger. Readers (load_ledger, verify_chain, the viewer) resolve through the same function, so the auditor never inspects a different file than the writer appends to.
  • The ledger is stored in two segments. bench-ledger.json is the frozen legacy array: it is read by every reader and never written again, and ledger-meta.json is frozen with it as a permanent pin on that segment's tip and entry count. All new entries are written one per file to <ledger_dir>/entries/<entry_hash>.json via chain.resolve_entries_dir(). A single rewritten-in-full array could not survive two branches appending, so it was frozen rather than migrated: C-008 forbids moving or rewriting an existing entry, and a file that is never written can never conflict.
  • previous_hash therefore holds either a string (legacy, one parent) or a sorted list of parent hashes (new entries). append_entry sets it to compute_tips() — every entry no other entry claims as a parent — so a fork left by a git merge is reconciled by the next governed edit naming both tips.
  • verify_chain enumerates the entries directory itself and fails closed on MISSING_PARENT (a parent hash that resolves to nothing), ORPHAN_ENTRY (an entry unreachable from genesis), DUPLICATE_ENTRY, FILENAME_MISMATCH (the filename must equal the hash it contains), and MULTIPLE_GENESIS. The legacy array keeps its original positional walk at full strength. Nothing written to the entries directory escapes verification.
  • A whole chain may be retired under C-008's single bounded exception, and only when it contains content which must not be published. ledger/retire.py implements it: python -m cli retire --archive-dir PATH --reason TEXT archives every segment that exists, verifies the archive before removing anything, and opens a successor chain whose genesis is an ANCHOR entry recording the predecessor's tip hash, genesis hash, entry count, first and last timestamps, archive path, and reason. python -m cli audit-retirement re-runs that check against the archive. Retirement refuses a chain that does not verify, an empty or forked chain, and any invocation that is not a human at a plain TTY, which means it cannot be run from inside a Claude Code session. A storage-format change is not a permitted trigger; freezing the legacy array was the answer there, not retirement.
  • On VETO: JSON permissionDecision "deny" with remediation feedback
  • On PASS: JSON permissionDecision "allow"
  • Exit code is ALWAYS 0. Flow control is via JSON, not exit codes.

Project Structure

bench/
  bench.json              # Constitution file. User-editable. Versioned.
  .claude/
    settings.json         # Claude Code hook config
  hooks/
    pre-tool-use.py       # Hook entry point
  pipeline/
    challenger.py         # Adversarial analysis (Sonnet)
    defender.py           # Soundness argument (Sonnet)
    oracle.py             # Binding verdict (Opus)
    constitution.py       # Load, snapshot, hash
    runner.py             # Sequential orchestration
  ledger/                 # Code only. No chain data is tracked here.
    chain.py              # Hash-chaining, append
    verify.py             # Independent chain validation
    retire.py             # C-008 chain retirement: archive, anchor, audit
    migrate.py            # One-time upgrade for pre-privacy clones
    sanitize.py           # Validation and audit for published-copy removal
    attestation.py        # Public checkpoint export: commitments, no content
  .bench/                 # Operational chain. Gitignored, never committed.
    bench-ledger.json     # Frozen legacy chain segment (read, never written)
    ledger-meta.json      # Frozen pin on that segment's tip and count
    entries/              # One JSON file per new entry, named <entry_hash>.json
  cli/
    __main__.py           # python -m cli
    commands.py           # verify, ledger, stats, constitution, viewer,
                          # retire, audit-retirement, record-sanitation,
                          # audit-sanitation, verify-sanitation-binding,
                          # verify-purge, migrate-ledger, attest
  utils/
    diff.py               # Diff extraction and formatting
    api.py                # Anthropic API client
    formatting.py         # Stdlib diff-info formatting for pipeline display
    stats.py              # Shared ledger stats helpers (CLI + viewer)
    viewer.py             # Self-contained HTML ledger viewer
  tests/

Models

Role Constant (in utils/api.py) Purpose
Challenger CHALLENGER_MODEL Find problems in proposed change
Defender DEFENDER_MODEL Argue soundness of the change
Oracle ORACLE_MODEL Issue binding PASS or VETO
Utility UTILITY_MODEL Reserved for future summarization (utils/formatting.py is currently stdlib-only)

utils/api.py is the single source of truth for model IDs. This document names the constants rather than restating version strings, so a model change is a single-file edit to utils/api.py and cannot drift from the docs. Each constant holds the exact first-party Anthropic model ID. For current-generation models these are bare aliases (for example claude-sonnet-5, claude-opus-4-8), which are complete as-is; a dated suffix is used only for models that publish dated snapshots (as with the reserved UTILITY_MODEL). Confirm each ID resolves on the target provider(s) before shipping.

Models are Anthropic by default. The wrapper in utils/api.py also supports OpenRouter as a routing backend (selected via the BENCH_PROVIDER env var) so the same Anthropic models can be reached through either path. Direct calls to non-Anthropic model families remain out of scope.

Rules

Absolute

  1. Every file change made through Claude Code's Write/Edit/MultiEdit tools is governed. No exceptions and no bypasses on that path. Other tool paths are not governed at all, which is a boundary to state rather than a gap to assume closed. Files written through Bash (redirection, tee, sed -i, a heredoc) and through any MCP server's tools never reach the PreToolUse hook, so they carry no verdict and no ledger entry. Do not add Bash to the matcher to try to close this: utils.diff.build_diff_info returns no payload for those tools, so the Challenger rejects the input and every such call hard-fails into a VETO, denying ordinary shell work outright. Audit an MCP server's package before registering it in a governed project — that raises confidence in one version at one point in time, and it is not governance. A further category sits outside it and cannot be brought in: a bot-authored dependency PR, where Dependabot edits requirements.txt on GitHub, never reaches the PreToolUse hook, so it merges without a verdict or a ledger entry. Merge those by deliberate human decision after reading the diff, and do not enable auto-merge on them. That last point is project policy, not constitutional text: C-007 binds changes to the pipeline files themselves, so it does not literally reach a GitHub merge setting, though auto-merge would defeat the same verification it exists to protect.
  2. The ledger is append-only. Never modify, delete, or overwrite entries.
  3. The hash chain must remain intact. Every entry references the previous.
  4. Constitution is loaded as a snapshot per pipeline run. All three stages see the same version. No mid-run constitution changes.
  5. Oracle verdicts are binding. VETO means the change does not land.
  6. Exit code from the hook is ALWAYS 0. Use JSON permissionDecision for flow control. Exit-2 causes Claude to stall.

Code Standards

  1. Python 3.11+. Type hints on all function signatures.
  2. All API calls wrapped in try/except with typed error returns.
  3. No silent error swallowing. Catch blocks must log, re-throw, or return a typed error. This is also constitutional constraint C-001.
  4. No undeclared dependencies. Every import has a corresponding entry in requirements.txt.
  5. JSON output from all pipeline stages. No free-form text responses.
  6. All structured output validated before use. Parse failures retry once, then record as PIPELINE_ERROR in the ledger.

Workflow

  1. Do not run tests with npm test or pytest in bulk. Test specific files or functions only.
  2. One change per tool call. Do not batch unrelated changes into a single Write/Edit operation.
  3. If you modify bench.json (the constitution), increment the version field.
  4. If you modify any file in pipeline/, ledger/, or hooks/, you are modifying the governance pipeline itself. Constraint C-007 applies. Be aware that Bench will scrutinize these changes.
  5. Commit messages follow: [bench] <component>: <what changed> Examples: [bench] oracle: add confidence scoring [bench] ledger: implement chain verification [bench] constitution: add C-009 logging constraint

Code Navigation: graphify

A knowledge graph of this repo lives in graphify-out/ (graph.json, GRAPH_REPORT.md, graph.html), built by the graphify skill. For structural questions (what calls X, what depends on Y, trace a data flow), query the graph before reaching for grep:

graphify query "<question>"   # BFS context; --dfs to trace, --budget N to raise the output cap
graphify path "A" "B"         # shortest path between two symbols, edges tagged with provenance
graphify explain "<node>"     # plain-language explanation of one node

The graph replaces exploratory grepping, not verification: read the cited files before editing, and use grep for exact strings or anything newer than the last build. After a stretch of commits, refresh with /graphify . --update. Do not use graphify claude install here; it writes this file directly, bypassing governance. Edit this section through governed tools like any other change.

Constitution Reference

The constitution lives in bench.json. Current constraints:

  • C-001: No silent error swallowing (veto)
  • C-002: Scope boundary enforcement (veto)
  • C-003: Dependency declaration (veto)
  • C-004: Type safety preservation (veto)
  • C-005: Test coverage for new logic (warning)
  • C-006: No hardcoded secrets (veto)
  • C-007: Governance pipeline integrity (veto)
  • C-008: Ledger immutability (veto). Its one bounded exception, chain retirement, is implemented in ledger/retire.py and validated by ledger.retire.validate_anchor, so a retirement that omits any element C-008 enumerates is refused rather than trusted. It is not a general-purpose reset: the sole permitted trigger is unpublishable content.

These are the core, and they are a floor. When Bench governs another project, pipeline.constitution.load_governing_constitution() stacks that project's <project>/bench.json on top: the project may add constraints in the reserved P- namespace and raise a core severity via severity_overrides, and may not remove, downgrade, restate, or redefine anything in C-. Attempts to do so raise and fail closed rather than being ignored. Governing Bench itself uses the core alone. BENCH_CONSTITUTION_PATH overrides which layer is stacked.

Readers resolve through that same function — python -m cli constitution prints the merged result and its sources — so the auditor never displays a different constitution than the pipeline enforced. Each ledger entry records constitution_sources (layer, path, and raw hash per file), and the entry's constitution_hash chains those raw hashes rather than digesting a merged re-serialization.

API Configuration

The LLM wrapper lives at utils/api.py and exposes a single call_model(model, system_prompt, user_content, max_tokens=...) -> dict function (the max_tokens default is defined in utils/api.py, the single source of truth). The provider is selected at call time by the BENCH_PROVIDER environment variable; the function signature is identical across all backends.

# Provider: anthropic (default if BENCH_PROVIDER is unset)
import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

# Provider: openrouter (BENCH_PROVIDER=openrouter)
import openai
client = openai.OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)
# Model strings are routed to their OpenRouter slug on this path. Most map to
# "anthropic/<id>", but ids whose OpenRouter slug differs (e.g. the dotted
# "anthropic/claude-opus-4.8") are translated via a small map in utils/api.py.
# The openai SDK is a soft dependency — install it only if you set
# BENCH_PROVIDER=openrouter.

# Provider: claude_code (BENCH_PROVIDER=claude_code) — no API key.
# Dispatches each stage through the local `claude` CLI in headless mode
# (`claude -p --output-format json --model ...`, with the system prompt and
# user content folded into the stdin payload) so calls ride the user's Claude
# Code subscription.
# The child is spawned with BENCH_SUBPROCESS=1 so Bench's own PreToolUse hook
# fails open instead of recursing. subprocess/shutil are stdlib, so this path
# adds no dependency. Per-stage timeout is BENCH_CLAUDE_TIMEOUT seconds
# (library default 120). This repo sets 300 in .claude/settings.json: at 120
# the Oracle timed out on constitutionally heavy diffs, and because a timeout
# fails closed it surfaced as a VETO carrying pipeline_error, which reads like
# a ruling but is a pipeline failure. Raising the ceiling does not weaken
# enforcement; it stops a slow judge from being mistaken for a strict one.

# Model strings live in utils/api.py as CHALLENGER_MODEL, DEFENDER_MODEL,
# ORACLE_MODEL, and UTILITY_MODEL (the single source of truth). This section
# does not restate the literal IDs, so they cannot drift. Routing handles any
# prefix (the openrouter path prepends "anthropic/").

Hook Response Format

On PASS:

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "allow",
    "additionalContext": "Bench governance: PASS. All constraints satisfied."
  }
}

On VETO:

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "BENCH VETO [C-XXX]: ...",
    "additionalContext": "Remediation: ..."
  }
}

What Success Looks Like

Bench builds Bench. Every change in this repo was challenged, defended, ruled on, and recorded. The ledger is the proof. python -m cli verify confirms the chain is intact. python -m cli stats shows the full governance history.

The thesis: governance of AI reasoning is a primitive, not a feature.

Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed.

Tradeoff: These guidelines bias toward caution over speed. For trivial tasks, use judgment.

  1. Think Before Coding Don't assume. Don't hide confusion. Surface tradeoffs.

Before implementing:

State your assumptions explicitly. If uncertain, ask. If multiple interpretations exist, present them - don't pick silently. If a simpler approach exists, say so. Push back when warranted. If something is unclear, stop. Name what's confusing. Ask. 2. Simplicity First Minimum code that solves the problem. Nothing speculative.

No features beyond what was asked. No abstractions for single-use code. No "flexibility" or "configurability" that wasn't requested. No error handling for impossible scenarios. If you write 200 lines and it could be 50, rewrite it. Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.

  1. Surgical Changes Touch only what you must. Clean up only your own mess.

When editing existing code:

Don't "improve" adjacent code, comments, or formatting. Don't refactor things that aren't broken. Match existing style, even if you'd do it differently. If you notice unrelated dead code, mention it - don't delete it. When your changes create orphans:

Remove imports/variables/functions that YOUR changes made unused. Don't remove pre-existing dead code unless asked. The test: Every changed line should trace directly to the user's request.

  1. Goal-Driven Execution Define success criteria. Loop until verified.

Transform tasks into verifiable goals:

"Add validation" → "Write tests for invalid inputs, then make them pass" "Fix the bug" → "Write a test that reproduces it, then make it pass" "Refactor X" → "Ensure tests pass before and after" For multi-step tasks, state a brief plan:

  1. [Step] → verify: [check]
  2. [Step] → verify: [check]
  3. [Step] → verify: [check] Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.

These guidelines are working if: fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.