v0.18.11.0 feat: Track 2B (audit cutover) + Track 4C (skill prose corpus) — suite 113s→13s (#63)

* chore: gitignore node_modules

* feat(test): Track 4C — LLM-as-judge for skill prose (v0.18.10.0)

`tests/helpers/llm-judge.ts` exposes `callJudge<T>(prompt, validator)` —
an Anthropic SDK wrapper hardened against the failure modes the eng-review
surfaced: maxRetries:0 (no compounding retries), one explicit 1× 429 retry,
stop_reason check before regex extract, baked-in validator, strict
isJudgeScore predicate (rejects NaN/Infinity/decimals/0/6/null/wrong-types/
empty-reasoning), pinned Sonnet model + temperature:0 + max_tokens:1024.

`tests/llm-judge.test.ts` runs 13 mocked-Anthropic-client unit tests
covering every helper branch: happy path, stop_reason mismatch, no-JSON,
JSON.parse failure, validator rejection, 429 retry success, non-429
no-retry, plus 6 isJudgeScore cases.

`tests/skill-llm-eval.test.ts` is the paid evaluation, gated on
`process.env.EVALS === '1'` (exact match) plus a strict ANTHROPIC_API_KEY
check that throws if EVALS=1 without a key. Sequential test.each over
4 fixtures in `tests/fixtures/skill-prose-corpus/` (3 positive: roadmap-
reassessment, test-plan-extraction, pair-review-test-list; plus 1 shallow
negative-control). Each fixture is markdown with YAML frontmatter carrying
provenance (source skill commit, repo commit, input prompt, model, UTC
timestamp, worktree state). Positive fixtures must score ≥3 on every axis;
negative-control must score ≤2 on at least one axis (catches the failure
mode where the judge rewards plausible-sounding prose over substance).
60s per-test timeout. Cost ~\$0.05–0.15 per EVALS=1 run.

Touchfiles map gains `tests/skill-llm-eval.test.ts → tests/fixtures/
skill-prose-corpus/**` so fixture edits select only the eval test.
CLAUDE.md ## Testing documents the EVALS=1 contract. PROGRESS row +
CHANGELOG entry capture the locked decisions and codex catches.
@anthropic-ai/sdk added as devDependency; bun.lock committed.

Source: /plan-eng-review session in workspace valletta (kbitz/llm-judge-
skill-prose, 2026-05-03), 14 decisions resolved, 10 codex catches batch-
applied. Tool-use migration + judge-floor tightening 3→4 deferred to
ROADMAP Future. The 4-fixture corpus is initial-synthesized (each
fixture's provenance documents this); replace with real captures as the
skills produce them.

* feat: Track 2B — cut bin/roadmap-audit over to TS implementation

Track 2A (v0.18.6.0) shipped the TS port at src/audit/** as "dark code"
with full byte-parity verified by tests/audit-shadow.test.ts. The binary
at bin/roadmap-audit was never wired up, so the snapshot suite kept paying
the bash cost — and tests/audit-shadow.test.ts paid it a second time per
fixture for parity verification.

Cutover: bin/roadmap-audit becomes a 7-line POSIX-sh shim
  exec bun "$(dirname "$0")/../src/audit/cli.ts" "$@"

tests/audit-shadow.test.ts deleted — parity check is obsolete when the
runner IS the oracle. The 3,868-line bash binary is preserved in git
history at e4f883b (PR #58, the original "dark code" commit).

Manual touchfile entries for tests/{audit-snapshots,audit-cli-contract}
.test.ts gain src/audit/** so audit code edits retrigger the snapshot
suite under diff selection (the static import graph doesn't connect them
because the tests spawn the binary rather than importing).

Measured impact on a 2-core MacBook:
  tests/audit-snapshots.test.ts   111s → 7.3s   (~15×)
  full suite (bun run test:full)  113s → 32s    (~3.5×)
  test count: 828 → 805 (delta = the 23 deleted shadow parity tests)

Phase 1's stated end-state ("bin/roadmap-audit is a compiled bun binary")
is now met — shim form rather than `bun build --compile` artifact, but
equivalent semantics with simpler deploy (no per-platform binary, source
grep-able at src/audit/**, instant dev iteration without rebuilds).

ROADMAP.md gains "Track 2B: Cut bin/roadmap-audit over to TS implementation
✓ Complete" under Group 2.

* refactor: drop SDK eval path, replace with in-session routing rule

The Track 4C eng-review locked an SDK-driven judge path: tests/helpers/
llm-judge.ts (Anthropic SDK wrapper), tests/llm-judge.test.ts (mocked
units), tests/skill-llm-eval.test.ts (paid eval gated on EVALS=1 +
ANTHROPIC_API_KEY). The plan was implemented and reviewed during /ship,
then reconsidered:

- The eval test only runs on machines that already have `claude`
  authenticated. It never runs in CI.
- Requiring a separate ANTHROPIC_API_KEY plus a separate Anthropic SDK
  dependency adds setup friction with no ecological payoff — the user
  is the only person running it, on a machine that's already paying for
  Claude via Claude Code.
- The corpus + rubric retain their value as calibration material; the
  test infrastructure around them does not.

Dropping:
  - tests/helpers/llm-judge.ts (callJudge helper, 173 LOC)
  - tests/llm-judge.test.ts (13 mocked unit tests, 135 LOC)
  - tests/skill-llm-eval.test.ts (paid eval test, 165 LOC)
  - @anthropic-ai/sdk devDependency
  - bun.lock (no deps remaining)
  - tests/skill-llm-eval.test.ts touchfiles entry

Keeping:
  - tests/fixtures/skill-prose-corpus/ (4 reference fixtures, full
    YAML-frontmatter provenance preserved)
  - The three-axis rubric (clarity / completeness / actionability,
    1–5 each, ≥3 on positives, ≤2 on at least one axis for the control)

Adding (CLAUDE.md ## Testing):
  Routing rule — when skills/*.md is edited in a session, Claude
  proactively recommends judging the changed prose against the corpus
  and rubric in-session. No new code, no new dep, no new env var, no
  separate billing — uses the existing Claude Code session with prompt
  caching across fixtures.

Stale Future entries removed:
  - "Migrate callJudge to tool-use forced JSON" (no callJudge to migrate)
  - "Raise judge floor 3 → 4 after 5–10 EVALS runs" (no scheduled run
    to retune)

ROADMAP Track 4C entry rewritten to reflect the actual shipped shape
and marked ✓ Complete. CHANGELOG and PROGRESS rows rewritten.

Suite runtime: 32s → 13.6s (the SDK was apparently loading 18s of
transitive deps just to skip the test).

Total impact across the PR (Track 2B + Track 4C combined): full bun
test:full was 113s before the cutover, now 13.6s on the same hardware
with one less dep.

Author: kbitz

.gitignore

@@ -1,2 +1,5 @@
 # Conductor context
 .context/
+
+# Bun / Node
+node_modules/

CHANGELOG.md

@@ -2,6 +2,74 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.18.11.0] - 2026-05-04
+
+### Changed (Track 2B — bin/roadmap-audit cutover to TS)
+
+`bin/roadmap-audit` is now a 7-line POSIX-sh shim that invokes
+`src/audit/cli.ts` via `bun`. Track 2A (v0.18.6.0) shipped the TS port
+with full byte-parity verified by `tests/audit-shadow.test.ts` but never
+wired the binary up — the test ran every snapshot fixture twice (bash
+oracle + TS shadow) and the snapshot suite continued paying bash cost.
+The cutover makes the TS implementation the single source of truth.
+`tests/audit-shadow.test.ts` is deleted (parity check is obsolete: oracle
+and runner are now the same code). Manual touchfile entries for
+`tests/audit-snapshots.test.ts` and `tests/audit-cli-contract.test.ts`
+gain `src/audit/**` so audit code edits retrigger the snapshot suite via
+diff selection. The 3,868-line bash binary is preserved in git history
+at commit `e4f883b` (PR #58) for archaeology — no recovery path needed
+since parity was verified at the time of port.
+
+**Measured impact on a 2-core MacBook:**
+`tests/audit-snapshots.test.ts` 111s → 7.3s (~15×). Full suite via
+`bun run test:full` 113s → 32s (~3.5×). Phase 1's stated end-state
+("`bin/roadmap-audit` is a compiled bun binary") is met (shim form, not
+`bun build --compile` artifact — equivalent semantics, simpler deploy:
+no per-platform binary, source is grep-able at `src/audit/**`). The
+remaining ~32s suite is dominated by `tests/update.test.ts` (~11s,
+shells out to `bin/update-{check,run}` and the bash `setup` script) —
+that's the next leverage point but lives in Group 5's install pipeline
+scope, not Group 2.
+
+### Added (Track 4C — Skill prose corpus + in-session judging routing rule)
+
+`tests/fixtures/skill-prose-corpus/` ships 4 markdown fixtures with rich
+YAML-frontmatter provenance: 3 positive examples representing what good
+output looks like for `/roadmap` reassessment, `/test-plan` extraction,
+and `/pair-review` test-list generation, plus a `4-shallow-control`
+negative example that's deliberately vague and generic. Each fixture
+records its source skill commit, the repo commit at capture time, the
+input prompt that produced the prose, the generation model, a UTC
+timestamp, and the worktree state — so future replacements with
+real-captured runs can match the structural contract.
+
+The corpus is **reference material, not a test fixture.** There is no
+automated judge running under `bun test`. The eng-review's original SDK
++ `EVALS=1` + `ANTHROPIC_API_KEY` pattern was reconsidered during /ship
+and dropped: the test only runs on machines that already have `claude`
+authenticated (it never runs in CI), so requiring a separate API key
+plus a separate Anthropic SDK dependency was setup friction with no
+ecological payoff. Instead, `CLAUDE.md ## Testing` adds a routing rule:
+when `skills/*.md` is edited in a session, Claude proactively recommends
+judging the changed prose against the corpus and the three-axis rubric
+(clarity / completeness / actionability, 1–5 each, ≥3 expected on
+positives, ≤2 expected on at least one axis for the control) in-session.
+Cost is whatever the existing Claude Code session is paying anyway with
+prompt caching across fixtures; cadence is opportunistic (only on real
+prose changes), not scheduled.
+
+This shape moves the workflow from "rare paid test that nobody runs" to
+"automatic prompt at the moment that warrants it" while keeping the
+fixtures + rubric as the calibration anchor. The two follow-ups the
+eng-review deferred to `Future` (tool-use migration, judge-floor
+tightening 3 → 4) are obsolete in this shape: there's no judge code to
+migrate and no scheduled run to retune.
+
+### Changed
+
+`node_modules/` added to `.gitignore` (was previously leaking into
+untracked file lists during /ship cycles).
+
 ## [0.18.10.0] - 2026-05-04
 
 ### Added (Track 4D — Audit-compliance test for gstack-extend invariants)

CLAUDE.md

@@ -36,6 +36,8 @@ git diff tests/fixtures/source-tag-hash-corpus.json   # review hash drift
 
 `scripts/score-extractor.ts` is a manual harness for scoring `/test-plan` extractor JSON output against vendored fixtures (`tests/fixtures/extractor-corpus/`). Run via `bun scripts/score-extractor.ts --help`.
 
+`tests/fixtures/skill-prose-corpus/` is a reference corpus of skill output prose — 3 positive examples (`/roadmap` reassessment, `/test-plan` extraction, `/pair-review` test list) plus a `4-shallow-control` negative control. There is no automated judge test; the corpus is calibration material for the rubric (clarity / completeness / actionability, 1–5 each, ≥3 expected on positives, ≤2 expected on at least one axis for the control). When skill prose changes — i.e. any edit to `skills/*.md` in a session — Claude should proactively recommend judging the changed prose against the corpus and rubric in-session. Sample request: "score the new skill prose against tests/fixtures/skill-prose-corpus and the rubric." This runs in the existing Claude Code session (no API key, no separate billing, prompt caching across fixtures), keeps the human in the loop for taste decisions, and skips the cost overhead of an SDK-driven test that would only run on this machine anyway.
+
 ## Skill routing
 
 When the user's request matches an available skill, ALWAYS invoke it using the Skill

VERSION

@@ -1 +1 @@
-0.18.10.0
+0.18.11.0