test(engine): token-shape regression harness for cache/cost invariants

[mgoldsborough]

Why

Token-usage regressions keep recurring (the #403/#405/#406/#408 cache work), and each fix has landed without a guardrail behind it. The reason they recur: the request shape the engine sends to the provider isn't locked anywhere. Yet the expensive bugs — cache-write thrash, a per-turn hint appended to the system prompt busting its 1h breakpoint and the whole prefix, an anchor sliding off the prior tail — are all properties of that request, which is fully deterministic given a fixed conversation. So they're catchable with a scripted model and zero API calls.

What

One harness, two tiers.

test/helpers/recording-model.ts — a LanguageModelV3 decorator that records the exact post-cache-policy prompt + tools at the provider boundary (what actually goes on the wire).

test/helpers/token-shape.ts — deriveShape() (compact per-step fingerprint) + checkInvariants() (named violations). Provider-aware (anthropic vs passthrough).

test/unit/token-shape.test.ts (+ committed golden) — runs the real engine loop through a recording model on a scripted tool-loop and asserts:

• system-stable — system block byte-identical every step (guards the system-append regression, fix(engine): deliver the final-step hint as a tail message, not a system append #408)
• anchor-chaining — anchor of call N+1 == prior call's tail (test(engine): cover length-continuation anchor in cache-policy (follow-up to #401) #405)
• anti-thrash-bounded — post-anchor write delta stays flat across the run
• ttl tiering (1h stable prefix / 5m rolling), prefix-monotonic, passthrough-no-cache

The golden is the artifact: promptTokens climbs every step (80 → 153 → 226…) while deltaTokensAfterAnchor stays pinned at 73. Prompt grows, per-turn cache-write flat — the optimization, locked. A regression moves those numbers into a reviewable diff. Regenerate intentional changes with TOKEN_SHAPE_UPDATE=1.

test/eval/token-shape.eval.test.ts (Tier 3) — the same scenario against real Anthropic/OpenAI/Google, reusing the same harness to verify invariants under real model behavior, plus realized cacheRead/cacheWrite health checks and a greppable metrics line for weekly drift tracking. Skips gracefully per provider when keys are absent.

Validation

• bun run test:unit — 3419 pass (incl. 5 new)
• format / lint / tsc clean
• Negative control: removing the rolling anchor (simulating the pre-test(engine): cover length-continuation anchor in cache-policy (follow-up to #401) #405 bug) turns the unit suite red on anchor-chaining (10×), the golden, and anti-thrash; reverting returns it to green. The guard isn't vacuous.

Notes

• Unit tier is the per-PR fence; the eval is run-on-a-cron (real tokens, provider nondeterminism).
• Building it surfaced one real thing: a naive uniform scenario trips the loop supervisor (identical repeated tool calls → tool released), so the scenario varies the query per step like a real agent.