feat(website): Claude Code / Fable 5 graph benchmark lane (N=1)

[samchon]

Intent

Add a Claude Code / Fable 5 (claude-fable-5) lane to the @ttsc/graph agent-cost benchmark, so the dashboard's model axis covers Anthropic's newest tier. Fable 5 is priced far above every existing lane, so the sweep ran at N=1 by design and every cell is recorded with runs: 1 preserved, per the benchmark skill's N=1 rule. Existing N=5 lanes are untouched.

Filed alongside #301 (charts + early observations).

Scope

• website/public/benchmark/graph.json — 80 new claude-code-fable cells (8 repos × 2 prompt families × baseline + ttsc-graph/codegraph/codebase-memory/serena), modelVersion: claude-fable-5.
• website/src/components/benchmark/GraphBenchmark.tsx — display the new lane as Claude Code / Fable 5 (label case, version parser, model ordering above Opus).
• experimental/benchmark/graph/agent-ab.mjs — modelVersion now picks the claude-* usage entry with the most output tokens; the previous first-key pick could stamp cells with the haiku title-generation helper's id (observed in a probe on this run).

Measurement integrity

• Baseline arm: 16/16 cells, every sample manually inspected per the benchmark rule — all ok, web: 0, 6–21 source touches, line-number citations spot-verified against fixtures.
• Graph arms: 64/64 first-attempt, zero retries. ttsc-graph and codegraph made MCP calls in all 16 cells each (medians 1 and 2). codebase-memory (13/16) and serena (14/16) were mostly ignored in favor of shell/source fallback — a real measured outcome kept visible in the samples, matching every existing lane (their published medians also sit at 0 MCP calls).
• Run reports: experimental/benchmark/.work/graph/20260702T030323 (baseline), experimental/benchmark/.work/graph/20260702T033847 (graph arms). Claude-only run, so no codex trace audit.

Deferred

• N=5 re-measurement of the fable lane if pricing/importance ever justifies it.
• Comparator MCP adoption (codebase-memory/serena being ignored by all models) is a tool-side problem, out of scope here.

Test plan

• pnpm exec prettier --check on the touched files (formatted in this PR).
• CI website build must render the dashboard and chart PNGs with the new lane labeled Claude Code / Fable 5 (local Node 22 cannot run the site build).
• Benchmark results table lands as the sticky <!-- ttsc-benchmark-results --> comment on this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_015JfqvojwkiTvoq4zz3vd9J

[samchon]

Claude Code / Fable 5 — @ttsc/graph agent benchmark (N=1)

Model claude-fable-5, effort high, fixture branch ttsc, prompt families dedicated + common. Tokens = Σ per-turn (input + output + cache reads); savings vs the same lane's baseline cell.

repo	family	baseline	ttsc-graph	codegraph	codebase-memory	serena
excalidraw	dedicated	948,108	205,198 (−78%)	972,369 (+3%)	2,133,263 (+125%)	1,964,142 (+107%)
excalidraw	common	794,920	205,629 (−74%)	842,456 (+6%)	1,094,860 (+38%)	1,089,969 (+37%)
vue	dedicated	1,128,160	197,259 (−83%)	905,816 (−20%)	857,413 (−24%)	1,173,573 (+4%)
vue	common	1,031,487	432,139 (−58%)	1,001,020 (−3%)	1,071,318 (+4%)	1,205,027 (+17%)
rxjs	dedicated	570,137	432,577 (−24%)	466,570 (−18%)	701,444 (+23%)	677,828 (+19%)
rxjs	common	1,066,913	351,919 (−67%)	607,357 (−43%)	678,837 (−36%)	911,321 (−15%)
typeorm	dedicated	1,018,868	206,214 (−80%)	849,424 (−17%)	916,935 (−10%)	849,186 (−17%)
typeorm	common	1,029,261	206,690 (−80%)	791,291 (−23%)	855,608 (−17%)	636,026 (−38%)
zod	dedicated	1,274,617	204,737 (−84%)	425,611 (−67%)	1,314,428 (+3%)	1,100,572 (−14%)
zod	common	1,161,708	205,480 (−82%)	1,231,886 (+6%)	1,323,313 (+14%)	1,192,828 (+3%)
nestjs	dedicated	1,303,485	197,094 (−85%)	306,080 (−77%)	1,223,539 (−6%)	1,333,301 (+2%)
nestjs	common	1,011,666	424,160 (−58%)	606,691 (−40%)	938,189 (−7%)	1,068,787 (+6%)
vscode	dedicated	592,579	214,774 (−64%)	466,990 (−21%)	1,013,860 (+71%)	648,903 (+10%)
vscode	common	991,843	208,509 (−79%)	1,066,094 (+7%)	1,265,407 (+28%)	911,704 (−8%)
shopping-backend	dedicated	868,625	214,824 (−75%)	655,132 (−25%)	1,171,782 (+35%)	991,097 (+14%)
shopping-backend	common	1,231,980	348,301 (−72%)	635,094 (−48%)	941,769 (−24%)	1,131,445 (−8%)

ttsc-graph: −71% average (range −24%…−85%), positive in all 16 cells. Dedicated-prompt cells converge to ~197–215k tokens regardless of repo size; several finished with zero source reads (answered from the checker fact graph alone, e.g. vue dedicated: 1 MCP call, 197k, $0.79). The rxjs dedicated outlier (−24%) is a cheap-baseline effect: Fable answered the baseline from the single-file observable.ts core for 570k.

Sampling / validity

• N=1 recorded intentionally (runs: 1 preserved in graph.json); existing lanes remain N=5 medians.
• Baseline: 16/16 manually inspected — all ok, web: 0, real source reading verified by citation spot-checks.
• Graph arms: 64/64 first-attempt, zero retries. MCP usage: ttsc-graph 16/16 (median 1 call), codegraph 16/16 (median 2), codebase-memory 3/16, serena 2/16 — the comparator shell-fallback matches every existing lane and is kept visible in the samples.
• Baseline sweep cost $19.58 / 16.0M tokens; Fable's baselines total 77% of Opus's over the same cells, which compresses relative savings (denominator effect).

Reports: experimental/benchmark/.work/graph/20260702T030323 (baseline), experimental/benchmark/.work/graph/20260702T033847 (graph arms). Claude-only run — no codex trace audit. Charts: #301.

No invalid or missing cells.

[Copilot]

Pull request overview

Adds support for a new Claude Code / Fable 5 model lane in the website benchmark dashboard UI and improves benchmark log parsing so modelVersion is attributed to the dominant Claude model (rather than a helper model entry).

Changes:

• Add claude-code-fable label formatting (Claude Code / Fable 5) and include it in the model display order in GraphBenchmark.tsx.
• Extend Claude version parsing to recognize claude-fable-* version strings.
• Update agent-ab.mjs stream parsing to choose the claude-* modelUsage entry with the highest outputTokens as the measured model version.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
website/src/components/benchmark/GraphBenchmark.tsx	Adds Fable lane labeling/version parsing and includes it in the sorted model order used by the dashboard.
experimental/benchmark/graph/agent-ab.mjs	Improves modelVersion attribution by selecting the most output-heavy claude-* entry from modelUsage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.