feat(cli): add --json output for benchmark

[yashksaini-coder]

Stacked on #157 — please review and merge that one first.
GitHub can't target a base branch that only exists on my fork, so this PR's diff against main contains #157's commit as well. Review the second commit only — or git diff 01ef530..2bc7c68. Once #157 merges, this reduces to the benchmark commit alone.

Addresses the remaining piece of #69: --json for nanotune benchmark.

What

nanotune benchmark --json > run.json
nanotune benchmark --json | jq -e '.summary.passRate >= 0.9'

Prints exactly the document already written to .nanotune/benchmarks/benchmark-<timestamp>.json — one schema, not two. The issue notes people are already reading that file directly; this just makes it a first-class output. Progress goes to stderr while the suite runs, since a benchmark takes minutes and stdout has to stay a single parseable document.

Why this needed a refactor

Ink can't mount in JSON mode, and the entire benchmark run lived inside a ~550-line useCallback in the command component. There was no seam — so the run moves to src/lib/benchmark-run.ts as runBenchmark, an async generator yielding progress with the finished BenchmarkResult on its final done event. That's the same shape ensureModelDownloaded already uses to return its resolved path.

benchmark.tsx: 1038 → 249 lines. It now only translates events into view state.

The move is mechanical. One deliberate change: the ~10 setError(…); setStatus('error'); return paths become throw. An error path that returns normally is a smell, and a throw is what lets one function serve both consumers — the component's existing catch turns it into the error frame, emitJson turns it into a stderr line and exit 1.

Tests the run never had

Three pure functions fall out of the extraction, all previously buried in the component with zero coverage:

Extracted	Was	Now
resolveRunOptions	~55 lines of preset-vs-flags branching	7 tests
summarizeResults	~80 lines of averaging + partial-run arithmetic	8 tests
validateTests	inline loop	4 tests

Plus formatEventForStderr (3) and 11 tests driving runBenchmark through every guard that fires before startLlamaServer.

One behaviour improvement came free: resolveRunOptions now runs before model resolution, so a mistyped --preset fails immediately instead of after a multi-minute --base download.

BenchmarkResult.warning

New optional field recording why a run stopped early. When llama-server dies mid-suite the run saves what it has — and summary correctly scores against the tests that actually ran, which means a saved partial run was previously indistinguishable from a complete one with a worse pass rate. The terminal already showed this warning; now the result carries it, so a --json consumer can't mistake a partial score for a real one:

nanotune benchmark --json | jq -e 'has("warning") | not'

Optional and additive, so existing files and benchmark compare are unaffected.

One structural wrinkle worth flagging

--base runs two jobs concurrently (installing llama.cpp, downloading the base model) inside nested async functions — which cannot yield. They now report into a shared latest string that a drain loop turns into events. They already shared a single progress line before this was a generator, so no information is lost. The settle handler is attached at promise creation rather than later, because work can reject while the drain loop is sleeping and an unhandled rejection is fatal on Node 22 — the same reasoning as createServerHandle.

Exit codes unchanged

A completed run exits 0 whatever its pass rate — there's no configured threshold and inventing one would be surprising. A run where the server died before any test completed still exits 1 with nothing on stdout.

Testing

522 passing (+35). test:lint, test:format, test:types, test:knip, test:audit, test:ava all green locally.

Verified against the built binary:

Check	Result
Outside a project → 0 bytes stdout, message on stderr, exit 1	✅
--model + --base → mutual-exclusion error	✅
Invalid --preset → rejected before model resolution	✅
Missing dataset → starter tests.json written, exit 1	✅
Malformed test → error names the test id	✅
llm-judge test with no judge configured → correct hint	✅
Interactive nanotune benchmark still renders its error frame, exit 1	✅

⚠️ I could not run a real benchmark. I'm on Linux; assertSupportedPlatform gates --base and there's no llama.cpp binary here, so every path from startLlamaServer onward is verified by inspection and by keeping the move mechanical, not by execution. Please run one real suite on Apple Silicon before merging — ideally one with --samples > 1 and one that exercises --base. If it'd help, I'm happy to split the extraction out as a pure no-op refactor commit to make that diff easier to eyeball.

---

Update: coverage gate, and what it turned up

The first push failed Unit Tests & Coverage Analysis — 69.98% against a 71% floor and a 71.65% baseline. Worth writing up, because it wasn't a real regression.

c8 runs without all: true, so it only measures files a test actually imports. No spec imports BenchmarkCommand, so src/commands/benchmark.tsx never appeared in the coverage report at all — its ~800 uncovered lines were invisible to the metric. Moving that code into benchmark-run.ts, which its spec does import, made the same uncovered lines visible for the first time. The number went down because the measurement got more honest.

Rather than paper over it, I covered what was genuinely coverable:

generateMarkdownReport — ~170 lines of branchy template producing a user-facing artefact, never exercised in this repo's history. Now covered across base-vs-fine-tuned runs, the partial-run notice, optional config and summary fields, multi-turn conversations, sampling stats, judge scores and criteria, and the failures table including its pipe-escaping and newline flattening.

The scoring loop, via an optional deps argument on runBenchmark defaulting to the real startLlamaServer/chatCompletion/stopLlamaServer/callJudge. Production always takes the default; tests pass a fake. That loop decides every pass and fail the product reports — sampling and per-sample seeds, timeout handling, judge-vs-match dispatch, category tallying, failure recording, the partial-run abort — and had no tests at all, because exercising it needed a live server and a real model.

I'm aware "add a parameter for tests" deserves scrutiny. My argument: making the run callable outside React is the same property as making it testable, and 186 lines of untested scoring logic is the highest-value target in the file. Happy to drop it if you'd rather.

69.98%  ->  74.75%          (main baseline 71.65%)
benchmark-run.ts  45.45%  ->  86.53%
552 tests passing (+65 over main)

The branch now raises repo coverage rather than lowering it — including against the 73.04% baseline it will face once #157 merges, which I measured on that branch to be sure this doesn't just fail again after the stack lands.

Two things the new tests turned up

Fixed here: an empty tests.json made passRate compute 0/0, and JSON.stringify writes NaN as null — a field documented as a number reaching consumers as null. One-line guard plus a test.

Not fixed, flagging only: the abortReason && allResults.length === 0 branch looks unreachable. void serverHandle.exited.then(...) is registered after the await startLlamaServer, and the loop begins synchronously, so serverDied is always false on the first iteration — a server that dies instantly still runs test 1 (which fails to connect) and saves a 1-test partial run instead. Pre-existing and unchanged by this PR, so I left it alone; say the word if you'd like it handled.

Suggestion for a separate issue

Consider running c8 with all: true so untested files count as 0% instead of vanishing from the report. It would drop the headline number repo-wide and need the threshold retuned, so it's your call — but right now the figure means "of the code we already test, how much do we test", and moving code between files can swing it with zero behavioural change. Happy to open an issue if useful.

[yashksaini-coder]

@will-lamerton @akramcodez — ready for review, though #157 should go first since this is stacked on it.

Adds --json to nanotune benchmark, completing #69. All checks green.

One ask before merging: I'm on Linux, so I couldn't run a real benchmark — everything past startLlamaServer is covered against a fake rather than a live server. If either of you can run a suite on Apple Silicon (ideally one with --samples > 1 and one with --base), that would close the gap.