[Feature] Agent evaluation harness, measure steps, tokens and cost per task

[addyCooks]

Description

Add an agent evaluation harness under benchmarks/agent/ that measures steps, tokens, cost and pass/fail per task, so changes to the agent loop can be judged instead of argued about.

Split out of #1186 at @will-lamerton's request, and filed under #772. #1186 stays as the design record for the wider work; this issue is Phase 0 only and stands on its own regardless of what happens to Phases 1–4.

Use Case

#772 is open because of a measured gap: 37.4 average steps and 55,844 output tokens against OpenCode's 16.6 and 17,463, at statistically flat quality, with the model held constant.

Twelve wastage sub-issues have since closed, two of them (#760, #761) specifically step-count fixes. Nobody has measured since. We cannot currently answer "did that help?" for work already merged, let alone for work not yet started.

The existing benchmarks/ suite measures interactive_boot_ms_approx, dist_size_bytes and tool_count startup and packaging. Nothing in the repo can measure a step count.

This blocks three things:

• Closing Tracking: harness efficiency (step count and tool result tokens) #772, which needs a re-measurement to know where it stands
• Evaluating [Feature] Semantic Context Compiler (Intelligent Context Budget Management) #793's context work, whose whole premise is context efficiency
• Deciding whether [Feature] Repo Map in the agent loop: persistent index, lookup tools, dedup #1186's Phases 1–4 are worth building at all

Proposed Solution

A separate suite under benchmarks/agent/, run manually or nightly. The existing boot suite is untouched and must keep passing.

1. Drive the agent by shelling out to the CLI, not by importing in-process: nanocoder --plain --json run "<prompt>". Both flags already exist. This keeps the harness version-agnostic, which is what makes step 7 possible.
2. Task fixtures small vendored project trees under benchmarks/agent/fixtures/, each with a prompt and a machine-checkable assertion (a file contains X, a test passes, an exit code). Vendored rather than cloned so runs are reproducible, offline, and immune to upstream drift. Six to start, spanning navigation, single-file edit, multi-file change, and a read-only structural question. No LLM-graded scoring it reintroduces the nondeterminism the harness exists to control.
3. Score from source/usage/, which already records provider-reported tokens and prices them via models.dev. priceTokens() bills cache reads and writes at their own rates, which matters for any later context work.
4. One additive source change. PlainConversationOutcome returns toolCalls and token counts but no step count, and toolCalls.length is not a substitute a step may issue zero or several calls. Add steps: number to all three variants and surface it in the --json output.
5. Handle nondeterminism five runs per task per configuration, report median and IQR, pin model and temperature. A single run proves nothing.
6. Two model tiers a mandatory local Ollama tier any contributor can reproduce, plus an optional cloud tier behind an env var.
7. Baseline twice: current main, and the v1.29.0 tag the original benchmark used. If the gap has already narrowed, that changes what [Feature] Repo Map in the agent loop: persistent index, lookup tools, dedup #1186 is for, and it is much cheaper to learn now than later.
8. Wire pnpm run test:agent-eval. Deliberately not part of test:all too slow and too costly for the commit gate. scripts/test.sh stays as it is.

Exit criterion (Gate 0)

Published baselines for main and v1.29.0, posted back to #1186. If the harness does not reproduce a meaningful gap against a comparison harness on the same tasks, that is a real result and #1186's Phases 1–4 should be reconsidered on the strength of it.

Alternatives Considered

Keep reasoning from the original third-party benchmark. It was run on 1.29.0 and we cannot re-run it or vary it. Two relevant fixes have landed since.

Import runPlainConversation in-process instead of shelling out. Simpler, but it can only ever measure the working tree, which makes the v1.29.0 comparison impossible.

Add it to test:all. Rejected it needs real model calls, takes minutes per run, and would make every PR in the repo slower and flakier.

LLM-as-judge scoring. Rejected nondeterministic scoring on top of nondeterministic runs.

Additional Context

• I have searched existing issues to ensure this is not a duplicate
• This feature aligns with the project's goals (local-first AI assistance)

Implementation note. AVA's files glob covers only source/** and plugins/**, so it needs extending to benchmarks/**/*.spec.ts or the harness's own specs will silently never run.

Known measurement caveat. The v1.29.0 run will not have the steps field from step 4. Steps will be derived from its tool-call log for that run, and the two are not measured identically, this will be stated wherever the numbers are published rather than quietly averaged together.

Parent: #772 · Design record: #1186

[addyCooks]

Hey @akramcodez,
could you assign this to me ?