Request for guidance — measuring agent-level (not tool-level) impact

[GrowtricsAI]

Hi — we ran an A/B battery measuring codedb's effect on agent-level metrics (token usage + wall time of an LLM coding agent over representative tasks), separate from the tool-level ripgrep benchmarks you publish.

TL;DR: Tool-level speedup did not obviously translate to agent-level token reduction at our sample size. Likely a measurement problem on our end — would value your take on methodology.

Setup:

• 5-task battery across Python, Dart, and TypeScript code, plus cross-module navigation
• Baseline (grep + file-read tools only) vs treatment (codedb MCP registered, indexes pre-warmed)
• Fresh agent session per run (no shared context/cache); N = 1–2 per cell
• Ran the whole battery twice to check reproducibility (32 runs total)

Round 1 vs Round 2, Δ% tokens (baseline → treatment):

Task type	R1 Δ%	R2 Δ%
Python symbol caller lookup	−8.7%	+10.3%
Dart widget structural trace	−25.9%	+0.1%
Cross-module flow trace	−29.6%	+21.4%
TS config-value audit	+16.0%	−18.5%
Python async race-condition hunt	−0.6%	−24.2%

Every task produced contradictory signals across rounds. Baseline tokens alone varied 2× run-to-run on identical prompts. Agent exploration stochasticity appears to dominate codedb's effect at our N.

Open questions:

1. Recommended methodology for measuring agent-level impact? Any reference design you'd point to?
2. Does codedb's benefit compound over long-running multi-turn agent loops (vs single-shot sessions)? That seems plausible but we haven't tested it.
3. Any tell-tale signs in agent behavior that indicate codedb is being used well vs underutilized?

What we'd change next time:

• Deterministic-answer prompts (to remove agent exploration variance)
• N ≥ 10 per cell
• Test inside a long-running multi-turn agent loop

Happy to share more of the methodology if useful for future codedb benchmarking docs. Thanks for the tool — install was clean via the release binary and the MCP surface is well-designed.

[justrach]

Thanks for sharing this. From the issue body alone I don’t think we can validate or invalidate the result yet, mainly because the setup is underspecified and the reported variance is larger than the apparent effect size.

The closest internal methods we use today are tool-level rather than whole-agent:

• zig build bench -- --json: fixed-corpus E2E MCP tool dispatch benchmark, including response size and telemetry on/off deltas
• zig build benchmark -- --root /path --json: repo-level indexing/query/reindex benchmark
• bench/run-benchmarks.py: tool-level comparisons against rg / grep / ast-grep, including an approximate response-size token estimate

One important distinction: the “fewer tokens” numbers in our docs are derived from per-query response payload size, not from end-to-end agent session token accounting. So your A/B battery is measuring something we do not currently benchmark internally in the same way.

To compare your result to our internal methods, could you share the exact methodology?

1. Model + version, and temperature / randomness settings.
2. Exact prompts/tasks and which repos were used.
3. Agent bench shape: single-shot vs planner/executor loop, max turns, stop condition, retry policy, tool budget, and any important wrapper/system prompt constraints.
4. How tokens were counted: provider-reported tokens, wrapper logs, or an estimate.
5. Whether the same non-codedb tools were available in both arms.
6. Whether codedb tool usage was confirmed in treatment runs.
7. Whether index warmup time was excluded.
8. Whether these were single-turn runs or long-running multi-turn sessions.
9. One or two full tool-call traces or transcripts from representative baseline/treatment runs.

With N = 1–2 per cell and 2x baseline token variance, I’d treat the current numbers as inconclusive rather than contradictory. A stronger design would be:

• fixed prompts and model settings
• confirmed tool invocation in treatment
• separating deterministic tool-trace replay from free-form agent-loop runs
• N >= 10 per cell for the free-form condition

If you share the traces/methodology, we can map them more cleanly against the internal benchmark surfaces we already use.