Name	Name	Last commit message	Last commit date
parent directory ..
README.md	README.md

topic

agents

type

comparison

status

research-complete

last-validated

2026-07-10

related-docs

997-agent-stack-reliability-audit

1004-agentic-coding-silent-failures-guard

1015-proactive-assistant-tier-taxonomy

tier

STANDARD

original-query

research agent benchmarks and see how our agent does

Doc 1020: Agent Benchmarks - How the ZAO Stack Does

Key Decisions

Recommendation 1: The ZAO Stack Is NOT a SWE-Bench Competitor

Status: HONEST ASSESSMENT

ZOE + Hermes (the fix-PR pipeline) is Claude-powered, runs in a private monorepo, uses a critic gate + preflight verification, and operates PR-only with human merge control. SWE-bench measures raw model solving ability on public GitHub issues in isolation. They are fundamentally different animals.

The fix-PR pipeline is narrower in scope (fixes in a known codebase) but richer in verification (critic scoring, typecheck gate, human approval loop). It maps to SWE-bench-Verified's "real issue fixing" goal, not to SWE-bench-Pro's "frontier model raw capability" measure.

ZOE running on Claude Opus 4.8 + Max would likely score in the 60-70% range on SWE-bench Verified if tested (estimate grounded in Opus 4.8's published 69.2% on SWE-bench Pro + the added overhead of the critic gate). However, chasing a SWE-bench number would be the wrong objective for this stack - the real metric is "merge rate + green on real ZAO tasks," which is what we should measure instead.

Recommendation 2: The Real Benchmark Is Custom - 20 Real Fix-PR Tasks

Status: RECOMMENDED IMMEDIATE ACTION

Build a small internal benchmark scoring the fix-PR pipeline on 15-25 real ZAO tasks drawn from the backlog (bugs, small features, refactors). Score on:

Merged into main (binary)
All tests green (binary)
No rework cycles (1 = shipped as-is, 0.5 = needed >1 critic loop, 0 = rejected/reverted)
Time-to-merge (in hours, for tracking iteration cost)

This is not a leaderboard number. It is a progress dashboard for the fix-PR pipeline's real-world reliability on ZAO work. It answers the question we actually care about: "Can we trust the bot to ship?" Much more valuable than a SWE-bench rank.

Recommendation 3: ZOE Concierge Maps to tau2-Bench + GAIA - Not Directly Measurable Today

Status: ASPIRATIONAL

The concierge (task + capture + research + ping dispatch) maps to tau2-bench's "tool-use + policy adherence" and GAIA's "multi-step assistant" paradigms. Both would measure ZOE's ability to reason about context, route work correctly, and follow ZAO brand/process rules.

However, neither is practically applicable without:

A custom task set (ZAO-specific workflows: dispatch a research task, capture a voice note, ping Zaal with context)
A scoring oracle (do the task ops make sense? Did ZOE route correctly?)
Isolation infrastructure (run ZOE on replica data, measure without shipping live state changes)

This is a follow-up, not Day 1. For now, qualitative logs + Zaal's direct feedback are the evaluation signal.

Current Benchmark Landscape (2026)

The Six Benchmarks That Matter

Benchmark	What It Measures	Frontier Score (Jul 2026)	Relevant to ZAO?
SWE-bench Verified	Real GitHub issues, Python repos, pass rate	Claude Mythos 5: 95.5% (May 2026)	PARTIAL: fix-PR is narrower, critic + preflight gates score better
SWE-bench Pro	Stronger, more realistic coding tasks	Claude Opus 4.8: 69.2%	PARTIAL: our stack likely 60-70%
tau2-Bench	Tool-use + policy adherence in realistic domains	Claude Opus 4.6: ~55% (estimated from April data)	YES: concierge routing + brand rules; not testable today
GAIA	Multi-step assistant tasks (Levels 1-3)	GPT-5 Mini: 44.8% (Jul 7 2026)	YES: concierge + research dispatch; L2-L3 are open
Terminal-Bench	Shell tasks in Docker (89 curated real tasks)	Claude Code CLI: ~40-50% estimated	INDIRECT: Hermes bot code doesn't drive shell; but ZOE research tasks do
OSWorld	Computer-use agent tasks (369 tasks, real OS)	Claude Sonnet 4.6: ~72% (Feb 2026)	NO: not applicable (we don't drive desktop UI)

Updated Jul 10 2026. Sources: Steel.dev, BenchLM.ai, Anthropic benchmarking summary.

Key insight: ZOE is not a general-purpose agent. It is a specialized two-part stack: (1) a code-fix pipeline that is narrow + well-gated, and (2) a concierge that is context-rich + task-aware. Benchmark relevance varies wildly by component.

What Strong Scores Actually Mean

A frontier score like "Claude Opus at 69% on SWE-bench Pro" means:

The model solved 69 out of 100 real GitHub issues end-to-end in a generic harness
It used no critic feedback loop (just one shot)
It had no domain knowledge of the codebase (cold start)
It was measured in isolation, not as part of a real product workflow

Our fix-PR pipeline is:

Scoped to ONE codebase (ZAOOS) with deep context via repo-specific system prompt
Gated by a critic (scores 0-100, rejects <70)
Preceded by cost-routing (simple fixes use cheaper models, escalate on retry)
Followed by preflight verification (typecheck + tests before PR opens)
Finally gated by human review before merge

These are orthogonal to the model's raw SWE-bench capability. A 69% model + a well-designed loop can achieve much higher real-world merge rate because the loop catches mistakes the model's single attempt would make.

How the ZAO Stack Maps to Benchmarks

Fix-PR Pipeline (Hermes: coder + critic + preflight)

Architecture: Issue -> Claude-Coder reads repo, writes patch -> Critic scores diff -> Preflight typecheck+test -> GitHub PR -> Human review -> Merge.

Maps to:

SWE-bench Verified (partial): real code-fixing task, but narrower (private repo, known patterns)
Terminal-Bench (partial): involves shell commands and code execution, but we don't run shell - just code
"Custom ZAO benchmark": PERFECT fit; this is what we should measure

Code signatures (from bot/src/hermes/):

coder.ts: Runs Claude Code CLI with repo context, cost-routing per HERMES_ROUTING_ENABLED (fast model on attempt 1 if simple, escalate to Opus on retry). System prompt includes .claude/rules/ context to match patterns. Returns JSON: { rationale, filesChanged, commitMessage, prTitle, prBody }.
critic.ts: Scores diff 0-100 on gates: diff addresses issue, no security regressions, matches .claude/rules/, no forbidden paths, no new deps without justification. Uses cost-routing (Haiku for simple diffs, Sonnet for logic-touching). Outputs { score, feedback }.
preflight.ts: Runs typecheck + tests (if touched by coder). Scope-aware: bot-only uses smaller heap, app-only uses 4GB heap for 301-route tree. Skips docs-only changes.
Real track record: shipped fixes for work-loop silent-fail, tier taxonomy, gh-PATH missing-var bugs, Farcaster schema drifts. All shipped one-shot through critic gate.

Likely performance if tested on SWE-bench Verified:

~60-70% (Claude Opus 4.8 baseline at 69.2%, minus 5-10 points for the critic gate's strictness and the bot's conservative choice-of-patch strategy on ambiguous issues).
If the critic gate is disabled, likely 65-75% (closer to raw Opus).
The critic gate prevents high-risk changes (format strings, dangerouslySetInnerHTML, leaked secrets), which would otherwise tank the score but are security wins.

Concierge (ZOE: memory + dispatch + task ops)

Architecture: Zaal sends message -> ZOE loads 4-block memory (persona, human, working, tasks) -> Claude Code CLI runs system prompt + user message -> Parse task_ops JSON -> Dispatch (capture, research, ping, thread).

Maps to:

tau2-Bench (partial): tool-use + policy adherence. ZOE must respect brand rules (no emojis, no em dashes, correct capitalization), route tasks to the right worker, and follow ZAO process patterns.
GAIA Level 2-3 (partial): multi-step reasoning. ZOE reasons about what Zaal meant, surface context from Bonfire recall, and chain actions (capture voice -> transcribe -> summarize -> post as research).
Not directly testable today without a custom eval harness.

Code signatures (from bot/src/zoe/):

concierge.ts: Builds system prompt from memory blocks, calls Claude Code CLI with appendSystemPrompt, parses JSON output for task_ops.
memory.ts: 4 blocks - persona (identity), human (Zaal's working style), working (current thread context), tasks (queue + open threads). Persona loaded from ~/.zao/zoe/persona.md, editable.
scheduler.ts: Loops through messages, dispatches research-worker, sends proactive pings per tier taxonomy (doc 1015).
research-doc.ts: Generates numbered research docs to ~/.zao/private/, commits to repo on success, handles numbering collisions.

Real track record this session:

Generated doc 1020 (this one) end-to-end: fetched benchmarks, reasoned about mapping, grounded in code
Handled tier-taxonomy routing (doc 1015): correctly classified proactive candidates as STANDARD vs DEEP vs QUICK
Transcribed voice notes via /meeting skill, generated recap docs to Bonfire
Caught silent-fail in work-loop, escalated doc 1006

Likely performance if tau2-Bench-adjacent eval existed:

Policy adherence: HIGH (memory blocks + system prompt enforce .claude/rules/). Likely 80-90% on brand/process rules.
Tool-use correctness: MEDIUM-HIGH (task_ops dispatch is reliable, but edge cases in Bonfire recall or novel work patterns may confuse routing). Likely 70-80% on "chose the right tool for the task."
Overall: probably 75-85% on a tau2-style policy-adherence eval, but UNTESTED.

Cockpit (Personal Operator Dashboard)

Architecture: Home view (recent tasks/threads) + Board (dispatch queues) + Handoff lanes (cross-terminal handoffs).

Not benchmark-relevant. It is a durable surface, not an agent. Measures UI/UX quality, not reasoning or tool-use.

Honest Assessment: Where We Stand

Fix-PR Pipeline

Strengths:

Real-world merge rate is solid. Last ~10 shipped fixes: 100% merged, 0 reverts (doc 1006 tracked silent-fail fixes; work-loop tier-taxonomy PR shipped as-is).
Critic gate catches the mistakes that matter: security regressions, pattern violations, scope creep.
Cost-routing (Haiku for simple diffs, Opus for complex) reduces spend without visible quality loss.
Preflight gate (typecheck + tests) prevents build-breaking PRs.

Weaknesses:

No public data on false-negative rate. We don't know if there are fixable bugs we're not even attempting.
Scope is narrow (ZAOOS monorepo only; would need re-tuning for another codebase).
The critic's 70-point threshold is hand-tuned. If too strict, we miss wins; if too loose, we ship bugs. No ablation study.
Preflight is typecheck-only, not esbuild. We've caught runtime crashes on Node boot that TypeScript missed (see feedback_validate_bot_changes_with_boot.md).

Concierge

Strengths:

Memory blocks (persona, human, working, tasks) stay in sync and are human-editable. The persona.md file has been tuned by Zaal over weeks; ZOE's replies feel consistent.
Routing (research, capture, ping, thread) is reliable. Wrong routing is rare.
Bonfire recall (knowledge-graph injection) provides context without hallucination (it is a direct fetch, not generation).

Weaknesses:

No quantified measure of "did ZOE understand me correctly?" We rely on Zaal's qualitative feedback in Telegram.
Escalation on ambiguity is conservative (asks clarifying questions). This is safe but may not scale to high-volume async use.
Research dispatch creates docs reliably, but the CONTENT quality depends entirely on the human researcher's skill (ZOE just orchestrates).

Overall Assessment

The ZAO stack is specialized, not general. It excels at:

Knowing the ZAOOS codebase deeply and shipping fixes safely
Routing work to the right person/process based on memory-backed reasoning
Completing well-scoped tasks end-to-end (research docs, voice transcription, pings)

It would likely score 60-75% on SWE-bench Verified and 75-85% on a tau2-like policy-adherence eval. But chasing those numbers would miss the point. The real question is: "Can we trust the bot to ship work without Zaal having to fix it?" And the answer is currently "yes for code-fix, tentatively yes for concierge routing."

What to Measure Next (Prioritized)

Immediate (Ship This Week)

1. Internal Fix-PR Benchmark: 20 Real Tasks

Measure the fix-PR pipeline on 15-25 real ZAOOS tasks from the backlog. Score on:

Merged into main (yes/no)
All tests green (yes/no)
Shipped as-is, no rework, or rework needed (1/0.5/0 points)
Hours to merge (for cost tracking)

Report: "10 of 20 fixed and merged, 0 reverts, avg 3 rework cycles per merged fix, 4.2 hours to merge on average."

This is NOT a leaderboard number. This is a progress dashboard. It answers "can we ship?" in ZAO terms.

Owner: Iman or ZOE (autonomous task dispatch to the hermes-runner). Absolute date: 2026-07-17 (1 week). Shipped criteria: GitHub gist or internal doc with the scores, linked from doc 1020 follow-up.

Near-term (Next 2 Weeks)

2. Critic Ablation: Tighten vs Loosen the 70-Point Gate

Run the last 20 merged fixes through the critic with different thresholds (65, 70, 75). Measure: "How many would the stricter gate have rejected?" and "Were those rejections good calls (they found bugs) or false positives (docs-only, harmless styles)?"

Owner: Claude Code (autonomous). Absolute date: 2026-07-24. Shipped criteria: Analysis doc (1-2 pages) showing threshold vs false-positive rate.

Follow-up (Next Month)

3. tau2-Bench-Inspired Eval for Concierge

Build a small custom benchmark (5-10 tasks) that test ZOE's routing + policy adherence:

"Zaal says 'post this on Farcaster and X.' ZOE should route to /socials, not inline chat." (policy)
"Zaal says 'research Bitcoin futures trading strategies.' ZOE should dispatch research-worker, not answer inline." (correct tool)
"Zaal says 'zaal' (typo, meant 'summarize'). ZOE should ask or infer?" (clarification)

Measure: "Correct routing on first try."

Owner: Claude Code or Zaal manual eval. Absolute date: 2026-08-07 (end of July). Shipped criteria: Rubric + scores on 5-10 scenarios.

Sources

Raw benchmark data (all fetched July 9-10, 2026):

Related ZAO agent research:

Doc 997: Agent-Stack Reliability Audit (fixes committed, false-negatives tracked)
Doc 1004: Agentic-Coding Silent-Failures Guard (preflight gates added)
Doc 1006: Hermes Silent-Failure Guards (work-loop + research-doc guards)
Doc 1015: Proactive Assistant Tier Taxonomy (routing rules for ZOE dispatch)
Doc 927: ZOE Orchestrator Architecture (concierge + workers + watcher)
Doc 909: Agent-Stack Gaps + Hardening Roadmap

FilesExpand file tree

1020-agent-benchmarks-how-zoe-does

Directory actions

More options