| topic | agents |
|---|---|
| type | guide |
| status | research-complete |
| last-validated | 2026-05-21 |
| related-docs | 699, 699d, 161, 307, 355 |
| original-query | /zao-research harnesses - deep dive on agent harness engineering as a discipline. |
| tier | DEEP |
Goal: Define the agent harness as a discipline - anatomy, the harness-beats-model evidence, how to engineer one, how to measure it - and apply it to ZAO's Hermes, QuadWork, and ZOE.
| # | Decision | Why |
|---|---|---|
| 1 | TREAT the harness, not the model, as ZAO's primary agent-quality lever | Across every 2026 study, the scaffold explains more performance than model choice - documented swings of 13-26 points on the same model |
| 2 | INSTRUMENT Hermes with the 4 missing metrics: token efficiency, retry depth, Critic-accuracy audit, PR keep-rate | Hermes already has the structural patterns; what it lacks is measurement. You cannot improve a harness you do not measure. |
| 3 | KEEP ZOE a thin human-in-the-loop harness on purpose - do NOT add autonomous verification loops to it | ZOE is a capture/route concierge, not a fire-and-forget executor. A heavy harness there is wasted complexity. |
| 4 | ADOPT the fail-loud rule everywhere: verification gates pass 100% or hard-stop - no soft margins, no silent degradation | Silent failure is the #1 production agent failure mode (doc 699). Fail-loud makes it visible. |
| 5 | When in doubt, spend tokens on the harness (verification, isolation, retries), not on a bigger model | At equal budget, a great harness + weaker model beats a weak harness + stronger model |
An agent harness is the complete scaffolding that turns a language model from a text generator into a functioning agent: the control loop, tools, memory, context management, error handling, sandbox, verification gates, and observability. The 2026 insight: the harness is not a wrapper around the agent - it IS the agent. The model is just the reasoning component inside it.
| Component | What it does |
|---|---|
| Control loop | The state machine: observe -> decide -> act -> check -> retry/advance. Deliberately boring; ~6 lines. Complexity belongs elsewhere. |
| Tool / action layer | Registered functions + execution; call dedup, per-status retry policy, error normalization |
| Memory + context management | History, intermediate state, long-term memory; sub-agent isolation, summary caps, checkpoints at token high-watermarks |
| Execution environment / sandbox | Where decisions become real actions; budget caps, timeouts, hard stops |
| Verification + safety gates | Independent validation - the dual-agent (coder + critic) split; fail-loud, binary pass/fail |
| Error classification + recovery | Transient (429/5xx -> backoff) vs terminal (401/403/422 -> escalate); hallucination-loop hard stops |
| State checkpointing | Persist agent state after every major action; enables resume + human review |
| Observability | Structured logging of every decision, tool call, outcome - component / experience / decision observability |
The claim is not rhetoric. Independent 2026 results, same model held constant:
| Study | Finding |
|---|---|
| Anthropic multi-agent research system | Token usage alone explains ~80% of performance variance; model + tool-call count explain the rest |
| LangChain, Terminal-Bench 2 | Same Claude Opus: 52.8% -> 66.5% from a harness rebuild alone (+13.7 pts, rank 30 -> rank 5) |
| Cursor vs native Codex, GPT-5.5 | Same model, two harnesses: 87.2% vs 61.5% functionality - a 25.7-point swing |
| Claude Code vs raw Opus, SWE-bench | Raw Opus ~75% -> Claude Code (same Opus + Anthropic harness) ~80.9% - the gap is pure harness |
| arXiv 2604.25850 (Agentic Harness Engineering) | Same GPT-5.4: 69.7% baseline harness -> 77.0% evolved harness over 10 iterations |
The pattern holds across Anthropic, LangChain, Cursor, Stanford, and arXiv teams. Caveat: these are 2026 benchmark figures and benchmarks are contested (doc 699, Section 5) - treat the direction as solid and the exact percentages as soft.
- Context engineering over prompt engineering - tight scope, explicit constraints, deterministic checks; bounded history to avoid context drift.
- Sub-agent isolation - delegate to child agents with filtered tools, separate context, own budget; a 50K-token exploration compresses to a 2K summary.
- Tool ergonomics + error classification - dedup calls, cap result sizes, classify errors transient vs terminal.
- Dual-agent verification - a primary (coder) proposes, a secondary (critic, often a lighter model) reviews and scores; loop on fail.
- Budget caps + hard stops - token/cost/step/timeout caps; on hit, escalate, do not degrade.
- Checkpointing + state recovery - save state after every major action; resume or hand to a human on failure.
- Observability-driven iteration - every harness component is editable + logged; every edit is a hypothesis verified against outcomes.
- Fail-loud, not graceful degradation - make errors visible; never return partial results silently.
The metrics that tell you a harness is good:
| Metric | Healthy target |
|---|---|
| Task completion rate | as high as the task allows; benchmark vs same model + different harness |
| Token efficiency (tokens / successful completion) | lower is better; track the trend |
| Cost per outcome (spend / successful completion) | track notional cost even on flat-rate Max auth |
| Tool error rate | <5% healthy, >15% = harness problem |
| Escalation rate | 5-30%; >30% brittle, <5% under-verified |
| Retry depth (avg retries to success) | <1.5 healthy; >2.5 = inefficient loop |
| Critic / gate accuracy vs real outcomes | >80% or recalibrate the gate |
Benchmark method: same harness on two models = the model effect; same model on two harnesses = the harness effect. A harness swing >5 points is architecturally significant.
- Framework (LangGraph, CrewAI, smolagents) - the library giving you a control-loop abstraction + tool registry + memory interface.
- Harness - the concrete instance you build on top: model choice, tool defs, prompts, retry policy, budget caps, verification gates, observability. ZAO's Hermes is a harness; Claude Code is Anthropic's harness.
- Agent - the model reasoning inside the loop.
You can have a great harness on a simple framework (Hermes on a Claude Code subprocess) and a terrible harness on a great framework. The framework matters less than the harness.
Hermes - already a real harness: pre-flight cost gate, Opus coder + Sonnet critic, 3-retry loop, PR gate, git reset --hard recovery. Gaps: no token-efficiency tracking, no per-step latency, no Critic-accuracy audit against real PR outcomes, no PR keep-rate. Those 4 metrics are the next upgrade - doc 699d's coding-pattern upgrades are the what, this doc's metrics are the how you know it worked.
ZOE - a deliberately thin harness: single model, message loop, no autonomous verification. Correct - ZOE is human-in-the-loop capture/routing, not an executor. Do not over-build it.
QuadWork - 4 agents in isolated windows = the sub-agent isolation pattern, each with its own budget. Parallelizing the reviewers (doc 699d) is a harness-level change.
- Anthropic - Multi-Agent Research System [FULL]
- arXiv 2604.25850 - Agentic Harness Engineering [FULL]
- LangChain - Improving Deep Agents with Harness Engineering [FULL]
- Cursor - Continually Improving Our Agent Harness [FULL]
- Arize - Context Management in Agent Harnesses [FULL]
- Steve Kinney - The Anatomy of an Agent Loop [FULL]
- Towards Data Science - 12-Metric Framework for Production Agents [PARTIAL]
- MindStudio - 9 Components Every Production Harness Needs [PARTIAL]
- arXiv 2603.02601 - AgentAssay (token-efficient regression testing) [FULL]
- Doc 699 - state of agentic AI 2026 (harness-beats-model is finding #1)
- Doc 699d - 10 concrete Hermes/QuadWork upgrade patterns
- Doc 699c - the eval tooling that makes harness metrics measurable
| Action | Owner | Type | By When |
|---|---|---|---|
| Instrument Hermes with token-efficiency, retry-depth, Critic-accuracy, PR-keep-rate metrics | @Claude | PR | Next agent sprint |
| Run a harness-swing benchmark: Hermes on Opus vs Sonnet coder, same task set, measure the gap | @Zaal | Experiment | After instrumentation |
| Adopt the fail-loud rule as a written ZAO agent standard | @Claude | Rule | Next session |
| Feed this doc's metrics into the doc 699c Langfuse/PromptFoo eval plan | @Zaal | Decision | With the eval rollout |