A benchmark for multi-agent orchestration strategies. Same task, same model, same tools. Only the way the agents are orchestrated changes.
Do self-orchestrating agents actually help, or do they just add cost, latency, and a new way to fail? This repo measures it. Provider-agnostic core (runs on local open models); a reference deployment runs the same four strategies on AWS serverless (Bedrock AgentCore + A2A + Step Functions). Apache-2.0.
We take one task and build it four ways, from fully scripted to fully autonomous, holding the model and tools fixed. Run across three settings (a 720-run local pilot, a live AWS deployment, and a τ²-bench validation run), the honest finding is:
The best orchestration depends on the shape of the task.
- Known, single-shot path → scripted (explicit) wins. In the local pilot, a hand-authored workflow was ≥ a self-orchestrating coordinator on all three models, and +0.58 more reliable on the weakest one (llama3.1:8b, paired bootstrap, p≈0).
- Open, multi-turn conversation → self-orchestration wins. On τ²-bench (Sierra's public multi-turn tool-use benchmark), the ordering flips: self-orch is the best strategy on Claude Haiku 4.5 (0.60 vs explicit's 0.20) and ties for best on Sonnet 4.5. The scripted approach that won the pilot is the worst strategy here.
- Autonomy is never free. Self-orchestration cascades (an unguarded coordinator ran to its 6-hop delegation cap every time on an open-ended AWS task; a guardrail held it to 2), and on Haiku it cost 8.5 delegations, 37 LLM calls/episode, and ~3× the latency to earn that win.
- Orchestration effects need a capable base model. On τ²-bench, Llama 3.3 70B scored 0 across the board and Pixtral Large was too slow to finish an episode.
So: script what you know; let the model self-orchestrate what you don't, behind a guardrail. This is a small study, run honestly, not a formal publication. See Limitations.
The single independent variable. Everything else (model, tools, tasks, temperature) is held constant.
| # | Strategy | What it is |
|---|---|---|
| ① | single | one agent with all tools |
| ② | explicit | a hand-authored workflow / DAG (deterministic) |
| ③ | self-orch | a coordinator that decides at runtime which specialists to delegate to (A2A) |
| ④ | hybrid | self-orch + a guardrail (a cap on delegations) |
| model | single | explicit | self-orch | hybrid |
|---|---|---|---|---|
| qwen3:8b | 0.95 | 0.95 | 0.75 | 0.75 |
| qwen3:14b | 0.85 | 0.95 | 0.65 | 0.65 |
| llama3.1:8b | 0.25 | 0.83 | 0.35 | 0.35 |
Success rate, n=60/cell. Explicit ≥ self-orch everywhere; the gap widens as the model weakens.
On llama3.1:8b, explicit − single = +0.583 (95% CI [+0.37, +0.80], p≈0, paired bootstrap).
On an open-ended "keep digging" task (live AWS), the unguarded self-orchestrator ran to its safety cap of 6 delegations on every run; the guardrail (hybrid) held it at 2. Scripted and single never delegate. Autonomy + an open-ended goal drifts toward runaway, quietly.
Explicit is the cheapest and among the fastest; self-orch is the slowest (sequential A2A hops) and its tool-selection rate is ~0; it reaches answers off the labelled path.
The local suite is our own (synthetic, labelled, single-shot). To check the finding against an independent, third-party benchmark, we ran the same four strategies on τ²-bench (Sierra) in the retail domain: a multi-turn setting where the agent holds a conversation with a user-simulator and is graded on final database state plus LLM-judged assertions.
| model | single | explicit | self-orch | hybrid |
|---|---|---|---|---|
| Claude Haiku 4.5 | 0.40 | 0.20 | 0.60 | 0.40 |
| Claude Sonnet 4.5 | 0.53 | 0.47 | 0.53 | 0.33 |
Success rate, n=15/cell (5 tasks × 3 trials), judge = Sonnet 4.5.
The ordering flips. On these multi-turn tasks the scripted explicit DAG that won the local
pilot is now the worst strategy on Haiku (0.20), while runtime self-orch is the best (0.60).
On Sonnet, self-orch ties single for the top. The intuition: a fixed DAG can't adapt to where
an open conversation actually goes, so letting the model route at runtime pays off, at a real cost
in delegations and latency (see the panels above).
Two models produced no usable signal, and we report that plainly. Llama 3.3 70B scored 0.00 on every strategy (it engaged but never completed a retail task; half its episodes hit the step or error cap). Pixtral Large is excluded: all 59 episodes hit our 420 s per-episode timeout (median latency 468 s), so its zero is a harness limit, not a capability result.
This is a validation-scale run (n=15/cell, 5 of 114 retail tasks), not a full benchmark sweep. See Limitations.
All four strategies share one foundation; only the orchestration layer changes.
ORCHESTRATION (1) direct (2) Step Functions (3) A2A coordinator (4) SFN + delegation cap
────────────────────────────────────────────────────────────────────────────────────────
Strands agents on Bedrock AgentCore Runtime (native A2A)
| MCP (Cognito JWT)
AgentCore Gateway ─────────► Lambda tools
|
Bedrock (Claude Sonnet 4.5 / Haiku / Llama / Mistral)
Observability: CloudWatch + X-Ray
- AgentCore Runtime hosts the role-parameterized Strands agent; the coordinator delegates to specialists via A2A (invoking the same runtime).
- AgentCore Gateway exposes a Lambda tool backend as MCP tools (Cognito-auth).
- Step Functions drives the explicit DAG and the guarded hybrid.
git clone <this-repo> && cd orchestra-bench
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtollama pull qwen3:8b qwen3:14b llama3.1:8b
python -m bench.taskgen --per-category 120 --out tasks/generated.json # labelled suite
python -m bench.experiment --provider ollama --models "qwen3:8b,qwen3:14b,llama3.1:8b" \
--trials 20 --tasks tasks/generated.json --out results/local
python -m bench.report # tables (md/csv/tex) + significance
python -m bench.plots # figuresgit clone https://github.com/sierra-research/tau2-bench third_party/tau2-bench
cd third_party/tau2-bench && uv sync && uv pip install boto3 && cd ../..
# local models:
third_party/tau2-bench/.venv/bin/python research/run_tau2.py --domain retail \
--limit 10 --trials 3 --models "qwen3:8b,glm4:9b" --user-model qwen3:14b \
--rpm 1000 --out results/tau2_local
# Bedrock (rate-limited accounts: keep --rpm low + --workers 1):
research/run_tau2.py --domain retail --models "bedrock/us.anthropic.claude-haiku-4-5-..." \
--user-model "bedrock/...haiku..." --rpm 8 --workers 1 --out results/tau2_bedrockResumable (skip-completed + retry budget), per-episode timeout, cost + delegation telemetry,
and a global rate-limiter with backoff. See research/run_tau2.py --help.
python aws/deploy_foundation.py # IAM + Lambda tools
python aws/deploy_gateway.py # AgentCore Gateway (MCP)
python aws/deploy_agents.py # Strands agents on AgentCore Runtime
python aws/deploy_stepfunctions.py # explicit + hybrid state machines
python aws/run_live.py --per-category 3 --trials 3 --out results/aws_live
python aws/teardown_aws.py # remove everythingDeploying incurs AWS cost. Set a budget alarm.
teardown_aws.pyremoves all resources (Runtime, Gateway, Cognito, Step Functions, Lambdas, roles).
Provider-agnostic RunRecord → success (answer-based), pass^k (τ-bench-style reliability),
tool-selection accuracy (BFCL-style, reported separately from success), path-entropy
(determinism), a typed failure taxonomy (loop / cascade / wrong_tool / error), latency,
tokens, and cost, with bootstrap CIs and paired-bootstrap significance.
bench/ provider-neutral harness (orchestrators, metrics, scorers, taskgen, report, plots)
aws/ AWS reference deployment (Lambda, Gateway, Runtime agents, Step Functions, teardown)
research/ tau2-bench adapter (the 4 strategies as tau2 agents) + matrix runner
config/ default run config
data/ synthetic incident dataset
- The local suite is synthetic-but-labelled. The τ²-bench run above adds external validity, but it is validation-scale: n=15/cell (5 of 114 retail tasks, 3 trials), so the confidence intervals are wide. Treat it as directional, not a full benchmark sweep.
- Only two models produced usable τ²-bench signal (Haiku 4.5, Sonnet 4.5). Llama 3.3 70B scored 0 throughout; Pixtral Large is excluded (100% episode timeouts at the 420 s cap).
- Success saturates at frontier scale on the local suite: on easy tasks every strategy wins, so the signal is in behavior (delegations, cascade) and cost/latency, not accuracy.
- n is modest; treat the numbers as directional. Determinism is measured at temp 0.
- Roadmap: a deeper τ²-bench sweep (more tasks/trials for tight CIs) · more domains (airline, telecom) · temperature sweep · larger n.
Apache-2.0. See LICENSE.