Reproducible benchmark for Mixture-of-Agents (MoA) presets on security-critical TypeScript wallet code.
8 presets x 10 tasks = 80 runs. Judge: GPT-5.5 (xhigh).
| # | Preset | Score | Time | Tokens |
|---|---|---|---|---|
| 1 | kodamoa-minimax | 20.8/25 | 64.5s | 7785 |
| 2 | single-glm | 20.7/25 | 26.2s | 1069 |
| 3 | kodamoa | 19.9/25 | 58.7s | 7909 |
| 4 | current-baseline | 19.4/25 | 62.1s | 7108 |
| 5 | single-deepseek | 19.3/25 | 26.9s | 1342 |
| 6 | single-minimax | 16.8/25 | 37.0s | 2738 |
| 7 | gpt-5.5-baseline | 16.6/25 | 27.3s | N/A |
| 8 | gpt-5.5-moa | 15.3/25 | 63.1s | 5623 |
Takeaway: MoA barely beats single GLM-5.2 (20.8 vs 20.7) but costs 2.5x latency and 7.3x tokens. GPT-5.5 underperforms open-source models on wallet code.
Full analysis: ANALYSIS.md Visual dashboard: dashboard/index.html
6 presets x 10 tasks x 1 judge = ranked comparison of which MoA constellation produces the best security-critical TypeScript wallet code.
- Tasks: 10 real TypeScript Bitcoin wallet coding tasks (sats/BTC conversion, PSBT operations, timing-safe compare, secure random, address validation, Lightning invoices, BIP-39 mnemonic)
- Presets: 8 configurations — 3 MoA (3 references + aggregator), 3 single-model, 2 GPT-5.5 (alone and as MoA aggregator)
- Judge: GPT-5.5 via codex CLI (xhigh reasoning), blinded — does not know which preset generated the code
- Linter: 8 custom security rules for wallet code (no Math.random near crypto, no float satoshi, no eval, timing-safe compare, no key in logs, no hardcoded xpub, etc.)
kodamoa-bench/
configs/
presets/ # 8 preset YAML configs
judges/ # 3 judge configs (GPT-5.5, Opus 4.8, Gemini 2.5 Pro)
security/
eslint-rules/ # 8 security linter rules
linter.ts # Linter engine
linter.test.ts # 8 vitest tests
harness/
call-moa.ts # MoA caller (ollama-cloud + codex-cli)
call-judge.ts # Judge caller (codex/claude/gemini CLI)
run-task.ts # Single task pipeline
run-all.ts # Full benchmark runner
score.ts # Aggregation + leaderboard
tasks/
domain/
manifest.json # 10 task definitions
results/
leaderboard.json # Full results
dashboard/
index.html # Visual dashboard
chart-scores.svg # Score chart
chart-wallclock.svg # Time chart
chart-tokens.svg # Token chart
heatmap.svg # Per-task heatmap
ANALYSIS.md # Full analysis + recommendations
git clone https://github.com/eminogrande/kodamoa-bench
cd kodamoa-bench
npm install
cp .env.example .env # add OLLAMA_API_KEY from ~/.hermes/.env
npx tsx harness/run-all.ts --smoke # 1 task, 1 preset (~30s)
npx tsx harness/run-all.ts # full 80-run benchmark (~40min)- Node.js 18+
- Ollama Cloud API key (in .env)
- codex CLI installed and logged in (for GPT-5.5 judge)
- Optional: claude CLI, gemini CLI (for additional judges)
MIT