Decision: Dropping MoA, switching to single-glm as default for nuri-expo
Date: 2026-06-29
Status: Accepted
Author: Emin Mahrt (@eminogrande)
TL;DR
After running 80 benchmark runs (8 presets × 10 tasks) and analyzing 527 real review comments from the nuri-expo codebase, I'm dropping Mixture-of-Agents (MoA) and switching to single-glm (GLM-5.2 alone) as the default model for nuri-expo development. MoA burned 7x more tokens for +0.1 score improvement — not worth it. The real bottleneck isn't model capability, it's codebase context.
The MoA Hypothesis (and why I was wrong)
What I hoped would happen
Combine "different brains" — deepseek-v4-pro, glm-5.2, kimi-k2.7-code — to generate diverse solutions, then aggregate them with a final model. The theory: different models have different strengths, combining them produces better output. This is the MoA paper's promise.
What the benchmark showed
Rank Preset Score Linter Wall Tokens
1 kodamoa-minimax 20.8 100% 64.5s 7785
2 single-glm 20.7 100% 26.2s 1069
3 kodamoa 19.9 100% 58.7s 7909
4 current-baseline 19.4 100% 62.1s 7108
5 single-deepseek 19.3 100% 26.9s 1342
6 single-minimax 16.8 100% 37.0s 2738
7 gpt-5.5-baseline 16.6 100% 27.3s 0
8 gpt-5.5-moa 15.3 100% 63.1s 5623
MoA (kodamoa-minimax) scored 20.8/25. Single GLM-5.2 scored 20.7/25. That's +0.1 points — within noise — for 2.5x latency and 7.3x token cost.
GPT-5.5 as MoA aggregator scored 15.3/25 — the worst result. It second-guesses good open-source references and makes them worse.
Why MoA failed for nuri-expo
I analyzed 527 real review comments from nuri-expo PRs (#604–#764). The top issues:
| Pattern |
Count |
Requires |
| OTA/native boundary violations |
104 |
Knowing which APIs are native-only |
| Stale state / ghost balances |
76 |
App lifecycle knowledge |
| Race conditions / in-flight cancellation |
60 |
Understanding async flow |
| Accessibility (aria, focus, zoom) |
36 |
Web standards knowledge |
| Silent catch / swallowed errors |
27 |
Error handling discipline |
| Missing runtime validation at boundaries |
18 |
Defensive coding |
| localStorage for secrets |
16 |
Security model awareness |
| Missing zeroization of sensitive buffers |
11 |
Crypto hygiene |
| Dead code / unreachable branches |
11 |
Code hygiene |
| WebView sandbox / cross-origin |
10 |
Web security |
None of these are caught by "3 models see the problem differently." They require codebase context. A model that doesn't know your Arkade wallet lifecycle can't catch a ghost balance bug, no matter how many copies of it you run.
The MoA pattern assumes diverse reasoning helps. But for this codebase, the problem isn't reasoning diversity — it's context depth. Models need to understand:
- The wallet state machine (boarding → spendable → confirmed)
- Native module boundaries (OTA updates vs binary releases)
- Async patterns (race conditions in balance updates, SSE push cancellation)
- Security model (what's secret, what isn't, where keys live)
What I actually changed
1. Switched active preset to single-glm
# ~/.hermes/config.yaml
moa:
default_preset: single-glm
active_preset: single-glm
presets:
single-glm:
enabled: true
aggregator:
provider: ollama-cloud
model: glm-5.2
aggregator_temperature: 0.3
max_tokens: 8192
reference_models: [] # No references, no aggregation overhead2. No more MoA overhead
- No reference models (was: 3)
- No aggregator synthesis step (was: deepseek-v4-pro rewriting good code)
- 26s wall time, 1069 tokens per task (down from 64.5s, 7785 tokens)
- Temperature 0.3 for deterministic output — same prompt → same code
3. Kept MoA presets available
The kodamoa presets stay in config as a "premium review" mode for when extra scrutiny is needed on hard tasks. But the default is now simple, fast, and good enough.
The real bottleneck: PR review friction
The benchmark tested code generation (can models write functions). But the real pain point is PR review friction — the back-and-forth on PRs before they merge.
70%+ of the top 10 review patterns can be automated with lint rules. That's the next step — not more models, but better tooling.
What I'm building next
1. PR Review Simulation Benchmark
Instead of "can models write functions," test "can models catch the bugs reviewers catch."
- Take 20 real merged PRs from nuri-expo
- For each, extract the diff
- Run a model review on the diff
- Compare agent findings to actual review comments (the ground truth)
- Score precision + recall
This is the test that tells you if GLM-5.2 is good enough for this codebase.
2. Automated lint rules from review history
Convert the top 10 review patterns into ESLint rules:
no-silent-catch: Flag empty catch blocks and void on promisesno-localStorage-secrets: Flag localStorage.setItem near key/password/seedno-stale-snapshot: Flag getLastGood...Snapshot() without freshness checkota-native-boundary: Flag native module imports in OTA-shippable coderequire-zeroize: Flag Uint8Array holding sensitive data without .fill(0) after userequire-abort-controller: Flag async ops with cleanup but no cancellation
This catches 70% of review friction before a human reviewer sees it.
3. Parallel review board (not sequential MoA)
Instead of MoA (sequential: generate → aggregate), use a parallel review board:
- Agent A (security): checks for localStorage secrets, missing zeroization, WebView sandbox
- Agent B (state): checks for stale state, race conditions, in-flight cancellation
- Agent C (accessibility): checks for missing aria labels, focus traps, zoom blocking
- Merge findings, deduplicate, present as review checklist
This is faster (parallel), cheaper (each agent sees only the diff), and more targeted (each agent is prompted with the specific patterns from review history).
Why I don't regret the MoA experiment
I had to test it. The hypothesis was reasonable. The benchmark proved it wrong. Now I know:
- Model diversity ≠ codebase understanding
- Token cost ≠ quality (sometimes inverse — GPT-5.5 as aggregator made code worse)
- The bottleneck is context, not capability
- PR friction is the real metric, not benchmark scores
The benchmark repo stays public so others can learn from this: https://github.com/eminogrande/kodamoa-bench
Metrics to watch
- PR review cycles (target: -30%)
- Bugs caught pre-review by lint rules (target: 70% of top 10 patterns)
- Time to first response (target: -50% vs MoA)
- Token cost per PR (target: -70% vs MoA)
Links
Decision accepted. Moving forward with single-glm.
Decision: Dropping MoA, switching to single-glm as default for nuri-expo
Date: 2026-06-29
Status: Accepted
Author: Emin Mahrt (@eminogrande)
TL;DR
After running 80 benchmark runs (8 presets × 10 tasks) and analyzing 527 real review comments from the nuri-expo codebase, I'm dropping Mixture-of-Agents (MoA) and switching to single-glm (GLM-5.2 alone) as the default model for nuri-expo development. MoA burned 7x more tokens for +0.1 score improvement — not worth it. The real bottleneck isn't model capability, it's codebase context.
The MoA Hypothesis (and why I was wrong)
What I hoped would happen
Combine "different brains" — deepseek-v4-pro, glm-5.2, kimi-k2.7-code — to generate diverse solutions, then aggregate them with a final model. The theory: different models have different strengths, combining them produces better output. This is the MoA paper's promise.
What the benchmark showed
MoA (kodamoa-minimax) scored 20.8/25. Single GLM-5.2 scored 20.7/25. That's +0.1 points — within noise — for 2.5x latency and 7.3x token cost.
GPT-5.5 as MoA aggregator scored 15.3/25 — the worst result. It second-guesses good open-source references and makes them worse.
Why MoA failed for nuri-expo
I analyzed 527 real review comments from nuri-expo PRs (#604–#764). The top issues:
None of these are caught by "3 models see the problem differently." They require codebase context. A model that doesn't know your Arkade wallet lifecycle can't catch a ghost balance bug, no matter how many copies of it you run.
The MoA pattern assumes diverse reasoning helps. But for this codebase, the problem isn't reasoning diversity — it's context depth. Models need to understand:
What I actually changed
1. Switched active preset to single-glm
2. No more MoA overhead
3. Kept MoA presets available
The kodamoa presets stay in config as a "premium review" mode for when extra scrutiny is needed on hard tasks. But the default is now simple, fast, and good enough.
The real bottleneck: PR review friction
The benchmark tested code generation (can models write functions). But the real pain point is PR review friction — the back-and-forth on PRs before they merge.
70%+ of the top 10 review patterns can be automated with lint rules. That's the next step — not more models, but better tooling.
What I'm building next
1. PR Review Simulation Benchmark
Instead of "can models write functions," test "can models catch the bugs reviewers catch."
This is the test that tells you if GLM-5.2 is good enough for this codebase.
2. Automated lint rules from review history
Convert the top 10 review patterns into ESLint rules:
no-silent-catch: Flag empty catch blocks andvoidon promisesno-localStorage-secrets: FlaglocalStorage.setItemnear key/password/seedno-stale-snapshot: FlaggetLastGood...Snapshot()without freshness checkota-native-boundary: Flag native module imports in OTA-shippable coderequire-zeroize: FlagUint8Arrayholding sensitive data without.fill(0)after userequire-abort-controller: Flag async ops with cleanup but no cancellationThis catches 70% of review friction before a human reviewer sees it.
3. Parallel review board (not sequential MoA)
Instead of MoA (sequential: generate → aggregate), use a parallel review board:
This is faster (parallel), cheaper (each agent sees only the diff), and more targeted (each agent is prompted with the specific patterns from review history).
Why I don't regret the MoA experiment
I had to test it. The hypothesis was reasonable. The benchmark proved it wrong. Now I know:
The benchmark repo stays public so others can learn from this: https://github.com/eminogrande/kodamoa-bench
Metrics to watch
Links
Decision accepted. Moving forward with single-glm.