feat(agents): multi-model parallelism — small classifier + large writer via Lemonade multi-model loading

[itomek]

Problem

GAIA agents today bind to a single LLM per agent instance (LemonadeClient is single-model). That's fine for a chat-only flow, but it forces every cognitive sub-task — fast classification, slow reasoning, structured-data emission, prose summarization — onto the same model. In practice we already see this break:

Concrete trigger surfaced in #995: the EmailTriageAgent's pre_scan_inbox returns a {\"kind\": \"email_pre_scan\", ...} envelope; the system prompt asks the LLM to echo it verbatim inside fenced code blocks so the frontend's EmailPreScanCard can mount. On Gemma-4-E4B (a chat-tuned model), the LLM reliably paraphrases the JSON into prose instead — its RLHF reward explicitly disprefers regurgitating structured input. Card never mounts. We landed a backend-injected fence as a deterministic workaround in src/gaia/ui/sse_handler.py (search for _pending_render_payloads and _capture_render_payload).

The hack works but is the wrong shape long-term: as we add more structured-render tools (calendar cards, doc-summary cards, code-diff cards), the SSE handler accumulates more bypass paths around the LLM. The right architecture is two models running in parallel — one tool-use-tuned for structured emission, one chat-tuned for prose — routed per cognitive role.

Why this matters now

This unblocks several things simultaneously, not just email-triage card rendering:

Use case	Small fast model	Large smart model	Win
Tier-1 triage in triage_heuristics.py	0.6B classifier	35B reasoning for confident=False cases	Bounds the 56s pre_scan_inbox latency — currently the dominant Adrian/Ramin demo blocker
Ambient mailbox watcher (planned for the daily-driver vision)	0.6B–1B classifier	35B for triggers	Background polling at near-zero cost
Speculative decoding (Lemonade-native)	Draft model	Target model	2-3× throughput on the same hardware
Reply draft + tone classifier	Tone model	Draft writer	Quality + speed
Structured-emission split (this issue's named driver)	Tool-use-tuned (Hermes-3, Qwen-Coder)	Gemma / Qwen chat	Removes the src/gaia/ui/sse_handler.py _pending_render_payloads hack

Proposed scope (v1)

• LemonadeMultiClient (or extend LemonadeClient with model_for_role: dict[Role, str]) under src/gaia/llm/.
• Per-role routing inside the agent loop: structured/tool-use turns → small instruct-tuned model; conversational turns → chat-tuned model.
• Memory budget management — two large models loaded at once on Strix Halo's 32 GB requires explicit pinning + fallback (e.g. unload the chat model when the classifier is hot if RAM is tight).
• Eval harness extension: per-role model selection in src/gaia/eval/ so we can A/B model pairings against today's single-model baseline.
• Concrete first agent migration: EmailTriageAgent (this issue's named consumer) — see acceptance criteria.

Acceptance criteria

• LemonadeMultiClient lands with role-routed dispatch and at-least 2-model concurrent loading on Strix Halo (32 GB).
• EmailTriageAgent consumes the new client: tool-use-tuned model handles pre_scan_inbox → structured emission; chat-tuned model handles prose summary.
• Remove the SSE backend-fence hack added in PR #995. Specifically delete _pending_render_payloads / _capture_render_payload / _drain_render_payloads / the _RENDER_TOOL_TO_LANG map from src/gaia/ui/sse_handler.py; the SSE handler should pass-through the LLM's output unmodified for structured payloads. The system prompt for EmailTriageAgent re-tightens to instruct the structured-emission model to emit the fence (with a worked example).
• Eval harness reports card-mount-success-rate for pre_scan_inbox over 20 trials × 3 model pairings; deterministic-100% baseline (today's hack) included for comparison.
• Tier-1 triage wired in triage_heuristics.py using a 0.6B classifier on confident=False cases instead of falling through to the heavy 35B model.
• Docs: a short note in docs/sdk/sdks/llm.mdx on the role-routing pattern + memory implications.

Effort estimate

1–2 weeks with eval harness, including the model A/B work to pick the right tool-use-tuned default. Lemonade-side work (ensuring concurrent model load + per-request routing) needs to be confirmed against https://lemonade-server.ai/models.html — at first glance this is a client-side responsibility (set model= per request) but memory pinning is the open question.

Refs

• PR #995 — adds the hack this issue is about removing.
• #645 — Email Triage Agent umbrella; this is one of its dependencies.

[github-actions]

Solid write-up — the named driver (the email_pre_scan fence-emission failure on Gemma-4-E4B) is concrete and the win extends well beyond it. A few things worth tying in before this gets scoped:

Cross-references the proposal should pick up

• Hybrid model routing: local + cloud task routing for cost optimization #632 (hybrid model routing: local + cloud) is the same architectural primitive, just with the "second model" potentially living on Claude/OpenAI instead of a second Lemonade slot. If LemonadeMultiClient exposes a model_for_role map, the natural generalization is client_for_role: dict[Role, LLMClient] — that lets feat(agents): multi-model parallelism — small classifier + large writer via Lemonade multi-model loading #1000 and Hybrid model routing: local + cloud task routing for cost optimization #632 share the routing layer instead of building two parallel ones. Worth aligning the API shape now.
• Agent loop rejects parallel tool_calls from native tool-calling models (Gemma-4-E4B default) #944 (parallel tool_calls rejected by Agent._parse_response) is in the same neighbourhood: the structured-emission model in this proposal will likely be a tool-calling-trained one (Hermes-3, Qwen-Coder), and those models routinely emit multi-call responses. If Agent loop rejects parallel tool_calls from native tool-calling models (Gemma-4-E4B default) #944 isn't fixed first, the structured-role path in this design will hit the NotImplementedError at src/gaia/agents/base/agent.py:978-982. Suggest making Agent loop rejects parallel tool_calls from native tool-calling models (Gemma-4-E4B default) #944 a hard dependency.
• Capability-based agent routing and multi-agent orchestration #464 (capability-based agent routing & multi-agent orchestration) is the agent-level analogue of what this issue proposes at the model level. Worth a sentence on whether role-routing-within-an-agent and agent-routing-across-agents share machinery (I think they should — both are dispatch-on-capability).

Surfacing open questions for the v1 scope

🟡 Lemonade memory pinning is correctly flagged as the open question, and it's the load-bearing one. Per-request model= against lemonade-server is documented and works, but cold-loading a 35B GGUF on confident=False cases will torch the latency budget the issue is trying to defend (the 56s pre_scan_inbox will get worse, not better, if the heavy model isn't already resident). Two pre-conditions worth explicit acceptance criteria:

1. Confirm with @kovtcharov-amd / @jeremyfowers whether Lemonade can hold two concurrent llama-server processes pinned without thrashing on Strix Halo's 32 GB. A 0.6B + 35B GGUF pair is ~22-24 GB resident — tight but workable. A 4B + 35B pair won't fit.
2. Define the eviction policy when memory is tight: LRU on the chat slot? Pin the small classifier always-resident, page the writer? This decision drives whether the small-model tier is "ambient watcher" viable (Always-on agent: heartbeat, scheduler, and event hooks (autonomy engine) #634-adjacent) or just "occasional triage helper".

🟢 The single-model assumption is at src/gaia/llm/lemonade_client.py:731 (self.model = model) — the OpenAI-compatible body in lemonade_client.py already accepts a model field per request, so a LemonadeMultiClient that overrides _dispatch to inject model_for_role[role] is mostly a constructor change plus role-tagged callsites. The bigger lift is the agent-loop awareness: who decides what role a turn is? Static (per-tool annotation), dynamic (router LLM), or hybrid? This isn't in the v1 scope above and probably should be — without a role-tagging convention, the routing logic is just "structured tool turns vs. conversational turns" heuristic, which gets brittle fast.

On removing the SSE hack (the named acceptance criterion)

⚠️ PR #995 is still open as of this comment — the hack this issue proposes removing hasn't actually landed on main yet (_pending_render_payloads doesn't exist in src/gaia/ui/sse_handler.py on main). That's good in one sense — if #1000 ships before #995, the hack never enters the codebase and the acceptance criterion becomes "ensure the structured-emission model emits the fence reliably enough that EmailPreScanCard mounts ≥95% of trials" rather than "delete code that exists." Worth deciding the merge order intentionally:

• Land feat(email): daily-driver UI pass — pre-scan card, in-chat Connect, session prefs #995 first → feat(agents): multi-model parallelism — small classifier + large writer via Lemonade multi-model loading #1000 removes the hack. Unblocks the daily-driver demo immediately, accepts ~2 weeks of tech debt in sse_handler.py.
• Land feat(agents): multi-model parallelism — small classifier + large writer via Lemonade multi-model loading #1000 first → feat(email): daily-driver UI pass — pre-scan card, in-chat Connect, session prefs #995 rebases without the hack. Cleaner architecturally but pushes the demo dependency on multi-model work that the effort estimate calls "1–2 weeks" (which usually means 3-4 in practice once eval A/B is honest).

The first option seems right given the demo timing pressure mentioned in #645 — but worth flagging explicitly so #995's _capture_render_payload doesn't get reviewed-and-merged with the false expectation that #1000 will land in the same sprint.

Suggestion: add to the eval criteria

The card-mount-success-rate over 20 trials × 3 model pairings is the right shape. Worth adding:

• Latency budget per role — the multi-model win evaporates if the small classifier adds more cold-load latency than it saves on inference. Suggest p50/p95 wall-clock for pre_scan_inbox end-to-end, not just card-mount-success.
• Memory pressure regression — gaia kill && gaia llm "hi" after a multi-model triage run, to confirm Lemonade isn't leaking processes when both models stay resident.

Tagging @kovtcharov-amd for the Lemonade-side memory-pinning question — that's the one external dependency that gates the whole effort estimate.