GLM robustness: 5xx tool-session fallback ladder + dual thinking-key normalization

[cboettig]

Summary

While evaluating ordingj/nrp-glm-proxy (a single-user local Copilot proxy that hits the same ellm.nrp-nautilus.io GLM endpoint we do), two GLM-specific robustness techniques stood out as worth adopting here. Both are self-contained changes to llm_proxy.py.

1. 5xx fallback ladder for GLM tool sessions

Problem. GLM/vLLM on NRP can return an empty 500 late in long tool-use sessions. Today we map any provider 5xx straight to a 502 (llm_proxy.py L533-544) and the agent's turn just dies — no recovery.

Proposal. On a GLM 5xx (or a 200 with empty content — see below) on a request that carries tool history, retry with a progressively reduced tool payload before giving up. The upstream proxy's ladder:

1. set tool_choice: "none" (keep tools + transcript)
2. drop tools / tool_choice / parallel_tool_calls
3. drop tools + append a "give a final answer now" user message
4. (opt-in) flatten the tool transcript into plain chat messages

Because our proxy is non-streaming (we already buffer the full response.json()), this is easier for us than for the upstream streaming proxy — we can inspect the buffered response and retry inline.

Caveat / scope. This ladder targets the transcript-too-complex failure mode. We've also seen a separate NRP failure where the vLLM pod is wedged (vllm:generation_tokens_total=0 while up=1), which no retry shape can fix. The ladder recovers the first class, not the second — so we should log the two distinctly (e.g. distinguish "5xx with tool history, retry succeeded/failed" from "5xx, upstream wedged") rather than assume the ladder fixes all GLM 500s.

Before building: pull ~a week of our own logs to confirm how often GLM 5xx actually hits us mid-tool-session vs. the wedged-pod class, so the fix is sized to a real signal.

2. Thinking-key robustness

Problem. We inject exactly one chat_template_kwargs key per model — glm-5 → enable_thinking (config.json, llm_proxy.py L502-509). The upstream proxy sends both enable_thinking and thinking (plus clear_thinking: false) because "GLM/vLLM paths can disagree about which key the chat template expects." If a GLM 5.2 template revision ever expects thinking, our toggle silently no-ops and we'd never notice.

Proposal. For GLM, set both enable_thinking and thinking to the same value (and clear_thinking: false when thinking is enabled). Cheap, defensive, no behavior change when the current key is the right one.

3. (Bundle) Empty-200 detection

GLM sometimes returns 200 OK with finish_reason: stop and zero content. We currently log has_content=false and hand the agent an empty turn. Flagging "200 but empty" distinctly in our logs makes this class queryable, and it's the natural trigger condition for the fallback ladder above.

Out of scope

The upstream proxy's launchd/local/single-user design, streaming pass-through, and Copilot-specific reasoning_effort mapping are artifacts of its laptop-Copilot use case and don't apply here.