Embedded tool-call parser: cover the non-JSON dialect tail (bare python-call, tool-name-tag, Claude <invoke>) — follow-up to #295

[cboettig]

Follow-up to #295 / PR fix/embedded-tool-call-dialects (2da1b99).

That patch closes the JSON-family embedded tool-call dialects (<tool_call>/<tool_code> wrappers, flat + OpenAI-nested bodies, parameters alias, bare/concatenated JSON) — verified, and it takes the barred-owl leak rate from ~50% to ~0% in a 12-trial sweep. This issue tracks the non-JSON tail it does not cover.

Status: Step 1 (recovery via looksLikeAttemptedToolCall) is done in #298 — log-validated: 0 of 23 real terminal-leak strings leak with zero recovery now. Steps 2–3 (first-pass parsing of the tail) remain. Also note: the merged #296 added a <function=NAME> handler, so that row of the table now parses first-pass (updated below).

Why this matters (don't let 50%→0% read as "solved")

The 0% sweep was a stochastic round where nimbus-qwen happened to emit JSON dialects. In an earlier round the model emitted the bare python-call form and the "Show me barred owl occupancy" question leaked 4/4 — that exact form is still uncovered. Under reasoning-on this model rotates dialects unpredictably, so the tail recurs.

Critically, these forms leaked with zero recovery attempt: parseEmbeddedToolCalls doesn't parse them and looksLikeAttemptedToolCall returned false, so the #288 re-prompt never fired either. Step 1 (#298) fixes the recovery half — the looksLikeAttemptedToolCall column below is now ✅ for the whole tail; first-pass parsing is still ❌ pending steps 2–3.

Status (unit-tested; recovery column reflects #298)

Dialect	first-pass parse	looksLikeAttemptedToolCall	net
show_layer(layer_id="…") bare python-call (dbl & single quote)	❌	✅ (#298)	re-prompt only
<show_layer>{"layer_id":"…"}</show_layer> tool-name-as-tag	❌	✅ (#298)	re-prompt only
<show_layer><layer_id>…</layer_id></show_layer> tool-name-as-tag (XML body)	❌	✅ (#298)	re-prompt only
<parameter=function>show_layer</parameter><parameter=layer_id>…</parameter> mangled	❌	✅ (#298)	re-prompt only
<model_calls><invoke name="get_schema"><parameter name="dataset_id">…</parameter></invoke></model_calls> Claude XML	❌	✅ (#298)	re-prompt only
<tool_call>\n<function=query", "arguments":{…} corrupted tag inside wrapper	✅ (#296)	✅	parsed first-pass

All observed live from qwen / DSE-nimbus on barred-owl with reasoning on. (Reasoning off = clean, but that isn't an option — the app needs reasoning for answer accuracy.)

Proposed fixes, in priority order

1. Broaden looksLikeAttemptedToolCall (cheapest, do first). ✅ Done in fix(agent): flag the non-JSON tool-call tail so #288 recovery fires (#297 step 1) #298. Flags bare known_tool(, tool-name-as-tag <known_tool>/<known_tool , and the <function=/<parameter=/<invoke/<model_calls> markers so the #288 re-prompt fires for the whole tail. Gated on toolRegistry.has(name); scans all matches (matchAll) so a real tool tag preceded by hallucinated reasoning tags (<antThinking>, </think>) is still caught.
2. Run the tool_name(...) funcMatch on unwrapped content. Today the funcMatch handler only runs inside a <tool_call>/<tool_code> wrapper (_pushEmbeddedCall). Extend it to scan bare content for known_tool(<args>), signature-gated. Also parse python-style kwarg="val" / kwarg='val' args (both quote styles) — not just JSON — since parseLenientJSON('layer_id="x"') fails.
3. (Optional) tool-name-as-tag and Claude <invoke name=…>. Recognize <known_tool>…</known_tool> (JSON or <param>…</param> body) and <invoke name="tool"><parameter name="k">v</parameter> and normalize to {name, args}.
4. Grow stripEmbeddedCalls alongside steps 2–3. Whatever new dialects the first-pass handlers recognize, the display-strip must strip too, or a recovered call echoes back to the user as prose. fix(agent): parse foreign tool-call dialects first-pass instead of re-prompting (#295) #296 wired stripEmbeddedCalls to reuse _scanJSONObjects, but the bare python-call / tool-name-tag / <invoke> forms are non-JSON, so each needs an explicit strip pattern.

Verification

• Unit-test each row above → expected {name, args} (mirror the JSON cases already in the patch).
• Replay via open-llm-proxy/headless/run.js --model qwen on barred-owl config with reasoning on; the "Show me barred owl occupancy across the range" question is the reliable trigger for the bare-python-call form.

[cboettig]

Step 1 done → PR #298; two refinements to this issue

Step 1 (broaden looksLikeAttemptedToolCall) is implemented in #298, gated on toolRegistry.has. One implementation note worth recording: scanning only the first <word>/word( match isn't enough — the real tool tag is routinely preceded by hallucinated reasoning tags (<antThinking>, </think>), so it must matchAll. My first cut missed exactly the <show_layer><layer_id>… rows for this reason; caught by replaying the logs.

Log-validated (barred-owl/qwen, 2026-07-09, 23 real terminal-leak strings): 10 now parse first-pass (merged #296) + 13 now trigger the recovery re-prompt = 0 leak with zero recovery (was 13 silent).

Two refinements to the issue itself:

1. The status table's last row is stale. It's against 2da1b99; the merged fix(agent): parse foreign tool-call dialects first-pass instead of re-prompting (#295) #296 added a <function=NAME> handler, so <tool_call>\n<function=query", "arguments":{…} now parses first-pass, not "re-prompt only." One fewer row for steps 2–3.

2. stripEmbeddedCalls must grow alongside the parser in steps 2–3. Whatever new dialects the first-pass handlers recognize, the display-strip has to strip too, or a recovered call echoes back to the user as prose. fix(agent): parse foreign tool-call dialects first-pass instead of re-prompting (#295) #296 wired stripEmbeddedCalls to reuse _scanJSONObjects, but the bare python-call / tool-name-tag / <invoke> forms are non-JSON, so each needs an explicit strip pattern. Worth adding as an explicit checklist item under step 3.

[cboettig]

48h log check on nimbus (qwen) — the residual dialects are recovered, not user-facing

Pulled the proxy logs for the past 48h (2026-07-11 → 07-13), model qwen (nimbus): 1816 responses / 240 sessions. Foreign-dialect leak signature = a RESPONSE with has_tool_calls=false whose content matches a tool-call shape (<tool_call|<tool_code|<function=|"type":"function"|<invoke|<|python_tag|); "terminal" = it's the last response in its session (leaked to the user), else the loop continued (handled).

origin class	sessions	responses	dialect emissions	terminal (to user)
browser-app (real client)	27	243	11	0
headless (origin=None)	213	1573	3	3

Findings

1. Zero terminal leaks on real browser-client traffic. All 11 foreign-dialect emissions were mid-loop — first-pass parsed or #288 recovery fired and the loop continued; the user never saw a dead end. This is the merged #295/#296/#298 + #288 stack doing its job on live nimbus output.
2. All 3 terminal leaks are headless benchmark runs — origin=None, client=None, and the two canonical hard gold questions ("reptile richness top-20%", "major rivers order 6+ in 30x30"). Each ran the loop to msg_count 43–50 and hit the iteration cap; the leak is the capped final turn's output, not a recovery failure.
3. The leaked form is specifically the pure <function=NAME><parameter=…> XML dialect (plus one <invoke> observed but handled on real traffic) — precisely the steps 2-3 residual this issue tracks.

Implication for this issue. The residual dialects are real, but on deployed nimbus they are recovered rather than user-facing. They surface terminally only when a hard question exhausts the loop cap under headless benchmarking. Steps 2-3 (first-pass handlers for <function=><parameter=>, <invoke>, bare python-call + matching stripEmbeddedCalls) remain worth doing — first-pass parse saves a recovery round-trip and would salvage the cap-exhaustion case — but this reclassifies them as a low-urgency correctness nicety, not a live user-facing leak.

Caveat: browser-app volume here is modest and skewed to stac30x-devcheck/parentprobe dev sessions (they still exercise the real client recovery path, so the "recovered, not terminal" conclusion holds). Query artifacts in scratchpad; method per the #295/#296 leak-classification technique.