fix(agent): recover per-arg tool-call dialect leaks + short-circuit identical failing calls (#313) - #314
Merged
Merged
Conversation
…dentical failing calls (#313) Two model/tool-agnostic harness defenses, motivated by a live ca-30x30 z-ai/glm-5.2 failure (2026-07-14): GLM emitted a well-formed native add_hex_tile_layer call whose value_stats *value* was wrapped in its XML arg dialect (<arg_key>…</arg_key> <arg_value>{…}</arg_value>) — data intact inside the wrapper — then looped the identical broken call ~15x until the localOnlyStreak cap stopped it. - scrubArgDialectLeaks: unwrap <arg_value>/<parameter …> markers leaked into string-valued native tool args, wired into normalizeToolCallArguments so both history and execution see the recovered value. The existing parseEmbeddedToolCalls net misses this — it only runs when there are zero native tool calls, and here the outer arguments JSON is valid. - Repeated-failure short-circuit: when a round re-issues the byte-identical call(s) that just failed and all fail again, nudge once then checkpoint, instead of burning the full 15-round local-only cap. Defense-in-depth; root fix is normalizing the dialect at the proxy (open-llm-proxy#85), and the hex transcription surface itself is #276. 16 new tests in test/agent-tool-parse.test.js.
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #313.
What
Two model/tool-agnostic harness defenses, motivated by the live ca-30x30
z-ai/glm-5.2failure this morning (2026-07-14, "what fraction of ca hardwood is protected?").1. Per-argument dialect scrub (
scrubArgDialectLeaks, wired intonormalizeToolCallArguments)GLM emitted a well-formed native
add_hex_tile_layercall whosevalue_statsvalue was its XML arg dialect leaked in as a string:The existing
parseEmbeddedToolCallsrecovery misses this: it only runs when there are zero native tool calls (agent.js:219), and here the outerargumentsblob is valid JSON soJSON.parsesucceeds. The scrub unwraps<arg_value>/<parameter …>markers out of string-valued args and re-parses the payload — the data is intact inside the wrapper, so this call now succeeds instead of failing. Runs insidenormalizeToolCallArgumentsso both conversation history and execution see the recovered value.2. Repeated-failure short-circuit (agent loop)
After the leak, GLM degraded to
value_stats: ""and looped the byte-identical failing call ~15× until the bluntlocalOnlyStreakcap (=maxToolCalls, 15) stopped it. Now: when a round re-issues the identical call(s) that just failed and all fail again, the loop nudges once, then checkpoints — cutting ~15 wasted rounds to ~3.Scope
Defense-in-depth per
docs/design/tool-call-parsing.md. This is the client-side net; the root fix (normalize GLM-5.2's dialect at the proxy) is boettiger-lab/open-llm-proxy#85, and the architecture fix (stop transcribingvalue_statsthrough the LLM; fetch by hash) is #276 + boettiger-lab/mcp-data-server#316.Tests
16 new tests in
test/agent-tool-parse.test.js:scrubArgDialectLeaks(incl. the exact GLM leak,<parameter=>scalars, marker-free no-op),_isFailedResult, and three loop-level tests (leak recovered end-to-end so the tool executes with a real object; nudge-then-checkpoint on repeat; no short-circuit when the model recovers after the nudge). Full agent suite green (146 tests); no changes to browser-bound modules.