fix(agent): keep image tools armed for anaphoric follow-ups after an image turn#4124
Draft
nsgds wants to merge 1 commit into
Draft
fix(agent): keep image tools armed for anaphoric follow-ups after an image turn#4124nsgds wants to merge 1 commit into
nsgds wants to merge 1 commit into
Conversation
6 tasks
nsgds
force-pushed
the
fix/image-tools-followup-rearm
branch
2 times, most recently
from
June 15, 2026 15:30
0e489ab to
b61179e
Compare
…llow-ups
A vague reply right after a real tool turn ('do another', 'add a couple
more to it', 'try again') references the prior action only anaphorically:
no domain keyword, and not a terse continuation. _classify_agent_request
returns domains=[], the just-used tool is stripped from selection, and a
small model confabulates the action instead — observed live as a
fabricated image link, and as a wrong-tool cascade that built a phantom
document while claiming to update a note.
When a turn matches NO domain (and isn't a continuation), re-arm the
domain of any tool a recent assistant turn actually used, reusing
_DOMAIN_TOOL_MAP as the tool->domain reverse map — no per-domain special
casing. Deliberately narrow: keyword-matched turns are untouched,
casual/greeting turns exit before the re-arm, explicit continuations
already inherit context, and unmapped (user-registered MCP) tool names
are inert. Tool use is read from persisted tool_events metadata (survives
session reload) with a generated-image URL fallback. Disabled tools stay
disabled: domain seeding feeds the same schema filter as every other
selection path.
Co-Authored-By: Claude <noreply@anthropic.com>
nsgds
force-pushed
the
fix/image-tools-followup-rearm
branch
from
July 13, 2026 09:43
b61179e to
64be596
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After a successful tool turn, the most natural follow-up is anaphoric — "do another
one", "add a couple more to it", "try again". These carry content, so they are not
terse continuations (
_EXPLICIT_CONTINUATION_RE), and they reference the prior actionwithout naming it, so no domain keyword matches.
_classify_agent_requestreturnsdomains=[], the just-used tool is stripped from selection, and retrieval on theliteral follow-up text has no semantic signal. The model — which was just shown the
action succeeding — attempts to repeat it tool-less and confabulates. Observed live
on
dev: a fabricated markdown image link aftergenerate_image, and a wrong-toolcascade after
manage_notes(edit_document→create_document→ phantom document,while claiming to have updated the real note — silent data divergence).
The fix keys on behaviour instead of language: when a turn matches no domain (and
is not an explicit continuation), re-arm the domain of any tool a recent assistant turn
actually used, reusing the existing
_DOMAIN_TOOL_MAPas the tool→domain reverse map —no per-domain special casing. Recent tool use is read from the persisted
tool_eventsmessage metadata (survives session reload), with a generated-image URL fallback.
Deliberately narrow:
untouched, so this cannot bloat tool selection on keyword-matched turns.
runs.
domain matching.
disabled_toolsschema filterand tool-policy checks as every other selection path; unmapped (user-registered
mcp__) tool names are inert.tracked in
_classify_agent_requestdomain keywords are English-only — non-English agent requests skip tool retrieval entirely (low_signal short-circuit) #3713/[agent mode] _classify_agent_request() strips all tools for non-English queries — only 3 ALWAYS_AVAILABLE tools sent #3766.Controlled A/B (same model, identical two-turn scripts, with/without this commit):
turn 2 "do another one, but at sunrise" / "actually add a couple more to it". Without:
domains=[], retrieval surfaced unrelated tools (byte-identical lists across runs —embedding retrieval provably cannot rescue these turns), model confabulated. With:
domains=['images']/['notes_calendar_tasks']and the just-used tool stayed intools_sent; the image case re-executed end-to-end (real regenerated image). In thenotes case the fix restores availability and the domain rules pack, though a small
model can still mis-pick between offered sibling tools (notes vs. document editing) —
a separate selection-quality problem this PR doesn't claim to solve.
Relationship to neighbours: #5157 grows the terse-continuation list ("please proceed")
— complementary, since contentful anaphora can never be allowlisted. #4366 pins core
exec/IO tools into the static baseline — different layer; no static baseline should pin
per-domain tools like
manage_notes, which is exactly what this re-arm restoresdynamically. Scope note: this PR previously carried an image-specific version of the
same idea; testing showed the failure is domain-general, so it now re-arms via the
domain map generically (and the
imagesdomain from #3606 joins automatically whenthat lands — no dependency either way).
Target branch
dev, notmain.Linked Issue
Part of #3605 (the image half of this bug surfaced there). A dedicated issue for the
domain-general anaphoric case is forthcoming, aimed at the #5332 tool-selection
subtracker's scope (availability/recovery before a valid tool call); this PR will be
relinked to it when filed.
Type of Change
Checklist
devdocker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.How to Test
python -m pytest tests/test_anaphoric_followup_rearm.py -q— covers the livenotes repro, the reverse map for every domain in
_DOMAIN_TOOL_MAP, theown-domain-untouched guarantee, the casual-turn guard, the lookback bound, MCP-name
inertness, the generated-image URL fallback, and fresh-chat behaviour.
(
Make a note titled "Grocery run" to buy milk and eggs.), then sendactually add a couple more to it — bread and coffee. Without this PR the turn logslow_signal=True domains=[]and the model reaches for document tools; with it,domains=['notes_calendar_tasks']andmanage_notesstays in[agent-debug] ... tool_names=.do another one, but at sunrise, once an images domain exists).Visual / UI changes — REQUIRED if you touched anything that renders
Not applicable — backend classifier only (
src/agent_loop.py+ a test file); norendering code touched. Before/after screenshots of the user-visible effect (fabricated
result vs. real re-execution) attached in a comment.