Skip to content

fix(agent): keep image tools armed for anaphoric follow-ups after an image turn#4124

Draft
nsgds wants to merge 1 commit into
odysseus-dev:devfrom
nsgds:fix/image-tools-followup-rearm
Draft

fix(agent): keep image tools armed for anaphoric follow-ups after an image turn#4124
nsgds wants to merge 1 commit into
odysseus-dev:devfrom
nsgds:fix/image-tools-followup-rearm

Conversation

@nsgds

@nsgds nsgds commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Scope note: this PR began as an image-specific fix (keep generate_image armed
after an image turn). Testing showed the failure is domain-general — the same
anaphoric follow-up strips manage_notes, manage_calendar, or any just-used
domain tool — so the fix is now generic: one re-arm over the existing domain map
instead of image special-casing. Kept as draft while the dedicated issue is filed
(per the #5332 subtracker's issue-first intake).

After a successful tool turn, the most natural follow-up is anaphoric — "do another
one", "add a couple more to it", "try again". These carry content, so they are not
terse continuations (_EXPLICIT_CONTINUATION_RE), and they reference the prior action
without naming it, so no domain keyword matches. _classify_agent_request returns
domains=[], the just-used tool is stripped from selection, and retrieval on the
literal follow-up text has no semantic signal. The model — which was just shown the
action succeeding — attempts to repeat it tool-less and confabulates. Observed live
on dev: a fabricated markdown image link after generate_image, and a wrong-tool
cascade after manage_notes (edit_documentcreate_document → phantom document,
while claiming to have updated the real note — silent data divergence).

The fix keys on behaviour instead of language: when a turn matches no domain (and
is not an explicit continuation), re-arm the domain of any tool a recent assistant turn
actually used, reusing the existing _DOMAIN_TOOL_MAP as the tool→domain reverse map —
no per-domain special casing. Recent tool use is read from the persisted tool_events
message metadata (survives session reload), with a generated-image URL fallback.

Deliberately narrow:

Controlled A/B (same model, identical two-turn scripts, with/without this commit):
turn 2 "do another one, but at sunrise" / "actually add a couple more to it". Without:
domains=[], retrieval surfaced unrelated tools (byte-identical lists across runs —
embedding retrieval provably cannot rescue these turns), model confabulated. With:
domains=['images'] / ['notes_calendar_tasks'] and the just-used tool stayed in
tools_sent; the image case re-executed end-to-end (real regenerated image). In the
notes case the fix restores availability and the domain rules pack, though a small
model can still mis-pick between offered sibling tools (notes vs. document editing) —
a separate selection-quality problem this PR doesn't claim to solve.

Relationship to neighbours: #5157 grows the terse-continuation list ("please proceed")
— complementary, since contentful anaphora can never be allowlisted. #4366 pins core
exec/IO tools into the static baseline — different layer; no static baseline should pin
per-domain tools like manage_notes, which is exactly what this re-arm restores
dynamically. Scope note: this PR previously carried an image-specific version of the
same idea; testing showed the failure is domain-general, so it now re-arms via the
domain map generically (and the images domain from #3606 joins automatically when
that lands — no dependency either way).

Target branch

  • This PR targets dev, not main.

Linked Issue

Part of #3605 (the image half of this bug surfaced there). A dedicated issue for the
domain-general anaphoric case is forthcoming, aimed at the #5332 tool-selection
subtracker's scope (availability/recovery before a valid tool call); this PR will be
relinked to it when filed.

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.

How to Test

  1. Unit: python -m pytest tests/test_anaphoric_followup_rearm.py -q — covers the live
    notes repro, the reverse map for every domain in _DOMAIN_TOOL_MAP, the
    own-domain-untouched guarantee, the casual-turn guard, the lookback bound, MCP-name
    inertness, the generated-image URL fallback, and fresh-chat behaviour.
  2. Behavioral (agent mode, native-tool endpoint): ask for a note
    (Make a note titled "Grocery run" to buy milk and eggs.), then send
    actually add a couple more to it — bread and coffee. Without this PR the turn logs
    low_signal=True domains=[] and the model reaches for document tools; with it,
    domains=['notes_calendar_tasks'] and manage_notes stays in
    [agent-debug] ... tool_names=.
  3. Same shape works for any domain (e.g. an image turn followed by
    do another one, but at sunrise, once an images domain exists).

Visual / UI changes — REQUIRED if you touched anything that renders

Not applicable — backend classifier only (src/agent_loop.py + a test file); no
rendering code touched. Before/after screenshots of the user-visible effect (fabricated
result vs. real re-execution) attached in a comment.

@github-actions github-actions Bot added the ready for review Description complete — ready for maintainer review label Jun 13, 2026
@nsgds
nsgds force-pushed the fix/image-tools-followup-rearm branch 2 times, most recently from 0e489ab to b61179e Compare June 15, 2026 15:30
@alteixeira20 alteixeira20 added the bug Something isn't working label Jun 26, 2026
…llow-ups

A vague reply right after a real tool turn ('do another', 'add a couple
more to it', 'try again') references the prior action only anaphorically:
no domain keyword, and not a terse continuation. _classify_agent_request
returns domains=[], the just-used tool is stripped from selection, and a
small model confabulates the action instead — observed live as a
fabricated image link, and as a wrong-tool cascade that built a phantom
document while claiming to update a note.

When a turn matches NO domain (and isn't a continuation), re-arm the
domain of any tool a recent assistant turn actually used, reusing
_DOMAIN_TOOL_MAP as the tool->domain reverse map — no per-domain special
casing. Deliberately narrow: keyword-matched turns are untouched,
casual/greeting turns exit before the re-arm, explicit continuations
already inherit context, and unmapped (user-registered MCP) tool names
are inert. Tool use is read from persisted tool_events metadata (survives
session reload) with a generated-image URL fallback. Disabled tools stay
disabled: domain seeding feeds the same schema filter as every other
selection path.

Co-Authored-By: Claude <noreply@anthropic.com>
@nsgds
nsgds force-pushed the fix/image-tools-followup-rearm branch from b61179e to 64be596 Compare July 13, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants