fix(agent): add image domain to tool-RAG classifier#3606
Conversation
e042736 to
6b3a4a1
Compare
|
Rebased onto current Two notes for reviewers reconciling approaches:
|
|
GitHub reports this PR is currently conflicting with |
6b3a4a1 to
2061b70
Compare
…e turn A follow-up like "do another" / "how about you try again to match your vision" refers to a just-generated image only anaphorically, so it matched no domain keyword and wasn't a recognized continuation -> _classify_agent_request flagged it low_signal -> the low_signal short-circuit stripped every tool to bare ALWAYS_AVAILABLE (no generate_image). Tool-less but asked to regenerate, nemotron-nano-12b confabulated a fake turn: invented <turn_data>/<tool_response> scaffolding, a bogus "SDXL2" call, and a fabricated image URL — shown to the user as if real. Add _assistant_used_image_tool(): if a recent assistant turn actually used generate_image/edit_image (recorded tool_events, or a /api/generated-image/ URL in its content), re-arm the `images` domain so the tools stay offered. Additive — retrieval_query and any other matched domains are untouched, so a genuine new request after an image turn still classifies normally (e.g. "now search the web" keeps web and merely gains images). This is the anaphoric-redo case odysseus-dev#3606's keyword-only images domain structurally cannot catch. Verified on the real session: "try again to match your vision" now yields domains=['images'], low_signal=False. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2061b70 to
d0ec9da
Compare
…e turn A follow-up like "do another" / "how about you try again to match your vision" refers to a just-generated image only anaphorically, so it matched no domain keyword and wasn't a recognized continuation -> _classify_agent_request flagged it low_signal -> the low_signal short-circuit stripped every tool to bare ALWAYS_AVAILABLE (no generate_image). Tool-less but asked to regenerate, nemotron-nano-12b confabulated a fake turn: invented <turn_data>/<tool_response> scaffolding, a bogus "SDXL2" call, and a fabricated image URL — shown to the user as if real. Add _assistant_used_image_tool(): if a recent assistant turn actually used generate_image/edit_image (recorded tool_events, or a /api/generated-image/ URL in its content), re-arm the `images` domain so the tools stay offered. Additive — retrieval_query and any other matched domains are untouched, so a genuine new request after an image turn still classifies normally (e.g. "now search the web" keeps web and merely gains images). This is the anaphoric-redo case odysseus-dev#3606's keyword-only images domain structurally cannot catch. Verified on the real session: "try again to match your vision" now yields domains=['images'], low_signal=False. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Rebased onto current |
The domain classifier has no image/media domain, so image-generation requests match no domain -> low_signal=True. On a FRESH first turn the direct low-signal reply path then offers the model NO tools at all, so every phrasing of 'draw/create/paint me X' deterministically fails; on contextual turns generate_image's inclusion is left to embedding- retrieval luck and no Image-rules pack exists to steer the model. This is the same failure shape the integrations domain fixed for api_call (odysseus-dev#3794 note in _classify_agent_request): matched no domain -> low-signal -> the tool never deterministically reaches the model. Add an images domain: detection regex + _DOMAIN_TOOL_MAP (generate_image, edit_image) + _DOMAIN_RULES pack, mirroring the contacts/integrations domains. Lookbehinds keep software artifacts (docker/disk image) out of the visual-media domain. Co-Authored-By: Claude <noreply@anthropic.com>
Pins the classifier half of the image-tool starvation bug: generation and edit phrasings resolve to the images domain (never low_signal -- including the fresh-first-turn phrasings observed failing 4/4 live, where low_signal would trigger the no-tools direct-reply path), the domain seeds both image tools, a rules pack exists (KeyError guard), and software 'images' (docker/disk) stay out of the visual-media domain. Co-Authored-By: Claude <noreply@anthropic.com>
d0ec9da to
d4bf893
Compare
|
Sorry for the slow turnaround on this! Rebased onto current Two substantive changes since your last look, both from re-verifying live on current
Body rewritten to describe the failure as it behaves on current Before (stock
After (this PR only, same request — image renders):
|


Summary
The tool-RAG domain classifier (
_classify_agent_request) has no image/media domain, soimage requests classify to
domains=[]→low_signal=True. On a fresh first turnthe direct low-signal reply path then offers the model no tools at all — "Draw a
picture of…" / "Create an image of…" deterministically fail with "I'm a text-based AI…"
(reproduced 4/4 phrasings on current
dev; log signature below). On contextual turns,image-tool inclusion is left to embedding-retrieval luck, and no "Image rules" pack is
ever injected (rules packs key off
_DOMAIN_TOOL_MAPmembership).This is the same failure shape the
integrationsdomain fixed forapi_call(see thein-code note citing #3794: "matched no domain, classified as low-signal, and the tool
never reached the schema filter"), and the fix has the same three-part shape as the
existing
contacts/integrationsdomains:_classify_agent_request(with lookbehinds so "docker image" /"disk image" stay out of the visual-media domain)
_DOMAIN_TOOL_MAP["images"] = {"generate_image", "edit_image"}(deterministic seeding)_DOMAIN_RULES["images"]usage pack (also required: a_DOMAIN_TOOL_MAPkey withouta matching rules entry makes
_domain_rules_for_toolsraiseKeyError)Verified live (A/B on the same stack and model, fresh first-turn prompt): without this
commit the request is refused; with it,
domains=['images'],low_signal=False, theagent calls
generate_imageonce and the image renders in chat. Companion PR #4123covers the second, independent half of the linked issue (missing native tool schema);
this PR makes the tool reach the model deterministically, #4123 makes the call
first-class once it does.
Target branch
dev, notmain.Linked Issue
Fixes #3605
Type of Change
Checklist
devdocker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.How to Test
python -m pytest tests/test_tool_rag_image_domain.py -q— covers: generation andedit phrasings resolve to the
imagesdomain and are never low-signal (includingfour fresh-first-turn phrasings observed failing live), the domain seeds both tools,
the rules pack exists, and non-image / software-artifact prompts ("pull the docker
image…") stay out.
devwithout this PR: open a new chat, agent mode, web searchoff, send "Draw a picture of a red fox sitting in a snowy forest." → the log shows
low_signal=True domains=[]+direct low-signal reply path, and the model refuses.domains=['images'], tool retrieval runs,generate_imageis seeded, and (for models that act on the tool guide) the imagegenerates and renders. Before/after screenshots in the comment below.
Visual / UI changes — REQUIRED if you touched anything that renders
Not applicable — backend classifier only (
src/agent_loop.py+ a test file); norendering code touched. The user-visible effect (image bubble instead of a refusal) is
shown in before/after screenshots attached in a comment.