Skip to content

fix(agent): add image domain to tool-RAG classifier#3606

Open
nsgds wants to merge 2 commits into
odysseus-dev:devfrom
nsgds:fix/agent-image-tool-domain
Open

fix(agent): add image domain to tool-RAG classifier#3606
nsgds wants to merge 2 commits into
odysseus-dev:devfrom
nsgds:fix/agent-image-tool-domain

Conversation

@nsgds

@nsgds nsgds commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

The tool-RAG domain classifier (_classify_agent_request) has no image/media domain, so
image requests classify to domains=[]low_signal=True. On a fresh first turn
the direct low-signal reply path then offers the model no tools at all — "Draw a
picture of…" / "Create an image of…" deterministically fail with "I'm a text-based AI…"
(reproduced 4/4 phrasings on current dev; log signature below). On contextual turns,
image-tool inclusion is left to embedding-retrieval luck, and no "Image rules" pack is
ever injected (rules packs key off _DOMAIN_TOOL_MAP membership).

[agent-intent] latest='Draw a picture of a red fox...' low_signal=True domains=[]
[agent] direct low-signal reply path for latest='Draw a picture of a red fox...'

This is the same failure shape the integrations domain fixed for api_call (see the
in-code note citing #3794: "matched no domain, classified as low-signal, and the tool
never reached the schema filter"
), and the fix has the same three-part shape as the
existing contacts/integrations domains:

  • detection regex in _classify_agent_request (with lookbehinds so "docker image" /
    "disk image" stay out of the visual-media domain)
  • _DOMAIN_TOOL_MAP["images"] = {"generate_image", "edit_image"} (deterministic seeding)
  • _DOMAIN_RULES["images"] usage pack (also required: a _DOMAIN_TOOL_MAP key without
    a matching rules entry makes _domain_rules_for_tools raise KeyError)

Verified live (A/B on the same stack and model, fresh first-turn prompt): without this
commit the request is refused; with it, domains=['images'], low_signal=False, the
agent calls generate_image once and the image renders in chat. Companion PR #4123
covers the second, independent half of the linked issue (missing native tool schema);
this PR makes the tool reach the model deterministically, #4123 makes the call
first-class once it does.

Target branch

  • This PR targets dev, not main.

Linked Issue

Fixes #3605

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.

How to Test

  1. Unit (deterministic string classifier — no DB/embeddings needed):
    python -m pytest tests/test_tool_rag_image_domain.py -q — covers: generation and
    edit phrasings resolve to the images domain and are never low-signal (including
    four fresh-first-turn phrasings observed failing live), the domain seeds both tools,
    the rules pack exists, and non-image / software-artifact prompts ("pull the docker
    image…") stay out.
  2. Behavioral, on dev without this PR: open a new chat, agent mode, web search
    off, send "Draw a picture of a red fox sitting in a snowy forest." → the log shows
    low_signal=True domains=[] + direct low-signal reply path, and the model refuses.
  3. With this PR, same request → domains=['images'], tool retrieval runs,
    generate_image is seeded, and (for models that act on the tool guide) the image
    generates and renders. Before/after screenshots in the comment below.

Visual / UI changes — REQUIRED if you touched anything that renders

Not applicable — backend classifier only (src/agent_loop.py + a test file); no
rendering code touched. The user-visible effect (image bubble instead of a refusal) is
shown in before/after screenshots attached in a comment.

@nsgds

nsgds commented Jun 13, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current dev to refresh CI — the branch had drifted behind. No content change: still just the images domain in _classify_agent_request plus its test.

Two notes for reviewers reconciling approaches:

@pewdiepie-archdaemon

Copy link
Copy Markdown
Collaborator

GitHub reports this PR is currently conflicting with dev, so I can’t review or approve it cleanly yet. Please rebase/update against current dev and rerun the focused tool/RAG classifier tests once it is mergeable again.

@nsgds
nsgds force-pushed the fix/agent-image-tool-domain branch from 6b3a4a1 to 2061b70 Compare June 15, 2026 15:02
nsgds added a commit to nsgds/odysseus that referenced this pull request Jun 15, 2026
…e turn

A follow-up like "do another" / "how about you try again to match your
vision" refers to a just-generated image only anaphorically, so it matched
no domain keyword and wasn't a recognized continuation -> _classify_agent_request
flagged it low_signal -> the low_signal short-circuit stripped every tool to
bare ALWAYS_AVAILABLE (no generate_image). Tool-less but asked to regenerate,
nemotron-nano-12b confabulated a fake turn: invented <turn_data>/<tool_response>
scaffolding, a bogus "SDXL2" call, and a fabricated image URL — shown to the
user as if real.

Add _assistant_used_image_tool(): if a recent assistant turn actually used
generate_image/edit_image (recorded tool_events, or a /api/generated-image/
URL in its content), re-arm the `images` domain so the tools stay offered.
Additive — retrieval_query and any other matched domains are untouched, so a
genuine new request after an image turn still classifies normally (e.g. "now
search the web" keeps web and merely gains images). This is the anaphoric-redo
case odysseus-dev#3606's keyword-only images domain structurally cannot catch.

Verified on the real session: "try again to match your vision" now yields
domains=['images'], low_signal=False.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nsgds
nsgds force-pushed the fix/agent-image-tool-domain branch from 2061b70 to d0ec9da Compare June 15, 2026 15:30
nsgds added a commit to nsgds/odysseus that referenced this pull request Jun 15, 2026
…e turn

A follow-up like "do another" / "how about you try again to match your
vision" refers to a just-generated image only anaphorically, so it matched
no domain keyword and wasn't a recognized continuation -> _classify_agent_request
flagged it low_signal -> the low_signal short-circuit stripped every tool to
bare ALWAYS_AVAILABLE (no generate_image). Tool-less but asked to regenerate,
nemotron-nano-12b confabulated a fake turn: invented <turn_data>/<tool_response>
scaffolding, a bogus "SDXL2" call, and a fabricated image URL — shown to the
user as if real.

Add _assistant_used_image_tool(): if a recent assistant turn actually used
generate_image/edit_image (recorded tool_events, or a /api/generated-image/
URL in its content), re-arm the `images` domain so the tools stay offered.
Additive — retrieval_query and any other matched domains are untouched, so a
genuine new request after an image turn still classifies normally (e.g. "now
search the web" keeps web and merely gains images). This is the anaphoric-redo
case odysseus-dev#3606's keyword-only images domain structurally cannot catch.

Verified on the real session: "try again to match your vision" now yields
domains=['images'], low_signal=False.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nsgds

nsgds commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current dev. The only conflict was the new contacts domain landing in the same three spots (_DOMAIN_RULES, _DOMAIN_TOOL_MAP, the detector) — kept both contacts and images. Reran the focused tool-RAG classifier tests (image + contacts + keyword-hint suites, all green) and the PR is mergeable again with CI passing.

nsgds and others added 2 commits July 13, 2026 08:04
The domain classifier has no image/media domain, so image-generation
requests match no domain -> low_signal=True. On a FRESH first turn the
direct low-signal reply path then offers the model NO tools at all, so
every phrasing of 'draw/create/paint me X' deterministically fails; on
contextual turns generate_image's inclusion is left to embedding-
retrieval luck and no Image-rules pack exists to steer the model.

This is the same failure shape the integrations domain fixed for
api_call (odysseus-dev#3794 note in _classify_agent_request): matched no domain ->
low-signal -> the tool never deterministically reaches the model.

Add an images domain: detection regex + _DOMAIN_TOOL_MAP
(generate_image, edit_image) + _DOMAIN_RULES pack, mirroring the
contacts/integrations domains. Lookbehinds keep software artifacts
(docker/disk image) out of the visual-media domain.

Co-Authored-By: Claude <noreply@anthropic.com>
Pins the classifier half of the image-tool starvation bug: generation
and edit phrasings resolve to the images domain (never low_signal --
including the fresh-first-turn phrasings observed failing 4/4 live,
where low_signal would trigger the no-tools direct-reply path), the
domain seeds both image tools, a rules pack exists (KeyError guard),
and software 'images' (docker/disk) stay out of the visual-media domain.

Co-Authored-By: Claude <noreply@anthropic.com>
@nsgds

nsgds commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Sorry for the slow turnaround on this! Rebased onto current dev as requested — the
domain-registration spots remain the only overlap area (the new integrations domain
landed there since the last rebase; kept all). The focused classifier tests were rerun
and pass (7/7, tests/test_tool_rag_image_domain.py), plus a live A/B on a running
deployment (before/after screenshots below).

Two substantive changes since your last look, both from re-verifying live on current
dev:

  • The detection regex missed one failing phrasing from my test set ("Paint a
    watercolor of…" — no image-noun in it). Added paint/paintings, dropped an
    unreachable second pattern (anything it matched already matched the first), and
    excluded "docker image"/"disk image" via lookbehinds. Term choice in these
    detectors is judgment — happy to trim if you'd rather keep the vocabulary
    minimal.
  • Tests now pin the deterministic failure mode directly: four fresh-first-turn
    phrasings (observed failing 4/4 live) must never classify low-signal, since a
    low-signal first turn takes the direct no-tools reply path and cannot succeed
    regardless of retrieval.

Body rewritten to describe the failure as it behaves on current dev (the earlier
"retrieval is skipped" framing predates the low-signal retrieval changes). Full
two-defect analysis is in #3605; the native-schema half is #4123.

Before (stock dev, fresh first turn — refusal):

3606-before-stock-dev-refusal

After (this PR only, same request — image renders):

3606-after-fix-alone-cabin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent never offered generate_image/edit_image — image-generation requests starved of tools by domain classifier (no image domain)

3 participants