Skip to content

fix(agent): execute fenced tool calls with inline args and route bare email tool names#3681

Merged
alteixeira20 merged 19 commits into
odysseus-dev:devfrom
botinate:fix/fenced-tool-inline-args-email-dispatch
Jun 30, 2026
Merged

fix(agent): execute fenced tool calls with inline args and route bare email tool names#3681
alteixeira20 merged 19 commits into
odysseus-dev:devfrom
botinate:fix/fenced-tool-inline-args-email-dispatch

Conversation

@botinate

@botinate botinate commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Local models that use the fenced-block tool channel (Ollama, gemma, etc.) could not call any of the email tools in chat/agent mode. The model emits ```list_email_accounts {}, nothing executes, the raw fence leaks into the chat as text, and the model concludes it has no email access. Two bugs caused this: (1) the fence regex in src/tool_parsing.py required a newline directly after the tool name, so a call with arguments on the same line never matched; (2) even when a fence did parse, bare email tool names had no dispatch branch in src/tool_execution.py and fell through to "Unknown tool type".

Review follow-up (thanks @vgalin): routing bare email names had a security gap — is_public_blocked_tool() checks the bare name before dispatch, and NON_ADMIN_BLOCKED_TOOLS listed only 4 of the 14 email tools, so non-admin owners could have executed the other 10 (delete_email, bulk_email, …). The full email tool set is now defined once (BUILTIN_EMAIL_TOOLS in src/tool_security.py) and the non-admin blocklist, fence tags, bare-name dispatch, and native-call mapping all derive from it, so the sets can't drift apart. This also fixed the native-call mapping for 4 tools (search_emails, draft_email, draft_email_reply, ai_draft_email_reply) missing from the old tool_schemas.py copy; advertising them to models (schemas, prompt sections, RAG index) is part of the follow-up #4053. Additionally, the relaxed fence regex from the first commit could prefix-match longer fence tags (```python3 would have executed 3\nprint(...) as a python tool call — the same false-positive class as #3604); a (?![\w-]) boundary now prevents that.

Second review round (thanks @RaresKeY): two more gaps fixed. (1) The runtime denylist (disabled_tools) and the turn ToolPolicy were checked against the incoming block name only, while plan mode and the MCP settings toggle write the qualified mcp__email__... name — a bare list_emails fence bypassed a qualified block entry. Both gates now match bare and qualified spellings, in either direction. (2) The relaxed fence regex accepted any same-line text after a recognized tag, which made Markdown info strings executable (python title="example.py" ran as a python tool call). Same-line content now only counts as tool input when it starts with {/[ (JSON args); anything else stays display text, with parse/strip mirrored.

Third round (thanks @RaresKeY): (1) the same-line rule is now tag-aware and JSON-validated through one predicate shared by parse and strip — bash/python never take same-line args, other tags require valid standalone JSON — so brace-shaped fence attributes like bash {title="setup"} stay visible and inert. (2) The friendly disable_tool email toggle derives from BUILTIN_EMAIL_TOOLS in both bare and mcp__email__* spellings. Scope: per review, the wider registry/advertising consolidation has been moved out to #4053 (reverted here in ee35896); this PR keeps only the parser fix, the bare-email routing, and the directly required safety gates. Final hardening from corpus replay: empty email fences (list_email_accounts with no body) now dispatch with empty args instead of vanishing, and non-object JSON args return a correctable error.

Target branch

  • This PR targets dev, not main.

Linked Issue

Part of #337 — fixes the email-in-chat failure reported there (UI test button works, but the model claims no email access). The calendar half of that issue needs retesting after this lands; the regex fix applies to every fenced tool with inline args (including manage_calendar), but that is unconfirmed.

Also related: #2447 (this fixes the fenced inline-args variant; the bare unfenced variant is still open), #1789 (possibly the same dispatch gap for user-added MCP servers), #3141 (closed; same symptom surface on Windows/Ollama).

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end. (Manual UAT on macOS with a local Ollama gemma model — results in the comment below. One failing scenario (chat-phrase email kill-switch) decomposes entirely into pre-existing defects outside this PR's files, tracked separately.)

How to Test

  1. Automated: python -m pytest tests/test_fenced_inline_args.py tests/test_review_regressions.py -q — covers inline-args parsing, the ```python3/hyphenated-tag non-matches, parse/strip display mirroring, and non-admin blocking of all 14 bare email names plus the mcp__email__... form.
  2. Dispatch check: start the app (uvicorn app:app), configure an email account (Settings → Email; confirm the test button lists your inbox).
  3. In chat/agent mode, select a local Ollama model (no native function calling — e.g. a gemma variant) and ask: "can you see my email accounts?"
  4. Before this PR: the reply is a raw ```list_email_accounts {} fence and no account data. After: the tool executes and the model answers with the configured account(s).
  5. Repeat with "list my latest emails" (list_emails) to confirm coverage beyond the accounts tool.
  6. Security check (multi-user deployments): as a non-admin user, ask the agent to delete/archive an email — every email tool must come back BLOCKED ("restricted to admin users"), bare or mcp__email__-qualified.

Visual / UI changes — REQUIRED if you touched anything that renders

N/A — no UI/rendering changes; only backend tool parsing/dispatch/policy (src/tool_parsing.py, src/tool_execution.py, src/tool_security.py, src/tool_schemas.py, src/agent_tools/__init__.py) and tests.


This PR was written via Claude and verified by a human.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the needs work PR description incomplete — please update before review label Jun 9, 2026
@github-actions github-actions Bot added ready for review Description complete — ready for maintainer review and removed needs work PR description incomplete — please update before review labels Jun 9, 2026
@botinate
botinate force-pushed the fix/fenced-tool-inline-args-email-dispatch branch 2 times, most recently from d3ca43c to 25dc723 Compare June 9, 2026 19:26
@vgalin

vgalin commented Jun 10, 2026

Copy link
Copy Markdown

This PR may also needs to update src/tool_security.py.

On current dev, execute_tool_block() checks is_public_blocked_tool(tool) before dispatch. Routing bare email names makes additional raw tool names executable, but NON_ADMIN_BLOCKED_TOOLS currently blocks only some of them (send_email, reply_to_email, list_emails, read_email). Names like list_email_accounts, bulk_email, archive_email, delete_email, mark_email_read, and download_attachment should probably be added too, matching SECURITY.md’s guidance that email/MCP capabilities are privileged admin functionality.

A regression test could extend test_public_agent_policy_blocks_sensitive_tools to cover those bare names as well as mcp__email__....

botinate and others added 2 commits June 11, 2026 01:27
… tool names

Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:

1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
   tool call with args on the same line ("```list_email_accounts {}")
   never matched and was never executed. The fence now matches with
   optional spaces/newline after the tag.

2. Even when parsed, bare email tool names had no dispatch branch in
   tool_execution.py and fell through to "Unknown tool type". They now
   route to the email MCP server as mcp__email__<name>, matching how
   function_call_to_tool_block already maps them for native callers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… fence-tag regex

Review follow-up on odysseus-dev#3681 (thanks @vgalin):

1. Routing bare email names made 10 of the 14 email tools executable by
   non-admin owners — is_public_blocked_tool() runs on the bare name
   before dispatch, and NON_ADMIN_BLOCKED_TOOLS only listed 4. Define the
   full email tool set once (BUILTIN_EMAIL_TOOLS in tool_security.py) and
   derive the blocklist, the fence tags (TOOL_TAGS), the bare-name
   dispatch, and the native-call mapping from it so they can't drift.
   This also fixes 4 tools (search_emails, draft_email, draft_email_reply,
   ai_draft_email_reply) that were missing from the old tool_schemas copy
   and therefore unreachable even for native function-calling models.

2. The relaxed fence regex from the previous commit could prefix-match
   longer fence tags: ```python3 parsed as tool "python" with content
   "3\nprint(...)" and executed as code. Add a (?![\w-]) boundary after
   the tag.

Tests: test_public_agent_policy_blocks_sensitive_tools now covers all 14
bare email names + the mcp__email__ form; new tests/test_fenced_inline_args.py
pins inline-args parsing, the python3/hyphenated-tag non-matches, and
strip/parse display mirroring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@botinate
botinate force-pushed the fix/fenced-tool-inline-args-email-dispatch branch from 25dc723 to 544a664 Compare June 11, 2026 08:32
@botinate

Copy link
Copy Markdown
Contributor Author

Good catch — confirmed and fixed in 544a664, plus a bit more than you listed.

The gap was real: execute_tool_block() runs is_public_blocked_tool() on the bare name before my dispatch branch qualifies it to mcp__email__<name>, so the mcp__ prefix rule never applied. And it was slightly wider than your list — search_emails, draft_email, draft_email_reply, and ai_draft_email_reply were also routable but unblocked, so 10 of the 14 email tools were reachable by non-admins.

Rather than appending names to NON_ADMIN_BLOCKED_TOOLS (a fourth hand-maintained copy of the email tool list), I defined the set once — BUILTIN_EMAIL_TOOLS in src/tool_security.py — and derived the blocklist, the fence tags (TOOL_TAGS), the bare-name dispatch, and the native-call mapping from it. A tool added to the email server now can't become routable without also being blocked for non-admins. Side effect: the four names missing from the old tool_schemas.py copy now work for native function-calling models too.

Your test suggestion is in: test_public_agent_policy_blocks_sensitive_tools now iterates all 14 bare names (spelled out rather than imported, so accidentally dropping one from the constant fails the test) plus the mcp__email__... form.

While re-reviewing the parsing change I also found and fixed a false-positive risk it introduced: the relaxed fence regex could prefix-match longer tags, so ```python3 would have parsed as a python tool call with content 3\nprint(...) and executed as code. There's now a (?![\w-]) boundary after the tag, with regression tests in tests/test_fenced_inline_args.py.

This comment was written via Claude and verified by a human.

@RaresKeY

Copy link
Copy Markdown
Collaborator

Findings

P1 Badge issue (tool-policy): Bare email aliases bypass qualified disabled-tool blocks

  • Problem: execute_tool_block() checks disabled_tools only against the incoming block name before dispatch. The new bare-email branch then maps that block to mcp__email__{tool} and calls the MCP manager without checking the qualified name. The email settings toggle and plan-mode MCP runtime blocks use qualified mcp__... names, so a bare block like list_emails or download_attachment can still reach the email MCP server.

  • Impact: This breaks the hard runtime block for sensitive email tools. Users who turn off email access can still have bare aliases execute, and plan-mode MCP blocks can be bypassed for MCP-only email tools.

  • Ask: Normalize bare email names to their qualified MCP names before applying runtime blocks, or check both bare and qualified names before calling mcp.call_tool(). Please add a regression where a bare email block is rejected when disabled_tools contains the qualified MCP name.

  • Location: src/tool_execution.py:770

P2 Badge issue (security): Do not execute Markdown fence metadata as tool input

  • Problem: The relaxed fenced-tool regex now accepts arbitrary same-line text after a recognized tag. That fixes inline tool args, but it also makes ordinary Markdown info strings executable on the local/fenced-tool path, such as python title="example.py" or bash title="setup".

  • Impact: Local agent turns can execute code or other privileged tools from code fences the model meant to display.

  • Ask: Keep inline-args support narrow enough that Markdown fence metadata is left visible/non-executable. Add regression tests for python title="..." and bash title="...".

  • Location: src/tool_parsing.py:28

Validation

Ran focused parser/tool-policy tests, git diff --check, isolated disabled-tool probes, and parser false-positive probes. Full suite and live email integration were not run.

botinate and others added 2 commits June 11, 2026 19:36
… executing Markdown info strings

Review follow-up on odysseus-dev#3681 (thanks @RaresKeY):

1. P1: execute_tool_block() checked disabled_tools / the turn ToolPolicy
   only against the incoming block name, then the bare-email branch
   qualified it to mcp__email__<name> and called the MCP manager. Plan
   mode and the MCP settings toggle write the QUALIFIED name into the
   denylist, so a bare fence like ```list_emails``` sailed past a
   mcp__email__list_emails entry. Both gates now match on both
   spellings (bare <-> mcp__email__-qualified), in either direction.

2. P2: the relaxed fence regex accepted arbitrary same-line text after
   a recognized tag, which made ordinary Markdown info strings
   executable: ```python title="example.py" ran as a python tool call.
   Same-line content now only counts as tool input when it starts with
   { or [ (JSON args); anything else leaves the fence as display text,
   and strip_tool_blocks mirrors that (the fence stays visible).

Tests: disabled-tools alias regression (qualified entry blocks bare
name and vice versa, never reaching the MCP manager), ToolPolicy alias
regression, python/bash title="..." non-execution + display retention,
and inline JSON-array args still parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@botinate

Copy link
Copy Markdown
Contributor Author

Both confirmed and fixed in 9076be3; the branch is also merged up to current dev.

P1 (disabled-tools bypass): exactly as described — execute_tool_block() gated on the incoming block name, and the bare-email branch qualified it to mcp__email__<name> only afterwards, so qualified denylist entries (plan mode's runtime MCP blocks, the settings toggle) never matched a bare fence. I also checked there's no enforcement deeper down: MCPManager.call_tool() doesn't consult disabled tools, and the loop dropping mcp_mgr only hides schemas. The fix computes the alias pair once at the top of the dispatcher ({bare, mcp__email__<bare>}) and both the disabled_tools gate and the turn ToolPolicy gate now match either spelling — in both directions, so a bare denylist entry also blocks the qualified form. The regression tests assert blocked calls never reach the MCP manager.

P2 (fence metadata): the same-line allowance is narrowed to JSON args only — after the tag, the rest of the opening line must be empty or start with {/[. Fences like python title="example.py" and bash title="setup" no longer parse, and since strip mirrors parse they stay visible as display text instead of vanishing. The inline-args forms this PR exists for ({}, {"max_results": 5}, arrays) keep working, with tests in both directions.

Full suite after the dev merge: 3265 passed, 1 skipped.

This comment was written via Claude and verified by a human.

@botinate

Copy link
Copy Markdown
Contributor Author

One field note from human testing of this PR build (gemma4:26b-a4b-it via Ollama, macOS), for anyone following along: now that fenced email calls actually execute, two pre-existing behaviours became visible that are not caused by this PR — the parsing/dispatch/persistence layers all behave correctly — but its testing exposed them:

  1. The model answers before the tool result. gemma often emits a full prose answer in the same round as its tool call — including entirely fabricated results (e.g. a made-up 10-row email table) — then answers again correctly in the next round once the real output is fed back. Both round texts are persisted by design, so the user sees two contradictory answers. This looks like an agent-prompt fix ("when you call a tool, output only the call — never answer before the result arrives"), not a parser change.

  2. The streaming view keeps the raw fence visible after execution. The persisted round_texts are stripped correctly, and reloading the session renders the clean layout (tool chip + text). But the live streamed bubble is never re-rendered from the cleaned text when the turn completes, so the executed fence lingers as a code box until reload. Frontend-only gap in the stream-finish path.

Keeping these here for now since this PR's testing surfaced them — happy to split them into their own issues if maintainers prefer.

This comment was written via Claude and verified by a human.

@RaresKeY

Copy link
Copy Markdown
Collaborator

I rechecked the latest head after the follow-up fixes. The earlier disabled-tool alias and simple metadata cases are covered now, but two gaps remain in the narrowed parser/policy surface.

Findings

P2 Badge issue (security): Brace-style fence metadata still executes as tool input

  • Problem: The new fenced-tool regex still treats any same-line { or [ after a recognized tag as executable tool content. On the current branch, a display fence like bash {title="setup"} parses as a bash block with {title="setup"}\necho hi, and strip_tool_blocks() removes the fence from display. python {"title":"example.py"} has the same shape.

  • Impact: Local/fenced-tool models can still run ordinary Markdown code examples that use brace/bracket fence attributes, which keeps part of the accidental fence-execution class from Agent mode executes ordinary Markdown code fences as tool calls, causing accidental commands and loops (again) #3604 open.

  • Ask: Parse the fence header separately and only treat same-line content as inline tool args when it is a valid standalone JSON argument for tools that actually use JSON args; keep bash/python and other Markdown metadata fences visible and non-executable. Please add brace/bracket metadata regressions for parse and strip symmetry.

  • Location: src/tool_parsing.py:32

P2 Badge issue (tool-policy): The email category toggle leaves most bare email aliases enabled

  • Problem: disable_tool email still expands to only mcp__email__list_emails, mcp__email__read_email, and mcp__email__send_email. This PR now routes all 14 built-in email tools by bare fenced names, and the execution gate only blocks aliases whose bare or qualified spelling is actually present in disabled_tools. With the current category toggle's disabled set, reply_to_email, delete_email, draft_email, download_attachment, and search_emails still reach the email MCP server.

  • Impact: An operator who disables email tools can still have sensitive read/mutate/send-adjacent email tools execute through the new bare aliases.

  • Ask: Derive the friendly email toggle from the shared built-in email tool set and cover the full set consistently, preferably including both bare and mcp__email__... spellings so schema hiding and the runtime backstop agree. Please add a regression for disable_tool email covering the full built-in email set.

  • Location: src/tool_implementations.py:1100

Validation

Ran focused parser/tool-policy tests, chat-route tool-policy tests, adjacent fenced-parser tests, git diff --check, isolated parser probes, and disabled-tool probes. Full suite and live email integration were not run.

PR Hygiene

botinate and others added 2 commits June 11, 2026 21:54
…l set in the friendly toggle

Review follow-up round 3 on odysseus-dev#3681 (thanks @RaresKeY):

1. Brace-style fence metadata no longer executes. The previous narrowing
   still treated any same-line {/[ after a recognized tag as tool input,
   so ```bash {title="setup"} ran as a bash call. The fence header is now
   captured separately and judged by one predicate shared between
   parse_tool_blocks and strip_tool_blocks (_fenced_tool_call), so the
   execute and display decisions can't disagree: same-line content only
   counts as inline args when the tag is NOT a code tag (bash/python
   never take same-line args — that text is Markdown fence attributes)
   AND the inline text (plus any continuation lines) parses as standalone
   JSON. ```bash {title="setup"}, ```python {"title":"example.py"} and
   ```list_emails {title="x"} all stay visible and inert.

2. The friendly `disable_tool email` toggle covered 3 of the 14 email
   tools (mcp__email__{list_emails,read_email,send_email}); the other
   bare aliases this PR routes stayed executable after an operator
   disabled email. The alias now derives from BUILTIN_EMAIL_TOOLS in
   BOTH spellings — bare (function-schema hiding, bare-fence dispatch)
   and mcp__email__* (MCP schema hiding, qualified runtime blocks) —
   so the toggle and the runtime gate can't drift apart.

Tests: brace/bracket metadata regressions for parse and strip symmetry
(code tags, invalid-JSON inline on a JSON tool, multi-line inline JSON
still parsing), and disable_tool/enable_tool email covering all 14 names
in both spellings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@RaresKeY

RaresKeY commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Re: botinate's field note on PR #3681: https://github.com/pewdiepie-archdaemon/odysseus/pull/3681#issuecomment-4683549687

That is the comment calling out the two pre-existing behaviors:

The model answers before the tool result.
The streaming view keeps the raw fence visible after execution.

I rechecked both against latest dev (c0cc0f9) with source inspection, a fake two-round stream_agent_loop probe, and a focused frontend strip-regex probe. I agree these are pre-existing follow-ups exposed by this PR's testing, not parser/dispatch regressions from the branch.

Confirmed and split out:

So this PR can stay scoped to the inline-args and bare-email routing change.

botinate and others added 2 commits June 11, 2026 22:19
… email tool for plan mode

Deep self-review follow-up on odysseus-dev#3681. Three review rounds each found another
hand-maintained copy of the email tool list that had drifted; this commit
hunts down ALL remaining copies and pins them to BUILTIN_EMAIL_TOOLS.

The same 5 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply, download_attachment) were missing from every
advertising surface, so they were dispatchable but never offered:

- FUNCTION_TOOL_SCHEMAS: native function-calling models never saw them
  (the round-1 fix covered dispatch only); schemas added, mirroring the
  email server's inputSchema definitions.
- TOOL_SECTIONS: fenced-block models were never told about them; prompt
  sections added.
- tool_index: absent from the RAG embedding registry (never retrievable),
  the email keyword hints, and the scheduled assistant's always-available
  set — the latter two now derive from BUILTIN_EMAIL_TOOLS.
- agent_loop._DOMAIN_TOOL_MAP["email"], tool_policy._COMMON_TOOL_NAMES,
  the assistant tool-selector UI groups (assistant.js), and the default
  Assistant crew seed (task_scheduler) now derive from / cover the set.

Plan mode now classifies every email tool explicitly:

- list_email_accounts and search_emails join PLAN_MODE_READONLY_TOOLS.
  Without this, list_email_accounts sat in the plan-mode bare denylist
  (schema-derived) while its qualified form passed the MCP read-only
  filter — and the round-2 bare/qualified alias gate would have blocked
  the qualified call too, regressing read-only email discovery in plan
  mode.
- draft_email, draft_email_reply, ai_draft_email_reply, and
  download_attachment join the fail-closed mutator backstop (drafts
  create documents; download_attachment writes to disk).

Tests: tests/test_email_registry_sync.py pins every registry (including
the email server source and assistant.js) to BUILTIN_EMAIL_TOOLS and
asserts the plan-mode partition, so the next email tool can't drift; a
parse/strip mirror grid covers 192 fence shapes (tag x header x body)
asserting executed <=> stripped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ssistant seed constant

Code-quality pass over the PR's own changes:

- The bare<->qualified email aliasing rule lived inline in the generic
  dispatcher (_execute_tool_block_impl). It is policy knowledge, so it
  moves next to BUILTIN_EMAIL_TOOLS as email_tool_policy_names(); the
  dispatcher just consumes it, and the rule gets its own unit test
  (including the mcp__email__<not-a-tool> and mcp__other__ non-alias
  cases).

- The default Assistant's enabled_tools list was an inline literal
  inside the CrewMember seed, and its registry-sync test asserted a
  source-code substring. Extracted to DEFAULT_ASSISTANT_ENABLED_TOOLS
  so the test imports and checks the actual value.

- _fenced_tool_call return type tightened to Optional[Tuple[str, str]].

No behavior change; suite green (3295 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@botinate

Copy link
Copy Markdown
Contributor Author

Both round-3 findings are fixed, and a deeper self-audit closed the rest of the drift class. Branch is merged up to current dev (includes #3281).

Brace-style metadata (b3d43ad): the fence header is captured separately and judged by one predicate shared between parse_tool_blocks and strip_tool_blocks, so execute/display decisions can't disagree by construction. Same-line content only counts as inline args when (a) the tag is not a code tag — bash/python never take same-line args — and (b) the inline text (plus continuation lines) parses as standalone JSON. bash {title="setup"}, python {"title":"example.py"}, and list_emails {title="x"} stay visible and inert; {}, {"max_results": 5}, and array args keep working. A grid regression asserts the executed⇔stripped invariant across 192 fence shapes.

Email category toggle (b3d43ad): disable_tool email derives from BUILTIN_EMAIL_TOOLS in both spellings (14 bare + 14 mcp__email__*), so schema hiding and the runtime backstop agree; enable_tool email removes the full set. Regression iterates all 14 names in both spellings.

Drift audit (db29046, 016ce47): since each round found another hand-maintained email list, I audited the whole tree for copies. Seven more surfaces were missing the same 5 tools (search_emails, draft_email, draft_email_reply, ai_draft_email_reply, download_attachment): native function schemas, the fenced-prompt tool sections, the RAG description index and email keyword hints, the assistant always-available set, the guide-only known-names set, the frontend tool-selector groups, and the default assistant seed — meaning those tools were dispatchable but never advertised. All now derive from / are pinned to BUILTIN_EMAIL_TOOLS; the bare↔qualified alias rule is a tested helper (email_tool_policy_names); and plan mode classifies every email tool explicitly (list_email_accounts/search_emails join the read-only allowlist — without that, the alias gate would have blocked their qualified read-only calls in plan mode; the draft tools and download_attachment join the fail-closed mutator backstop). tests/test_email_registry_sync.py pins all of it — including the email server source and assistant.js — so the next email tool can't drift silently.

Validation: full suite 3294+ passed. Additionally, I replayed a corpus of real captured gemma/Ollama chat traffic through the parser: every legitimate fenced call from the previous revision still executes (including tag{ with no space and multi-line JSON opened on the tag line), two garbage-arg fences that used to execute with mangled args are now correctly display-only, and the 18 inline-args calls this PR exists for all execute. A human will do additional manual end-to-end testing tomorrow — the "ran the app" checklist box stays honest until then.

This comment was written via Claude and verified by a human.

@RaresKeY

Copy link
Copy Markdown
Collaborator

Findings

P2 Badge issue (email): Draft email tools skip saved writing-style context

  • Problem: _EMAIL_TOOL_HINTS is still the old hand-maintained email list. This PR exposes draft_email, draft_email_reply, and ai_draft_email_reply, but when tool selection returns only one of those new draft tools, _build_system_prompt() does not enter the email-context branch. That skips the saved email writing style plus the hard identity/mechanical email rules. I reproduced that with a saved style sentinel: send_email and list_emails include it, while draft_email and ai_draft_email_reply do not.

  • Impact: The new preferred draft path can generate emails that ignore the user's configured writing style/signature and the identity guardrails, even though the older send/read paths still get that context.

  • Ask: Derive _EMAIL_TOOL_HINTS from BUILTIN_EMAIL_TOOLS plus their mcp__email__... aliases, keeping the existing contact/UI helpers, and add a regression where a draft-only selected tool receives the saved writing-style context.

  • Location: src/agent_loop.py:1049

Validation

  • Ran focused parser/email/tool-policy regression tests, Python compile checks for the changed backend modules, and node --check static/js/assistant.js.
  • Not run: full local suite or live model/email-provider flow.

@RaresKeY

Copy link
Copy Markdown
Collaborator

One reviewer/triage scope note before this grows further.

The original #337 symptom is broad, but the narrow fix here looked like: local/fenced models need inline-arg parsing and bare email tool dispatch so configured email tools actually execute. Some follow-up broadening was necessary once bare email names became executable, especially the non-admin blocklist, disabled-tool alias checks, and fence metadata hardening.

The later registry “drift audit” is useful cleanup, but it moves this PR into a broader email tool registry/policy/prompt/UI consolidation. That may be worth doing, but it changes the review and validation surface.

Can we pick one framing?

Either way, I would avoid closing #337 solely from this PR until the email path has a real app retest and the calendar part has a clear disposition.

@botinate

Copy link
Copy Markdown
Contributor Author

Two more caught during self-audits and fixed (branch now at c1b1a3d, suite 3304):

  • cd2463b — round-6's manage_memory decode was dead code: _build_mcp_args only runs for _MCP_TOOL_MAP tools, and manage_memory routes through dispatch_ai_tool → do_manage_memory instead, so the decode never executed (the unit test passed on the dead path). Removed it and added test_mcp_json_primary_keys_are_all_live, asserting every JSON-primary tool is in _MCP_TOOL_MAP.
  • c1b1a3d — same class, sharper: there's no filesystem MCP server, so write_file always runs via _direct_fallback → WriteFileTool, never through _build_mcp_args. WriteFileTool — alone among its siblings (ReadFileTool/WebSearchTool/WebFetchTool all decode JSON) — took lines[0] as the path, so write_file {"path": "...", "content": "..."} wrote to a file literally named with the JSON. Fixed in the handler (matching ReadFileTool), with a regression that drives the live path (execute_tool_block, no MCP) and is verified to fail without the fix. web_search/web_fetch/read_file were already correct via their handlers; write_file was the gap.

Beyond those, the manual UAT and the audits turned up four pre-existing items not in this PR's core change. Rather than open issues on my own read, I'd value your call on whether any are worth filing — happy to write clean repros + tracebacks and/or send fixes:

  1. Positional dispatchers mishandle JSON content. Tools that parse fenced content positionally don't decode a JSON object. Verified at the dispatch boundary: search_chats {"query": "X"} passes do_search_chats the literal string {"query": "X"} rather than X (searches the literal JSON, no error). The document tools take a JSON object as title/body (visible junk artifact); manage_memory/create_session/ui_control/send_to_session instead return correctable errors. Pre-existing — dev mishandles their newline-JSON form the same way — and this PR's relaxed parser only adds the same-line trigger. The clean general fix would route fenced content through function_call_to_tool_block.
  2. do_ui_control 500 on empty content — empty ui_control hits parts[0] on "".split() at ai_interaction.py:1304IndexError → agent-run 500.
  3. streamingTTS ReferenceError in the stream-error handler — declared inside the streaming try in static/js/chat.js (~1048), referenced in its catch (~2858), so the handler throws on any stream error and the status indicator sticks.
  4. "disable my email tools" routes to ui_control (which has no email toggle) instead of manage_settings disable_tool email, so the chat-phrase kill-switch never engages.

Which, if any, would you like as issues?

This comment was written via Claude and verified by a human.

@RaresKeY

Copy link
Copy Markdown
Collaborator

While live-testing the model-driven email path here, I found a separate current-dev blocker: configured AUTH_ENABLED=false instances treat the agent request owner as non-admin, so email/MCP/local tool schemas are stripped before the model sees them. Direct email MCP worked, and Gemini can emit the list_emails tool call when the schema is provided, but the Odysseus agent path only sent ui_control, ask_user, and update_plan.

I split that into a narrow stabilization fix: #4195. It is not caused by this PR's fenced/native dispatch changes, but it affects live model-driven email validation for this PR.

I also retested this PR on a branch stacked over the separate auth-disabled tool-exposure fix. On that stack:

  • focused parser/email/security regression set: 63 passed
  • full pytest suite: 3305 passed, 1 skipped
  • Docker app started cleanly with AUTH_ENABLED=false
  • GreenMail direct MCP list_emails returned the seeded message
  • Gemini 2.5 Pro through /api/chat_stream emitted mcp__email__list_emails, the tool output contained the seeded inbox subject, and the final answer was that subject

Not run on the stacked branch: browser/UI E2E, mutating email actions, GitHub CI, and a direct retest of unstacked #3681 after the separate fix lands.

@RaresKeY

RaresKeY commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

I rechecked the current state after #4195 landed. #4195 is merged into dev now, so the auth-disabled tool-exposure blocker from the last validation pass is no longer an external open dependency for this PR.

The remaining blocker is that GitHub now marks #3681 stale/conflicting with current dev. Replaying the merge against current dev conflicts in src/tool_schemas.py and tests/test_review_regressions.py. The important bit during the rebase is to preserve the newer dev behavior in the conflict resolution:

  • keep the current fail-closed handling for non-object email native-function arguments, but wire it to the shared email tool set from this PR;
  • keep the newer MCP owner-propagation and auth-disabled regression tests, alongside this PR's parser/policy tests.

After that, can you rerun the unstacked validation on the rebased branch? The useful minimum would be the focused parser/email/security tests, full pytest, and the live email-agent path now that #4195 is in dev.

For your last list of follow-ups: streamingTTS already has #3603. The empty ui_control 500 and the "disable my email tools" phrase routing still look like separate issues, and the positional JSON dispatcher item only partly overlaps #4088. I would file/fix those separately unless maintainers want them folded into another tracker.

…-args-email-dispatch

# Conflicts:
#	src/tool_schemas.py
#	tests/test_review_regressions.py
@botinate

Copy link
Copy Markdown
Contributor Author

Rebased onto current dev (2620f6c). Conflicts resolved as you suggested: the fail-closed non-object email-arg handling now uses the shared BUILTIN_EMAIL_TOOLS set (dropped the stale local copy), and I kept your MCP owner-propagation + auth-disabled regression tests alongside the parser/policy ones. Focused parser/email/security set: 65 passed; full suite: 3870 passed, 2 skipped. The live email-agent retest on the rebased branch is still pending — I'll run that next before this is merge-ready.

This comment was written via Claude and verified by a human.

@RaresKeY

Copy link
Copy Markdown
Collaborator

Rechecked latest head 2620f6cd: it still conflicts with current dev, now in src/tool_implementations.py after manage_settings moved to src/agent_tools/admin_tools.py.

Please rebase, port the full disable_tool email coverage into admin_tools.py, preserve current search/web toggle behavior, then rerun focused parser/email/security tests, full pytest, and the live unstacked email-agent smoke.

botinate and others added 2 commits June 27, 2026 11:50
Resolve the conflict from the upstream tool_implementations.py ->
src/agent_tools/admin_tools.py move of manage_settings (odysseus-dev#4423/odysseus-dev#3629):

- src/tool_implementations.py: take upstream's facade. This branch's only
  change to that file was the manage_settings "email" tool-category
  expansion, which moved with do_manage_settings to admin_tools.py.
- src/agent_tools/admin_tools.py: port the full `disable_tool email`
  coverage there, deriving the set from BUILTIN_EMAIL_TOOLS in both
  spellings (bare + mcp__email__*) instead of the hand-picked 3-tool
  subset. The search/web/research toggles keep current dev behavior
  (["web_search", "web_fetch"]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Semantic conflict from the dev merge that textual auto-merge didn't flag:
dev added test_live_strip_email_tool_fences.py whose _tool_tags() helper
source-scrapes only the TOOL_TAGS literal `{...}`, which worked on dev
because the email tool names were listed inline there. This branch makes
TOOL_TAGS the single source — `{...} | BUILTIN_EMAIL_TOOLS` — so the email
names are no longer in the literal and the scraper missed them, leaving the
email-fence strip assertions failing even though TOOL_TAGS does contain them
at runtime.

Import the real TOOL_TAGS instead of scraping source, so the test mirrors
exactly what GET /api/tools serves (sorted(TOOL_TAGS)) and the live
EXEC_FENCE_RE derives from — robust to however the set is composed. The
source-level frontend/route guards in the same file are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@botinate

Copy link
Copy Markdown
Contributor Author

Brought the branch up to current dev (ebead80) and resolved the conflict (head now 40f1726); PR shows MERGEABLE again. As you called out, the only textual conflict was in src/tool_implementations.py from manage_settings moving to src/agent_tools/admin_tools.py (#4423/#3629):

  • tool_implementations.py: took dev's new facade. This branch's only change to that file was the manage_settings email tool-category expansion, which moved with do_manage_settings.
  • admin_tools.py: ported the full disable_tool email coverage there — "email" now derives from BUILTIN_EMAIL_TOOLS in both spellings (bare + mcp__email__*) instead of the hand-picked 3-tool subset. test_disable_tool_email_covers_full_builtin_set passes against the new location (re-exported through the facade, so the existing import path still resolves).
  • search/web/research toggles: left at current dev behavior (["web_search", "web_fetch"]) — only the email entry changed.

One semantic conflict that textual auto-merge did not flag, caught on the test run: dev added tests/test_live_strip_email_tool_fences.py, whose _tool_tags() helper source-scrapes only the TOOL_TAGS literal {...}. That worked on dev because the email names were inline there, but this branch makes TOOL_TAGS = {...} | BUILTIN_EMAIL_TOOLS the single source, so the names aren't in the literal anymore and the scraper missed them — the email-fence strip assertions failed even though TOOL_TAGS contains them at runtime. Fixed in 40f1726 by importing the real TOOL_TAGS (the exact set GET /api/tools serves and the live EXEC_FENCE_RE derives from) rather than scraping source; the frontend/route source-guards in that file are untouched.

Validation — focused parser/email/security/admin batch (22 files incl. test_email_registry_sync, test_review_regressions, test_admin_tools_registry, test_fenced_inline_args, test_plan_mode, test_live_strip_email_tool_fences, test_function_call_non_object_args, test_security_regressions, test_prompt_security): 309 passed. Caveat on full coverage: I couldn't run the whole suite in this sandbox — some unrelated route modules don't import here (the 6 model_routes discovery tests in test_review_regressions fail on a core.log_safety import that resolves fine standalone; they fail identically on a clean dev checkout, so they're environmental, not from this merge). Leaving the full pytest run to CI, and the live unstacked email-agent smoke is still pending — a human will run that before this is merge-ready.

This comment was written via Claude and verified by a human.

Resolve the strip_tool_blocks conflict from dev's tool-call-parser ReDoS
hardening (odysseus-dev#4941) landing on this branch's fenced-tool changes:

- Keep this branch's executed-vs-illustrative fence handling
  (_TOOL_BLOCK_RE.sub(_strip_executed_fence, ...)) so example fences from
  native function-calling models stay visible/inert while dispatched fences
  are stripped — the executed<->stripped invariant.
- Take dev's forward-only [TOOL_CALL]{...}[/TOOL_CALL] removal
  (_strip_delimited over _TOOL_CALL_OPEN_RE/_CLOSE_RE) in place of the old
  _TOOL_CALL_RE.sub(''), so untrusted output can't drive the O(n^2) lazy
  rescan. This mirrors the parse side, which already scans the same
  delimiters via _iter_delimited.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@botinate

Copy link
Copy Markdown
Contributor Author

dev moved again and re-conflicted — this time one hunk in src/tool_parsing.py, from dev's tool-call-parser ReDoS hardening (#4941) overlapping this branch's strip_tool_blocks changes. Brought the branch up to current dev (3021569); head now 215bb94, PR back to MERGEABLE.

The two edits were independent, so the resolution keeps both:

  • This branch: _TOOL_BLOCK_RE.sub(_strip_executed_fence, ...) for fenced ```tag blocks — only fences that actually dispatched get stripped; illustrative example fences from native function-calling models stay visible/inert (the executed⇔stripped invariant).
  • dev (fix(security): prevent ReDoS in XML and args tool-call parsers #4941): the [TOOL_CALL]{…}[/TOOL_CALL] removal switches from _TOOL_CALL_RE.sub('') to the forward-only _strip_delimited(_TOOL_CALL_OPEN_RE, _TOOL_CALL_CLOSE_RE), so untrusted output can't drive the O(n²) lazy rescan. This mirrors the parse side, which already scans the same delimiters via _iter_delimited.

Validation: parser/strip/ReDoS set — test_fenced_inline_args, test_fenced_example_not_executed_for_native_models, test_redos_llm_parsers, test_redos_xml_tool_parsers, test_tool_parsing_nonstring, test_live_strip_email_tool_fences and the rest — 146 passed, and the parse↔strip invariant tests (89) are green. The broader focused parser/email/security/admin batch still 309 passed (same 6 unrelated model_routes env failures deselected, fail identically on clean dev). Full pytest + live email-agent smoke remain for CI / a human.

This comment was written via Claude and verified by a human.

@RaresKeY

Copy link
Copy Markdown
Collaborator

Thanks for keeping this branch current through the rebases. I re-reviewed the latest head (215bb944) against current dev; the earlier parser/policy issues look addressed, but I found two remaining gaps in the new bare/inline email path that should be fixed before merge.

Findings

P2 Badge issue (email/owner): Bare email aliases drop the authenticated owner before calling MCP

  • Problem: The new bare-email branch calls mcp.call_tool(qualified, args) without adding the hidden _odysseus_owner argument, while the existing qualified mcp__email__... branch adds it before calling MCP. The email MCP server uses that hidden owner to filter visible accounts and to fail closed when multiple account owners are configured.

  • Impact: Local/fenced-model calls through the new bare aliases can run as ownerless calls. Multi-owner installs can reject them with the owner-required error even though the chat request has an owner, and single-owner/legacy layouts can use the default/global account instead of matching the authenticated owner behavior of the qualified MCP path.

  • Ask: Mirror the qualified email path before the bare branch calls MCP, for example by copying args and setting _odysseus_owner when owner is present. Please also add a regression for a bare alias such as list_emails with owner="alice" reaching the MCP manager with that hidden owner.

  • Location: src/tool_execution.py:910

P3 Badge issue (frontend): Live stripping misses executed inline-argument fences

  • Problem: The backend now executes same-line JSON fences such as an opener of list_email_accounts {}, but the live frontend stripper still builds EXEC_FENCE_RE with a required newline immediately after the tool tag.

  • Impact: The backend can execute and persistently strip the new inline-args shape, while the final live bubble still renders the raw executed fence until the session is reloaded. The existing live-strip tests cover classic newline body-form fences, but not this new inline form.

  • Ask: Update the live strip pattern to match the backend’s executable inline-args shape, and add a regression for a valid inline email fence such as list_email_accounts {}.

  • Location: static/js/chatRenderer.js:436

Validation

  • Visible GitHub checks: Passing at review time, with the Trivy SARIF upload job skipped.
  • Ran: Focused parser/email/security/policy test slices, git diff --check, owner-propagation probes, and live-strip regex probes.
  • Not run: Full local pytest on this prepared head, live app/browser/email/IMAP/SMTP validation, and live unstacked model-driven email-agent smoke.

@alteixeira20 alteixeira20 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Reviewed the PR body, full discussion, changed files, staged merge-result diff, and focused validation. This fixes the local/fenced-tool path for inline JSON arguments and bare email tool names, while preserving the required safety gates around email/MCP tools.

Validated:

  • PR discussion has no unresolved review threads, no review comments, and no formal blocking reviews.
  • Expected head guard passed.
  • Changed-file allowlist passed.
  • Merge probe is clean against current dev.
  • GitHub checks are green.
  • git diff --check is clean.
  • The 6 model/default-chat failures in tests/test_review_regressions.py reproduce on clean dev with the same core.log_safety import error, so they are not PR-caused.
  • Focused PR-relevant validation passed: py_compile plus pytest on email registry, fenced inline args, live strip email fences, and review regressions with the known unrelated model/default-chat failures deselected.

Source-level review notes:

  • parse_tool_blocks and strip_tool_blocks now share the same executed-vs-illustrative fence decision.
  • bash/python Markdown fence metadata remains visible and inert.
  • bare and mcp__email__ tool spellings are checked together by disabled-tool and ToolPolicy gates.
  • non-admin users are blocked from all built-in email tools.
  • plan mode explicitly classifies all built-in email tools.
  • disable_tool email covers all bare and qualified email tool names.
  • bare email calls reject non-object or invalid JSON args instead of silently falling back to default mailbox args.

@alteixeira20
alteixeira20 merged commit 69b9bb0 into odysseus-dev:dev Jun 30, 2026
14 checks passed
buluma pushed a commit to buluma/odysseus that referenced this pull request Jul 1, 2026
… email tool names (odysseus-dev#3681)

* fix(agent): execute fenced tool calls with inline args and bare email tool names

Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:

1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
   tool call with args on the same line ("```list_email_accounts {}")
   never matched and was never executed. The fence now matches with
   optional spaces/newline after the tag.

2. Even when parsed, bare email tool names had no dispatch branch in
   tool_execution.py and fell through to "Unknown tool type". They now
   route to the email MCP server as mcp__email__<name>, matching how
   function_call_to_tool_block already maps them for native callers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): block all bare email tool names for non-admins; harden fence-tag regex

Review follow-up on odysseus-dev#3681 (thanks @vgalin):

1. Routing bare email names made 10 of the 14 email tools executable by
   non-admin owners — is_public_blocked_tool() runs on the bare name
   before dispatch, and NON_ADMIN_BLOCKED_TOOLS only listed 4. Define the
   full email tool set once (BUILTIN_EMAIL_TOOLS in tool_security.py) and
   derive the blocklist, the fence tags (TOOL_TAGS), the bare-name
   dispatch, and the native-call mapping from it so they can't drift.
   This also fixes 4 tools (search_emails, draft_email, draft_email_reply,
   ai_draft_email_reply) that were missing from the old tool_schemas copy
   and therefore unreachable even for native function-calling models.

2. The relaxed fence regex from the previous commit could prefix-match
   longer fence tags: ```python3 parsed as tool "python" with content
   "3\nprint(...)" and executed as code. Add a (?![\w-]) boundary after
   the tag.

Tests: test_public_agent_policy_blocks_sensitive_tools now covers all 14
bare email names + the mcp__email__ form; new tests/test_fenced_inline_args.py
pins inline-args parsing, the python3/hyphenated-tag non-matches, and
strip/parse display mirroring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): gate bare and mcp-qualified email names together; stop executing Markdown info strings

Review follow-up on odysseus-dev#3681 (thanks @RaresKeY):

1. P1: execute_tool_block() checked disabled_tools / the turn ToolPolicy
   only against the incoming block name, then the bare-email branch
   qualified it to mcp__email__<name> and called the MCP manager. Plan
   mode and the MCP settings toggle write the QUALIFIED name into the
   denylist, so a bare fence like ```list_emails``` sailed past a
   mcp__email__list_emails entry. Both gates now match on both
   spellings (bare <-> mcp__email__-qualified), in either direction.

2. P2: the relaxed fence regex accepted arbitrary same-line text after
   a recognized tag, which made ordinary Markdown info strings
   executable: ```python title="example.py" ran as a python tool call.
   Same-line content now only counts as tool input when it starts with
   { or [ (JSON args); anything else leaves the fence as display text,
   and strip_tool_blocks mirrors that (the fence stays visible).

Tests: disabled-tools alias regression (qualified entry blocks bare
name and vice versa, never reaching the MCP manager), ToolPolicy alias
regression, python/bash title="..." non-execution + display retention,
and inline JSON-array args still parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): reject brace-style fence metadata; cover the full email set in the friendly toggle

Review follow-up round 3 on odysseus-dev#3681 (thanks @RaresKeY):

1. Brace-style fence metadata no longer executes. The previous narrowing
   still treated any same-line {/[ after a recognized tag as tool input,
   so ```bash {title="setup"} ran as a bash call. The fence header is now
   captured separately and judged by one predicate shared between
   parse_tool_blocks and strip_tool_blocks (_fenced_tool_call), so the
   execute and display decisions can't disagree: same-line content only
   counts as inline args when the tag is NOT a code tag (bash/python
   never take same-line args — that text is Markdown fence attributes)
   AND the inline text (plus any continuation lines) parses as standalone
   JSON. ```bash {title="setup"}, ```python {"title":"example.py"} and
   ```list_emails {title="x"} all stay visible and inert.

2. The friendly `disable_tool email` toggle covered 3 of the 14 email
   tools (mcp__email__{list_emails,read_email,send_email}); the other
   bare aliases this PR routes stayed executable after an operator
   disabled email. The alias now derives from BUILTIN_EMAIL_TOOLS in
   BOTH spellings — bare (function-schema hiding, bare-fence dispatch)
   and mcp__email__* (MCP schema hiding, qualified runtime blocks) —
   so the toggle and the runtime gate can't drift apart.

Tests: brace/bracket metadata regressions for parse and strip symmetry
(code tags, invalid-JSON inline on a JSON tool, multi-line inline JSON
still parsing), and disable_tool/enable_tool email covering all 14 names
in both spellings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(email): close remaining email-tool registry drift; classify every email tool for plan mode

Deep self-review follow-up on odysseus-dev#3681. Three review rounds each found another
hand-maintained copy of the email tool list that had drifted; this commit
hunts down ALL remaining copies and pins them to BUILTIN_EMAIL_TOOLS.

The same 5 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply, download_attachment) were missing from every
advertising surface, so they were dispatchable but never offered:

- FUNCTION_TOOL_SCHEMAS: native function-calling models never saw them
  (the round-1 fix covered dispatch only); schemas added, mirroring the
  email server's inputSchema definitions.
- TOOL_SECTIONS: fenced-block models were never told about them; prompt
  sections added.
- tool_index: absent from the RAG embedding registry (never retrievable),
  the email keyword hints, and the scheduled assistant's always-available
  set — the latter two now derive from BUILTIN_EMAIL_TOOLS.
- agent_loop._DOMAIN_TOOL_MAP["email"], tool_policy._COMMON_TOOL_NAMES,
  the assistant tool-selector UI groups (assistant.js), and the default
  Assistant crew seed (task_scheduler) now derive from / cover the set.

Plan mode now classifies every email tool explicitly:

- list_email_accounts and search_emails join PLAN_MODE_READONLY_TOOLS.
  Without this, list_email_accounts sat in the plan-mode bare denylist
  (schema-derived) while its qualified form passed the MCP read-only
  filter — and the round-2 bare/qualified alias gate would have blocked
  the qualified call too, regressing read-only email discovery in plan
  mode.
- draft_email, draft_email_reply, ai_draft_email_reply, and
  download_attachment join the fail-closed mutator backstop (drafts
  create documents; download_attachment writes to disk).

Tests: tests/test_email_registry_sync.py pins every registry (including
the email server source and assistant.js) to BUILTIN_EMAIL_TOOLS and
asserts the plan-mode partition, so the next email tool can't drift; a
parse/strip mirror grid covers 192 fence shapes (tag x header x body)
asserting executed <=> stripped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: move the email alias rule into tool_security; extract the assistant seed constant

Code-quality pass over the PR's own changes:

- The bare<->qualified email aliasing rule lived inline in the generic
  dispatcher (_execute_tool_block_impl). It is policy knowledge, so it
  moves next to BUILTIN_EMAIL_TOOLS as email_tool_policy_names(); the
  dispatcher just consumes it, and the rule gets its own unit test
  (including the mcp__email__<not-a-tool> and mcp__other__ non-alias
  cases).

- The default Assistant's enabled_tools list was an inline literal
  inside the CrewMember seed, and its registry-sync test asserted a
  source-code substring. Extracted to DEFAULT_ASSISTANT_ENABLED_TOOLS
  so the test imports and checks the actual value.

- _fenced_tool_call return type tightened to Optional[Tuple[str, str]].

No behavior change; suite green (3295 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* revert: move the email registry consolidation to a follow-up PR

Per review feedback on scope, this PR stays narrow: fenced inline-args
parsing, bare email tool routing, and the directly required safety
gates. This commit reverts the registry/advertising consolidation from
db29046 and 016ce47 (native schemas, prompt sections, RAG description
index + keyword hints, assistant always-available set, guide-only
known-names union, frontend tool-selector groups, default assistant
seed, and their sync tests) — all of that moves to a dedicated
follow-up PR together with the _EMAIL_TOOL_HINTS finding.

Kept here because the narrow scope needs them:
- email_tool_policy_names() in tool_security + its use in the
  execute_tool_block gates and its unit test (refactor of this PR's own
  round-2 alias fix),
- list_email_accounts in PLAN_MODE_READONLY_TOOLS (the alias gate works
  both ways, and the schema-derived plan-mode bare denylist would
  otherwise block the qualified read-only call too),
- the parse/strip mirror grid test (parser scope),
- the narrow registry sync tests (email server <-> BUILTIN_EMAIL_TOOLS
  match, fence-tag coverage, non-admin blocklist coverage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(email): execute empty email fences with empty args; reject non-object JSON args

Two gaps found by replaying captured local-model traffic against the
narrowed branch:

1. ```list_email_accounts``` with NO body — a shape gemma really emits
   for no-arg tools — was silently dropped (parse skips empty content),
   so the model concluded email was broken: the original odysseus-dev#337 symptom
   through a different door. Empty fences whose tag is a built-in email
   tool now dispatch with {} args and the tool's own validation answers
   (e.g. an empty send_email returns "to is required" instead of
   silence). Empty bash/python/other fences keep skipping, and strip
   stays mirrored (the fence was executed, so it is removed).

2. The fence parser accepts JSON arrays as inline args, but the email
   dispatch parsed only objects — an array silently became {} args.
   Non-object JSON now returns a correctable "arguments must be a JSON
   object" error before reaching the MCP server (same class as odysseus-dev#3966).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): classify all email tools for plan mode statically; reject invalid email JSON bodies

Review follow-up round 5 on odysseus-dev#3681 (thanks @RaresKeY):

1. This PR makes every BUILTIN_EMAIL_TOOLS name fence-taggable, so each
   one must be explicitly classified for plan mode — the draft tools and
   download_attachment were in neither the read-only allowlist nor the
   static denylist, leaving their bare-alias plan-mode safety dependent
   on the MCP read-only inventory being present and current.
   search_emails joins PLAN_MODE_READONLY_TOOLS (explicit, not
   allowed-by-omission); draft_email, draft_email_reply,
   ai_draft_email_reply, and download_attachment join the fail-closed
   _PLAN_MODE_KNOWN_MUTATORS backstop. (Moved back from the odysseus-dev#4053 split:
   the partition is directly required for this PR to merge
   independently.)

2. The classic tag/body fence form reaches execution unvalidated (only
   INLINE args are JSON-checked by the parser), so a body like
   {account: "work"} silently became {} args and read the DEFAULT
   mailbox instead of the intended one. JSON-looking bodies that fail to
   parse now return a correctable "not valid JSON" error before reaching
   the MCP server.

Tests: a partition invariant (every email tool is explicitly read-only
or plan-mode-denied), a mutating-alias probe that uses only the static
denylist with a fake MCP manager (no inventory layer), and the
body-form invalid-JSON regression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): decode inline JSON args for legacy MCP tools; reject all non-object email bodies

Review follow-up round 6 on odysseus-dev#3681 (thanks @RaresKeY) — both pre-existing
on this branch, surfaced by the relaxed inline-args parser:

1. The relaxed parser accepts inline JSON for every non-code tag, but
   the legacy line-based arg builders (web_search/web_fetch/read_file/
   write_file/generate_image/manage_memory) wrapped the whole JSON
   string as the query/url/path/prompt — so `web_search {"query": "x"}`
   executed as a search for the literal string `{"query": "x"}`.
   _build_mcp_args now uses a fenced JSON object directly when it carries
   the tool's primary arg key (query/url/path/prompt/action). Keyed off
   membership so it can't drift; an object without the primary key (e.g.
   a freeform JSON query, or bare object content for write_file) falls
   through to the line parser unchanged. Also fixes the same corruption
   for the classic newline-JSON form.

2. The bare-email dispatch only rejected bodies starting with { or [, so
   a non-empty non-JSON body like `account: work` still fell through to
   {} args and silently read the DEFAULT mailbox. Now ANY non-empty body
   must decode to a JSON object or it returns a correctable error; only a
   truly empty body keeps the no-arg path (```list_email_accounts```).

Tests: inline-JSON arg decoding for the five legacy tools plus the
freeform and missing-primary-key fallbacks; the email body rejection
extended to cover the brace-looking and bare `key: value` shapes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): drop dead manage_memory JSON-decode entry; pin the live-path invariant

Self-audit catch on the round-6 fix. manage_memory was added to
_MCP_JSON_PRIMARY_KEYS, but _build_mcp_args is only reached via
_call_mcp_tool, which only runs for _MCP_TOOL_MAP tools — and
manage_memory isn't one (its tag routes through dispatch_ai_tool ->
do_manage_memory, which line-parses). So the round-6 decode for
manage_memory was dead code: the unit test exercising _build_mcp_args
passed while a real `manage_memory {"action": ...}` fence still parsed
the whole JSON blob as the action.

Remove the dead entry and add test_mcp_json_primary_keys_are_all_live,
which asserts every JSON-primary tool is in _MCP_TOOL_MAP so a dead
decode can't be added again. The same inline-JSON corruption for
manage_memory and the other tools that route through positional
dispatchers (create_session, ui_control, send_to_session, search_chats,
the document tools, etc.) is pre-existing (dev corrupts their newline
JSON form too) and tracked separately; the proper fix there is to route
fenced JSON through function_call_to_tool_block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): decode inline JSON in WriteFileTool (its live path); round-6 fix was on the dead MCP path

Self-audit: round 6 claimed to fix inline JSON args for write_file via
_build_mcp_args, but there is no filesystem MCP server, so write_file
always runs through _direct_fallback -> WriteFileTool, never through
_build_mcp_args. WriteFileTool — unlike its siblings ReadFileTool /
WebSearchTool / WebFetchTool, which all decode JSON — took lines[0] as
the path, so `write_file {"path": "/tmp/x", "content": "y"}` wrote to a
file literally named with the JSON blob. The round-6 _build_mcp_args
entry decoded correctly but on a path that never executes (same class
as the manage_memory dead entry), and the round-6 unit test passed on
that dead path.

WriteFileTool now decodes a JSON object carrying "path" (matching
ReadFileTool directly above it), and the comment on _MCP_JSON_PRIMARY_KEYS
records that only generate_image has a live MCP server today — the other
entries are defense-in-depth for the MCP path; the live fix for each
server-less tool is in its handler.

Test: test_write_file_inline_json_args drives the LIVE path
(execute_tool_block with no MCP) and asserts the intended path is used —
verified to fail without the handler fix. web_search/web_fetch/read_file
were already correct (their handlers decode); write_file was the gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(strip-fence): derive the live-strip TOOL_TAGS from the real set

Semantic conflict from the dev merge that textual auto-merge didn't flag:
dev added test_live_strip_email_tool_fences.py whose _tool_tags() helper
source-scrapes only the TOOL_TAGS literal `{...}`, which worked on dev
because the email tool names were listed inline there. This branch makes
TOOL_TAGS the single source — `{...} | BUILTIN_EMAIL_TOOLS` — so the email
names are no longer in the literal and the scraper missed them, leaving the
email-fence strip assertions failing even though TOOL_TAGS does contain them
at runtime.

Import the real TOOL_TAGS instead of scraping source, so the test mirrors
exactly what GET /api/tools serves (sorted(TOOL_TAGS)) and the live
EXEC_FENCE_RE derives from — robust to however the set is composed. The
source-level frontend/route guards in the same file are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: botinate <285686135+botinate@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
entitycs pushed a commit to entitycs/odysseus that referenced this pull request Jul 5, 2026
… email tool names (odysseus-dev#3681)

* fix(agent): execute fenced tool calls with inline args and bare email tool names

Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:

1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
   tool call with args on the same line ("```list_email_accounts {}")
   never matched and was never executed. The fence now matches with
   optional spaces/newline after the tag.

2. Even when parsed, bare email tool names had no dispatch branch in
   tool_execution.py and fell through to "Unknown tool type". They now
   route to the email MCP server as mcp__email__<name>, matching how
   function_call_to_tool_block already maps them for native callers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): block all bare email tool names for non-admins; harden fence-tag regex

Review follow-up on odysseus-dev#3681 (thanks @vgalin):

1. Routing bare email names made 10 of the 14 email tools executable by
   non-admin owners — is_public_blocked_tool() runs on the bare name
   before dispatch, and NON_ADMIN_BLOCKED_TOOLS only listed 4. Define the
   full email tool set once (BUILTIN_EMAIL_TOOLS in tool_security.py) and
   derive the blocklist, the fence tags (TOOL_TAGS), the bare-name
   dispatch, and the native-call mapping from it so they can't drift.
   This also fixes 4 tools (search_emails, draft_email, draft_email_reply,
   ai_draft_email_reply) that were missing from the old tool_schemas copy
   and therefore unreachable even for native function-calling models.

2. The relaxed fence regex from the previous commit could prefix-match
   longer fence tags: ```python3 parsed as tool "python" with content
   "3\nprint(...)" and executed as code. Add a (?![\w-]) boundary after
   the tag.

Tests: test_public_agent_policy_blocks_sensitive_tools now covers all 14
bare email names + the mcp__email__ form; new tests/test_fenced_inline_args.py
pins inline-args parsing, the python3/hyphenated-tag non-matches, and
strip/parse display mirroring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): gate bare and mcp-qualified email names together; stop executing Markdown info strings

Review follow-up on odysseus-dev#3681 (thanks @RaresKeY):

1. P1: execute_tool_block() checked disabled_tools / the turn ToolPolicy
   only against the incoming block name, then the bare-email branch
   qualified it to mcp__email__<name> and called the MCP manager. Plan
   mode and the MCP settings toggle write the QUALIFIED name into the
   denylist, so a bare fence like ```list_emails``` sailed past a
   mcp__email__list_emails entry. Both gates now match on both
   spellings (bare <-> mcp__email__-qualified), in either direction.

2. P2: the relaxed fence regex accepted arbitrary same-line text after
   a recognized tag, which made ordinary Markdown info strings
   executable: ```python title="example.py" ran as a python tool call.
   Same-line content now only counts as tool input when it starts with
   { or [ (JSON args); anything else leaves the fence as display text,
   and strip_tool_blocks mirrors that (the fence stays visible).

Tests: disabled-tools alias regression (qualified entry blocks bare
name and vice versa, never reaching the MCP manager), ToolPolicy alias
regression, python/bash title="..." non-execution + display retention,
and inline JSON-array args still parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): reject brace-style fence metadata; cover the full email set in the friendly toggle

Review follow-up round 3 on odysseus-dev#3681 (thanks @RaresKeY):

1. Brace-style fence metadata no longer executes. The previous narrowing
   still treated any same-line {/[ after a recognized tag as tool input,
   so ```bash {title="setup"} ran as a bash call. The fence header is now
   captured separately and judged by one predicate shared between
   parse_tool_blocks and strip_tool_blocks (_fenced_tool_call), so the
   execute and display decisions can't disagree: same-line content only
   counts as inline args when the tag is NOT a code tag (bash/python
   never take same-line args — that text is Markdown fence attributes)
   AND the inline text (plus any continuation lines) parses as standalone
   JSON. ```bash {title="setup"}, ```python {"title":"example.py"} and
   ```list_emails {title="x"} all stay visible and inert.

2. The friendly `disable_tool email` toggle covered 3 of the 14 email
   tools (mcp__email__{list_emails,read_email,send_email}); the other
   bare aliases this PR routes stayed executable after an operator
   disabled email. The alias now derives from BUILTIN_EMAIL_TOOLS in
   BOTH spellings — bare (function-schema hiding, bare-fence dispatch)
   and mcp__email__* (MCP schema hiding, qualified runtime blocks) —
   so the toggle and the runtime gate can't drift apart.

Tests: brace/bracket metadata regressions for parse and strip symmetry
(code tags, invalid-JSON inline on a JSON tool, multi-line inline JSON
still parsing), and disable_tool/enable_tool email covering all 14 names
in both spellings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(email): close remaining email-tool registry drift; classify every email tool for plan mode

Deep self-review follow-up on odysseus-dev#3681. Three review rounds each found another
hand-maintained copy of the email tool list that had drifted; this commit
hunts down ALL remaining copies and pins them to BUILTIN_EMAIL_TOOLS.

The same 5 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply, download_attachment) were missing from every
advertising surface, so they were dispatchable but never offered:

- FUNCTION_TOOL_SCHEMAS: native function-calling models never saw them
  (the round-1 fix covered dispatch only); schemas added, mirroring the
  email server's inputSchema definitions.
- TOOL_SECTIONS: fenced-block models were never told about them; prompt
  sections added.
- tool_index: absent from the RAG embedding registry (never retrievable),
  the email keyword hints, and the scheduled assistant's always-available
  set — the latter two now derive from BUILTIN_EMAIL_TOOLS.
- agent_loop._DOMAIN_TOOL_MAP["email"], tool_policy._COMMON_TOOL_NAMES,
  the assistant tool-selector UI groups (assistant.js), and the default
  Assistant crew seed (task_scheduler) now derive from / cover the set.

Plan mode now classifies every email tool explicitly:

- list_email_accounts and search_emails join PLAN_MODE_READONLY_TOOLS.
  Without this, list_email_accounts sat in the plan-mode bare denylist
  (schema-derived) while its qualified form passed the MCP read-only
  filter — and the round-2 bare/qualified alias gate would have blocked
  the qualified call too, regressing read-only email discovery in plan
  mode.
- draft_email, draft_email_reply, ai_draft_email_reply, and
  download_attachment join the fail-closed mutator backstop (drafts
  create documents; download_attachment writes to disk).

Tests: tests/test_email_registry_sync.py pins every registry (including
the email server source and assistant.js) to BUILTIN_EMAIL_TOOLS and
asserts the plan-mode partition, so the next email tool can't drift; a
parse/strip mirror grid covers 192 fence shapes (tag x header x body)
asserting executed <=> stripped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: move the email alias rule into tool_security; extract the assistant seed constant

Code-quality pass over the PR's own changes:

- The bare<->qualified email aliasing rule lived inline in the generic
  dispatcher (_execute_tool_block_impl). It is policy knowledge, so it
  moves next to BUILTIN_EMAIL_TOOLS as email_tool_policy_names(); the
  dispatcher just consumes it, and the rule gets its own unit test
  (including the mcp__email__<not-a-tool> and mcp__other__ non-alias
  cases).

- The default Assistant's enabled_tools list was an inline literal
  inside the CrewMember seed, and its registry-sync test asserted a
  source-code substring. Extracted to DEFAULT_ASSISTANT_ENABLED_TOOLS
  so the test imports and checks the actual value.

- _fenced_tool_call return type tightened to Optional[Tuple[str, str]].

No behavior change; suite green (3295 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* revert: move the email registry consolidation to a follow-up PR

Per review feedback on scope, this PR stays narrow: fenced inline-args
parsing, bare email tool routing, and the directly required safety
gates. This commit reverts the registry/advertising consolidation from
db29046 and 016ce47 (native schemas, prompt sections, RAG description
index + keyword hints, assistant always-available set, guide-only
known-names union, frontend tool-selector groups, default assistant
seed, and their sync tests) — all of that moves to a dedicated
follow-up PR together with the _EMAIL_TOOL_HINTS finding.

Kept here because the narrow scope needs them:
- email_tool_policy_names() in tool_security + its use in the
  execute_tool_block gates and its unit test (refactor of this PR's own
  round-2 alias fix),
- list_email_accounts in PLAN_MODE_READONLY_TOOLS (the alias gate works
  both ways, and the schema-derived plan-mode bare denylist would
  otherwise block the qualified read-only call too),
- the parse/strip mirror grid test (parser scope),
- the narrow registry sync tests (email server <-> BUILTIN_EMAIL_TOOLS
  match, fence-tag coverage, non-admin blocklist coverage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(email): execute empty email fences with empty args; reject non-object JSON args

Two gaps found by replaying captured local-model traffic against the
narrowed branch:

1. ```list_email_accounts``` with NO body — a shape gemma really emits
   for no-arg tools — was silently dropped (parse skips empty content),
   so the model concluded email was broken: the original odysseus-dev#337 symptom
   through a different door. Empty fences whose tag is a built-in email
   tool now dispatch with {} args and the tool's own validation answers
   (e.g. an empty send_email returns "to is required" instead of
   silence). Empty bash/python/other fences keep skipping, and strip
   stays mirrored (the fence was executed, so it is removed).

2. The fence parser accepts JSON arrays as inline args, but the email
   dispatch parsed only objects — an array silently became {} args.
   Non-object JSON now returns a correctable "arguments must be a JSON
   object" error before reaching the MCP server (same class as odysseus-dev#3966).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): classify all email tools for plan mode statically; reject invalid email JSON bodies

Review follow-up round 5 on odysseus-dev#3681 (thanks @RaresKeY):

1. This PR makes every BUILTIN_EMAIL_TOOLS name fence-taggable, so each
   one must be explicitly classified for plan mode — the draft tools and
   download_attachment were in neither the read-only allowlist nor the
   static denylist, leaving their bare-alias plan-mode safety dependent
   on the MCP read-only inventory being present and current.
   search_emails joins PLAN_MODE_READONLY_TOOLS (explicit, not
   allowed-by-omission); draft_email, draft_email_reply,
   ai_draft_email_reply, and download_attachment join the fail-closed
   _PLAN_MODE_KNOWN_MUTATORS backstop. (Moved back from the odysseus-dev#4053 split:
   the partition is directly required for this PR to merge
   independently.)

2. The classic tag/body fence form reaches execution unvalidated (only
   INLINE args are JSON-checked by the parser), so a body like
   {account: "work"} silently became {} args and read the DEFAULT
   mailbox instead of the intended one. JSON-looking bodies that fail to
   parse now return a correctable "not valid JSON" error before reaching
   the MCP server.

Tests: a partition invariant (every email tool is explicitly read-only
or plan-mode-denied), a mutating-alias probe that uses only the static
denylist with a fake MCP manager (no inventory layer), and the
body-form invalid-JSON regression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): decode inline JSON args for legacy MCP tools; reject all non-object email bodies

Review follow-up round 6 on odysseus-dev#3681 (thanks @RaresKeY) — both pre-existing
on this branch, surfaced by the relaxed inline-args parser:

1. The relaxed parser accepts inline JSON for every non-code tag, but
   the legacy line-based arg builders (web_search/web_fetch/read_file/
   write_file/generate_image/manage_memory) wrapped the whole JSON
   string as the query/url/path/prompt — so `web_search {"query": "x"}`
   executed as a search for the literal string `{"query": "x"}`.
   _build_mcp_args now uses a fenced JSON object directly when it carries
   the tool's primary arg key (query/url/path/prompt/action). Keyed off
   membership so it can't drift; an object without the primary key (e.g.
   a freeform JSON query, or bare object content for write_file) falls
   through to the line parser unchanged. Also fixes the same corruption
   for the classic newline-JSON form.

2. The bare-email dispatch only rejected bodies starting with { or [, so
   a non-empty non-JSON body like `account: work` still fell through to
   {} args and silently read the DEFAULT mailbox. Now ANY non-empty body
   must decode to a JSON object or it returns a correctable error; only a
   truly empty body keeps the no-arg path (```list_email_accounts```).

Tests: inline-JSON arg decoding for the five legacy tools plus the
freeform and missing-primary-key fallbacks; the email body rejection
extended to cover the brace-looking and bare `key: value` shapes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): drop dead manage_memory JSON-decode entry; pin the live-path invariant

Self-audit catch on the round-6 fix. manage_memory was added to
_MCP_JSON_PRIMARY_KEYS, but _build_mcp_args is only reached via
_call_mcp_tool, which only runs for _MCP_TOOL_MAP tools — and
manage_memory isn't one (its tag routes through dispatch_ai_tool ->
do_manage_memory, which line-parses). So the round-6 decode for
manage_memory was dead code: the unit test exercising _build_mcp_args
passed while a real `manage_memory {"action": ...}` fence still parsed
the whole JSON blob as the action.

Remove the dead entry and add test_mcp_json_primary_keys_are_all_live,
which asserts every JSON-primary tool is in _MCP_TOOL_MAP so a dead
decode can't be added again. The same inline-JSON corruption for
manage_memory and the other tools that route through positional
dispatchers (create_session, ui_control, send_to_session, search_chats,
the document tools, etc.) is pre-existing (dev corrupts their newline
JSON form too) and tracked separately; the proper fix there is to route
fenced JSON through function_call_to_tool_block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): decode inline JSON in WriteFileTool (its live path); round-6 fix was on the dead MCP path

Self-audit: round 6 claimed to fix inline JSON args for write_file via
_build_mcp_args, but there is no filesystem MCP server, so write_file
always runs through _direct_fallback -> WriteFileTool, never through
_build_mcp_args. WriteFileTool — unlike its siblings ReadFileTool /
WebSearchTool / WebFetchTool, which all decode JSON — took lines[0] as
the path, so `write_file {"path": "/tmp/x", "content": "y"}` wrote to a
file literally named with the JSON blob. The round-6 _build_mcp_args
entry decoded correctly but on a path that never executes (same class
as the manage_memory dead entry), and the round-6 unit test passed on
that dead path.

WriteFileTool now decodes a JSON object carrying "path" (matching
ReadFileTool directly above it), and the comment on _MCP_JSON_PRIMARY_KEYS
records that only generate_image has a live MCP server today — the other
entries are defense-in-depth for the MCP path; the live fix for each
server-less tool is in its handler.

Test: test_write_file_inline_json_args drives the LIVE path
(execute_tool_block with no MCP) and asserts the intended path is used —
verified to fail without the handler fix. web_search/web_fetch/read_file
were already correct (their handlers decode); write_file was the gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(strip-fence): derive the live-strip TOOL_TAGS from the real set

Semantic conflict from the dev merge that textual auto-merge didn't flag:
dev added test_live_strip_email_tool_fences.py whose _tool_tags() helper
source-scrapes only the TOOL_TAGS literal `{...}`, which worked on dev
because the email tool names were listed inline there. This branch makes
TOOL_TAGS the single source — `{...} | BUILTIN_EMAIL_TOOLS` — so the email
names are no longer in the literal and the scraper missed them, leaving the
email-fence strip assertions failing even though TOOL_TAGS does contain them
at runtime.

Import the real TOOL_TAGS instead of scraping source, so the test mirrors
exactly what GET /api/tools serves (sorted(TOOL_TAGS)) and the live
EXEC_FENCE_RE derives from — robust to however the set is composed. The
source-level frontend/route guards in the same file are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: botinate <285686135+botinate@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
samooth pushed a commit to samooth/ulises that referenced this pull request Jul 13, 2026
… email tool names (odysseus-dev#3681)

* fix(agent): execute fenced tool calls with inline args and bare email tool names

Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:

1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
   tool call with args on the same line ("```list_email_accounts {}")
   never matched and was never executed. The fence now matches with
   optional spaces/newline after the tag.

2. Even when parsed, bare email tool names had no dispatch branch in
   tool_execution.py and fell through to "Unknown tool type". They now
   route to the email MCP server as mcp__email__<name>, matching how
   function_call_to_tool_block already maps them for native callers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(security): block all bare email tool names for non-admins; harden fence-tag regex

Review follow-up on odysseus-dev#3681 (thanks @vgalin):

1. Routing bare email names made 10 of the 14 email tools executable by
   non-admin owners — is_public_blocked_tool() runs on the bare name
   before dispatch, and NON_ADMIN_BLOCKED_TOOLS only listed 4. Define the
   full email tool set once (BUILTIN_EMAIL_TOOLS in tool_security.py) and
   derive the blocklist, the fence tags (TOOL_TAGS), the bare-name
   dispatch, and the native-call mapping from it so they can't drift.
   This also fixes 4 tools (search_emails, draft_email, draft_email_reply,
   ai_draft_email_reply) that were missing from the old tool_schemas copy
   and therefore unreachable even for native function-calling models.

2. The relaxed fence regex from the previous commit could prefix-match
   longer fence tags: ```python3 parsed as tool "python" with content
   "3\nprint(...)" and executed as code. Add a (?![\w-]) boundary after
   the tag.

Tests: test_public_agent_policy_blocks_sensitive_tools now covers all 14
bare email names + the mcp__email__ form; new tests/test_fenced_inline_args.py
pins inline-args parsing, the python3/hyphenated-tag non-matches, and
strip/parse display mirroring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): gate bare and mcp-qualified email names together; stop executing Markdown info strings

Review follow-up on odysseus-dev#3681 (thanks @RaresKeY):

1. P1: execute_tool_block() checked disabled_tools / the turn ToolPolicy
   only against the incoming block name, then the bare-email branch
   qualified it to mcp__email__<name> and called the MCP manager. Plan
   mode and the MCP settings toggle write the QUALIFIED name into the
   denylist, so a bare fence like ```list_emails``` sailed past a
   mcp__email__list_emails entry. Both gates now match on both
   spellings (bare <-> mcp__email__-qualified), in either direction.

2. P2: the relaxed fence regex accepted arbitrary same-line text after
   a recognized tag, which made ordinary Markdown info strings
   executable: ```python title="example.py" ran as a python tool call.
   Same-line content now only counts as tool input when it starts with
   { or [ (JSON args); anything else leaves the fence as display text,
   and strip_tool_blocks mirrors that (the fence stays visible).

Tests: disabled-tools alias regression (qualified entry blocks bare
name and vice versa, never reaching the MCP manager), ToolPolicy alias
regression, python/bash title="..." non-execution + display retention,
and inline JSON-array args still parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): reject brace-style fence metadata; cover the full email set in the friendly toggle

Review follow-up round 3 on odysseus-dev#3681 (thanks @RaresKeY):

1. Brace-style fence metadata no longer executes. The previous narrowing
   still treated any same-line {/[ after a recognized tag as tool input,
   so ```bash {title="setup"} ran as a bash call. The fence header is now
   captured separately and judged by one predicate shared between
   parse_tool_blocks and strip_tool_blocks (_fenced_tool_call), so the
   execute and display decisions can't disagree: same-line content only
   counts as inline args when the tag is NOT a code tag (bash/python
   never take same-line args — that text is Markdown fence attributes)
   AND the inline text (plus any continuation lines) parses as standalone
   JSON. ```bash {title="setup"}, ```python {"title":"example.py"} and
   ```list_emails {title="x"} all stay visible and inert.

2. The friendly `disable_tool email` toggle covered 3 of the 14 email
   tools (mcp__email__{list_emails,read_email,send_email}); the other
   bare aliases this PR routes stayed executable after an operator
   disabled email. The alias now derives from BUILTIN_EMAIL_TOOLS in
   BOTH spellings — bare (function-schema hiding, bare-fence dispatch)
   and mcp__email__* (MCP schema hiding, qualified runtime blocks) —
   so the toggle and the runtime gate can't drift apart.

Tests: brace/bracket metadata regressions for parse and strip symmetry
(code tags, invalid-JSON inline on a JSON tool, multi-line inline JSON
still parsing), and disable_tool/enable_tool email covering all 14 names
in both spellings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(email): close remaining email-tool registry drift; classify every email tool for plan mode

Deep self-review follow-up on odysseus-dev#3681. Three review rounds each found another
hand-maintained copy of the email tool list that had drifted; this commit
hunts down ALL remaining copies and pins them to BUILTIN_EMAIL_TOOLS.

The same 5 tools (search_emails, draft_email, draft_email_reply,
ai_draft_email_reply, download_attachment) were missing from every
advertising surface, so they were dispatchable but never offered:

- FUNCTION_TOOL_SCHEMAS: native function-calling models never saw them
  (the round-1 fix covered dispatch only); schemas added, mirroring the
  email server's inputSchema definitions.
- TOOL_SECTIONS: fenced-block models were never told about them; prompt
  sections added.
- tool_index: absent from the RAG embedding registry (never retrievable),
  the email keyword hints, and the scheduled assistant's always-available
  set — the latter two now derive from BUILTIN_EMAIL_TOOLS.
- agent_loop._DOMAIN_TOOL_MAP["email"], tool_policy._COMMON_TOOL_NAMES,
  the assistant tool-selector UI groups (assistant.js), and the default
  Assistant crew seed (task_scheduler) now derive from / cover the set.

Plan mode now classifies every email tool explicitly:

- list_email_accounts and search_emails join PLAN_MODE_READONLY_TOOLS.
  Without this, list_email_accounts sat in the plan-mode bare denylist
  (schema-derived) while its qualified form passed the MCP read-only
  filter — and the round-2 bare/qualified alias gate would have blocked
  the qualified call too, regressing read-only email discovery in plan
  mode.
- draft_email, draft_email_reply, ai_draft_email_reply, and
  download_attachment join the fail-closed mutator backstop (drafts
  create documents; download_attachment writes to disk).

Tests: tests/test_email_registry_sync.py pins every registry (including
the email server source and assistant.js) to BUILTIN_EMAIL_TOOLS and
asserts the plan-mode partition, so the next email tool can't drift; a
parse/strip mirror grid covers 192 fence shapes (tag x header x body)
asserting executed <=> stripped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: move the email alias rule into tool_security; extract the assistant seed constant

Code-quality pass over the PR's own changes:

- The bare<->qualified email aliasing rule lived inline in the generic
  dispatcher (_execute_tool_block_impl). It is policy knowledge, so it
  moves next to BUILTIN_EMAIL_TOOLS as email_tool_policy_names(); the
  dispatcher just consumes it, and the rule gets its own unit test
  (including the mcp__email__<not-a-tool> and mcp__other__ non-alias
  cases).

- The default Assistant's enabled_tools list was an inline literal
  inside the CrewMember seed, and its registry-sync test asserted a
  source-code substring. Extracted to DEFAULT_ASSISTANT_ENABLED_TOOLS
  so the test imports and checks the actual value.

- _fenced_tool_call return type tightened to Optional[Tuple[str, str]].

No behavior change; suite green (3295 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* revert: move the email registry consolidation to a follow-up PR

Per review feedback on scope, this PR stays narrow: fenced inline-args
parsing, bare email tool routing, and the directly required safety
gates. This commit reverts the registry/advertising consolidation from
db29046 and 016ce47 (native schemas, prompt sections, RAG description
index + keyword hints, assistant always-available set, guide-only
known-names union, frontend tool-selector groups, default assistant
seed, and their sync tests) — all of that moves to a dedicated
follow-up PR together with the _EMAIL_TOOL_HINTS finding.

Kept here because the narrow scope needs them:
- email_tool_policy_names() in tool_security + its use in the
  execute_tool_block gates and its unit test (refactor of this PR's own
  round-2 alias fix),
- list_email_accounts in PLAN_MODE_READONLY_TOOLS (the alias gate works
  both ways, and the schema-derived plan-mode bare denylist would
  otherwise block the qualified read-only call too),
- the parse/strip mirror grid test (parser scope),
- the narrow registry sync tests (email server <-> BUILTIN_EMAIL_TOOLS
  match, fence-tag coverage, non-admin blocklist coverage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(email): execute empty email fences with empty args; reject non-object JSON args

Two gaps found by replaying captured local-model traffic against the
narrowed branch:

1. ```list_email_accounts``` with NO body — a shape gemma really emits
   for no-arg tools — was silently dropped (parse skips empty content),
   so the model concluded email was broken: the original odysseus-dev#337 symptom
   through a different door. Empty fences whose tag is a built-in email
   tool now dispatch with {} args and the tool's own validation answers
   (e.g. an empty send_email returns "to is required" instead of
   silence). Empty bash/python/other fences keep skipping, and strip
   stays mirrored (the fence was executed, so it is removed).

2. The fence parser accepts JSON arrays as inline args, but the email
   dispatch parsed only objects — an array silently became {} args.
   Non-object JSON now returns a correctable "arguments must be a JSON
   object" error before reaching the MCP server (same class as odysseus-dev#3966).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(security): classify all email tools for plan mode statically; reject invalid email JSON bodies

Review follow-up round 5 on odysseus-dev#3681 (thanks @RaresKeY):

1. This PR makes every BUILTIN_EMAIL_TOOLS name fence-taggable, so each
   one must be explicitly classified for plan mode — the draft tools and
   download_attachment were in neither the read-only allowlist nor the
   static denylist, leaving their bare-alias plan-mode safety dependent
   on the MCP read-only inventory being present and current.
   search_emails joins PLAN_MODE_READONLY_TOOLS (explicit, not
   allowed-by-omission); draft_email, draft_email_reply,
   ai_draft_email_reply, and download_attachment join the fail-closed
   _PLAN_MODE_KNOWN_MUTATORS backstop. (Moved back from the odysseus-dev#4053 split:
   the partition is directly required for this PR to merge
   independently.)

2. The classic tag/body fence form reaches execution unvalidated (only
   INLINE args are JSON-checked by the parser), so a body like
   {account: "work"} silently became {} args and read the DEFAULT
   mailbox instead of the intended one. JSON-looking bodies that fail to
   parse now return a correctable "not valid JSON" error before reaching
   the MCP server.

Tests: a partition invariant (every email tool is explicitly read-only
or plan-mode-denied), a mutating-alias probe that uses only the static
denylist with a fake MCP manager (no inventory layer), and the
body-form invalid-JSON regression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): decode inline JSON args for legacy MCP tools; reject all non-object email bodies

Review follow-up round 6 on odysseus-dev#3681 (thanks @RaresKeY) — both pre-existing
on this branch, surfaced by the relaxed inline-args parser:

1. The relaxed parser accepts inline JSON for every non-code tag, but
   the legacy line-based arg builders (web_search/web_fetch/read_file/
   write_file/generate_image/manage_memory) wrapped the whole JSON
   string as the query/url/path/prompt — so `web_search {"query": "x"}`
   executed as a search for the literal string `{"query": "x"}`.
   _build_mcp_args now uses a fenced JSON object directly when it carries
   the tool's primary arg key (query/url/path/prompt/action). Keyed off
   membership so it can't drift; an object without the primary key (e.g.
   a freeform JSON query, or bare object content for write_file) falls
   through to the line parser unchanged. Also fixes the same corruption
   for the classic newline-JSON form.

2. The bare-email dispatch only rejected bodies starting with { or [, so
   a non-empty non-JSON body like `account: work` still fell through to
   {} args and silently read the DEFAULT mailbox. Now ANY non-empty body
   must decode to a JSON object or it returns a correctable error; only a
   truly empty body keeps the no-arg path (```list_email_accounts```).

Tests: inline-JSON arg decoding for the five legacy tools plus the
freeform and missing-primary-key fallbacks; the email body rejection
extended to cover the brace-looking and bare `key: value` shapes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): drop dead manage_memory JSON-decode entry; pin the live-path invariant

Self-audit catch on the round-6 fix. manage_memory was added to
_MCP_JSON_PRIMARY_KEYS, but _build_mcp_args is only reached via
_call_mcp_tool, which only runs for _MCP_TOOL_MAP tools — and
manage_memory isn't one (its tag routes through dispatch_ai_tool ->
do_manage_memory, which line-parses). So the round-6 decode for
manage_memory was dead code: the unit test exercising _build_mcp_args
passed while a real `manage_memory {"action": ...}` fence still parsed
the whole JSON blob as the action.

Remove the dead entry and add test_mcp_json_primary_keys_are_all_live,
which asserts every JSON-primary tool is in _MCP_TOOL_MAP so a dead
decode can't be added again. The same inline-JSON corruption for
manage_memory and the other tools that route through positional
dispatchers (create_session, ui_control, send_to_session, search_chats,
the document tools, etc.) is pre-existing (dev corrupts their newline
JSON form too) and tracked separately; the proper fix there is to route
fenced JSON through function_call_to_tool_block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tool-dispatch): decode inline JSON in WriteFileTool (its live path); round-6 fix was on the dead MCP path

Self-audit: round 6 claimed to fix inline JSON args for write_file via
_build_mcp_args, but there is no filesystem MCP server, so write_file
always runs through _direct_fallback -> WriteFileTool, never through
_build_mcp_args. WriteFileTool — unlike its siblings ReadFileTool /
WebSearchTool / WebFetchTool, which all decode JSON — took lines[0] as
the path, so `write_file {"path": "/tmp/x", "content": "y"}` wrote to a
file literally named with the JSON blob. The round-6 _build_mcp_args
entry decoded correctly but on a path that never executes (same class
as the manage_memory dead entry), and the round-6 unit test passed on
that dead path.

WriteFileTool now decodes a JSON object carrying "path" (matching
ReadFileTool directly above it), and the comment on _MCP_JSON_PRIMARY_KEYS
records that only generate_image has a live MCP server today — the other
entries are defense-in-depth for the MCP path; the live fix for each
server-less tool is in its handler.

Test: test_write_file_inline_json_args drives the LIVE path
(execute_tool_block with no MCP) and asserts the intended path is used —
verified to fail without the handler fix. web_search/web_fetch/read_file
were already correct (their handlers decode); write_file was the gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(strip-fence): derive the live-strip TOOL_TAGS from the real set

Semantic conflict from the dev merge that textual auto-merge didn't flag:
dev added test_live_strip_email_tool_fences.py whose _tool_tags() helper
source-scrapes only the TOOL_TAGS literal `{...}`, which worked on dev
because the email tool names were listed inline there. This branch makes
TOOL_TAGS the single source — `{...} | BUILTIN_EMAIL_TOOLS` — so the email
names are no longer in the literal and the scraper missed them, leaving the
email-fence strip assertions failing even though TOOL_TAGS does contain them
at runtime.

Import the real TOOL_TAGS instead of scraping source, so the test mirrors
exactly what GET /api/tools serves (sorted(TOOL_TAGS)) and the live
EXEC_FENCE_RE derives from — robust to however the set is composed. The
source-level frontend/route guards in the same file are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: botinate <285686135+botinate@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
ChicoCifrado added a commit to digitalsovereignsociety/ulises that referenced this pull request Jul 20, 2026
… resolve code quality items (#1)

* Code quality: security, deps, error handling, tests, frontend, database

Phase 0 — Security (5/5):
  - Fix bare except in require_admin() (middleware.py)
  - Replace shell=True with list-form subprocess (builtin_actions.py)
  - Add .catch() to 3 dynamic imports (app.js)
  - Encrypt Webhook.secret with EncryptedText
  - Verify XSS safety in chat.js

Phase 1 — Deps/Config (7/7):
  - Pin all 28 deps; move test deps to requirements-dev.txt
  - Set asyncio_loop_scope = module, trim CORS whitespace
  - Pin Docker to python:3.13-slim
  - Replace __import__ with standard imports

Phase 2 — Error Handling (3/3):
  - Create safe_create_task() helper; fix 13 fire-and-forget tasks
  - Lock _response_cache with threading.Lock()
  - Narrow bare excepts in subprocess_tools, builtin_actions,
    agent_loop, task_scheduler (24 instances fixed)

Phase 3 — Tests (2/2):
  - Fix test_session_manager explicit assertion
  - Remove src.database MagicMock fallback in conftest.py

Phase 4 — Frontend (5/5):
  - Scope MutationObservers to #app; add beforeunload cleanup
  - Create shared API_BASE module (14 files updated)
  - Fix admin.js undefined API_BASE bug
  - Fix chatRenderer.js window.API_BASE fallback
  - Fix _defaultChat race, rename guard in app.js

Phase 5 — Database (3/3):
  - Defer init_db() from module import to app startup
  - Add .limit() safety caps to 11 unbounded queries
  - Extract 26 ORM models to core/orm_models.py (709 lines)
  - Add index=True to 6 FK columns

Phase 6 — Cross-cutting (3/4):
  - Add Docker socket security warning
  - Fix dead-host cache: move _clear_host_dead after status check
  - Cache EncryptedText import to avoid per-call overhead
  - Add -> None to note_model_activity

* making ulises multilang

* finished multilang base & tests

* fix(security): scope owner-less email accounts to a mailbox match in route guards (#5238)
The HTTP email route guard `_assert_owns_account` and the explicit-account_id
path in `_get_email_config` gated cross-tenant access with
`if row.owner and row.owner != owner` -- which skips the check entirely when the
account row is owner-less (owner NULL or ""). `email_accounts` is the one
owner-scoped table left out of the legacy-owner migration backfill
(core/database.py), so such rows persist on multi-user deploys: an account
configured while auth was disabled, or an imported legacy row. Any authenticated
user could then pass that account's id to read/send/update-credentials/delete
another tenant's mailbox and read its decrypted IMAP/SMTP creds.

Both sibling paths already enforce the intended contract -- the same-file
`_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner`
gate (whose comment says it mirrors "the HTTP email route fallback") only expose
an owner-less account when its own mailbox (imap_user / from_address) is the
caller's. Factor that row-level predicate into `_account_visible_to_owner` and
use it in both guards, so owner-less accounts are visible only on a mailbox
match. Owned accounts, the legacy-claim path, and single-user mode (owner == "")
are unchanged.

Complements #5234 (which fixes the same class on the MCP tool layer); this is
the HTTP route layer it does not touch.

* fix(agent): fall back to keyword tool selection when retrieval times out
The retrieval-timeout branch hard-coded ALWAYS_AVAILABLE, silently skipping
the deterministic keyword hints whenever the embedding backend was slow
(e.g. a remote endpoint cold-loading its model). Queries that named email
or calendar outright lost those tools and the model concluded the
integrations did not exist. Let the timeout fall through to the existing
keyword fallback instead — same baseline, plus the hints.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(search): pin httpx connection to resolved IP to block DNS rebinding (#704)
* fix(search): pin DNS-validated fetch connections

Rebase the DNS-rebinding SSRF fix onto current dev after search content moved behind the services.search.content canonical module.

Integrate the pinned httpcore NetworkBackend/BaseTransport approach with the current size-capped Client.stream fetch path, preserving Host/SNI semantics while forcing TCP connect to the already validated resolved IP.

Keep src.search.content as the compatibility wrapper and preserve existing OG-image http(s) behavior; this avoids reintroducing the unrelated scope changes that previously blocked review.

Add the explicit httpcore>=1.0,<2.0 requirement used by the public httpcore NetworkBackend and ConnectionPool APIs.

* test(search): restore and rebase DNS rebinding regressions

Keep the current security regression coverage that the stale PR branch had deleted, including auth-disabled localhost bypass and Ollama cookbook hardening tests.

Carry forward the DNS-rebinding coverage for private resolve blocking, pinned TCP connect behavior, Host header preservation, redirect revalidation, and the BaseTransport/public-httpcore static guard.

Update redirect tests to mock the current Client.stream-based capped fetch path rather than the older httpx.stream/get path.

* test(search): adapt size-cap fetch tests to pinned client stream

The DNS-rebinding repair moved _get_public_url from the module-level httpx.stream shortcut to httpx.Client(...).stream(...) so the fetch can use the pinned transport.

Keep the existing size-cap test fakes by routing Client.stream through the monkeypatched httpx.stream only when a test has installed that fake; otherwise fall back to a real Client.

This fixes the CI failures in tests/test_web_fetch_size_caps.py without touching unrelated upload-handler atomicity behavior, which is already flaky on clean origin/dev.

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix: resolve RAG manager search signature TypeError (#4994)
* fix: resolve RAG manager search signature TypeError and adjust similarity threshold

* fix: revert similarity threshold change to keep PR focused

* test(rag): remove trailing whitespace

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix(security): scope send_to_session to an exact session owner

* fix(tools): handle non-dict JSON values in _parse_tool_args (closes #5043) (#5064)

* fix(search): extract non-ASCII capitalized names in _extract_entities

* fix(agent): skip deny-listed sensitive files in glob (#5094)

* fix(security): match the sensitive-file deny-list case-insensitively (#5097)

* fix(integrations): don't append a trailing slash when api_call path is '/'

* fix(calendar): honor list_events date range aliases (#3283)

* fix: auto-spam move/delete targets the wrong message (seqnum vs UID) (#1874)

* fix(session): use utcnow_naive across session routes (#1116) (#5003)

* fix(cookbook): stop Ollama runner from executing the install one-liner (#3926)

* fix(security): apply the webhook SSRF guard to the reminder ntfy sender (#5142)

* fix(security): pin webhook delivery to the SSRF-validated IP (DNS rebinding) (#5147)

* fix(security): validate integration api_call URLs with the outbound SSRF guard (#5145)

* fix(security): match grep's rg sensitive-file exclusions case-insensitively (#5189)

* Fix Cookbook download runner for Python 3.11

* fix: add grace period to document tidy to prevent deleting new documents (#5036)

* fix(email): enforce MCP account owner scope (#5234)

* fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704)

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

* fix(security): prevent ReDoS in XML and args tool-call parsers (#4941)

* fix(chat): sanitize web search query to strip markdown and code blocks (#4863)

* fix(security): batch A — 7 security fixes

* fix(bugfix): batch B — 16 critical bugfixes

* fix(chat): batch C — web search denial + explicit intent

* test(ci): batch D — test splits + CI guidance

* feat(refactor): batch E — features, refactors, catalog

* fix(infra): batch F — 16 infrastructure + bugfixes

* feat(docs): batch G — models, documents, docs updates

* fix(ux): batch H — email, gallery, cookbook UX fixes

* feat(models): batch I — models hardening + OCR captions

* ui: batch J1 — cookbook/llama model UI spacing

* ui: batch J2 — cookbook engine tint, MTP, vllm spacing

* feat(background): batch J3 — background tasks, CLI sanitization, info-budget fixes

* chore(release): bump version to 1.0.2

* fix(ui): lower cookbook serve top row (cherry-pick fc6a6dc)

* fix(chat): honor explicit web search denial

* fix(mobile): stack the model-comparison grid into one column on phones (#4979)

The comparison grid hard-codes 2-4 equal columns with no phone breakpoint, so at
390px two models get ~178px columns and four get ~88px columns. Each column is a
full scrolling chat, so content is unreadably over-wrapped and clipped. On
phones (<=768px), stack the panes into a single scrollable column. Desktop is
unaffected.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docs): correct broken backup-restore link in setup.md

Fixes #4926 - the link used docs/backup-restore.md from within docs/setup.md, which resolved to docs/docs/backup-restore.md (404). Changed to same-directory relative path.

* docs(setup): add a self-host troubleshooting cookbook of common traps (#4834)

ROADMAP "Self-host troubleshooting cookbook" asks to document the weird
30-second fixes that otherwise become 30-minute searches. Adds a "Common
self-host traps" subsection under Troubleshooting covering: the UTF-8 BOM
.env gotcha (app.py loads with utf-8-sig), macOS AirPlay holding port 7000
(the start script uses 7860), the plain-HTTP Tailscale/LAN clipboard
limitation, self-hosted ntfy delivery (NTFY_BIND/NTFY_BASE_URL + the ntfy
Android Instant-delivery toggle), Dovecot cleartext-auth on LAN mail stacks,
and Radicale full-collection-URL sync.

Docs only; grounded in existing repo behavior (.env.example NTFY_* block,
app.py utf-8-sig loader, start-macos.sh port choice).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task
executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike
the chat path which stores build_chat_url(normalize_base(base)) on the
session. A task carrying an explicit bare OpenAI-compatible base such as
"http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the
agent loop swallows the empty body into "The model returned an empty
response" and marks the run success, so nothing surfaces the failure.

Tasks that omit an endpoint dodge this only because _resolve_defaults()
cribs an already-full URL from a recent chat session. The API/token path
(e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1")
hits it every time.

Fix: route every resolved task endpoint through _normalize_chat_endpoint()
at the three resolution sites (_execute_llm_task, the persona/research
session path, and _execute_research_task). The helper is idempotent
(strips any existing chat suffix, re-appends the correct one) and leaves
native-Ollama (/api...) and already-concrete URLs untouched, so other
providers are unaffected. Proven via isolated repro: ".../v1" -> 404 ->
empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output.

Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency,
and the native-Ollama/empty passthroughs.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787)

setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv()
but never loaded .env, so on native Linux/macOS installs a password
pre-seeded in .env (documented in docs/setup.md and .env.example) was
silently ignored and a random one generated, breaking the first login.
Docker was unaffected because compose passes the vars into the container env.

Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(),
mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does
not override already-exported OS vars, so the existing precedence is kept.
python-dotenv is already a required dependency.

Adds a regression test that pre-seeds credentials only in .env (not the
shell) and asserts the stored bcrypt hash matches the pre-seeded password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(llm): detect mistral.ai provider and support reasoning_effort (#4698)

* fix(llm): detect mistral.ai provider and support reasoning_effort

Four coupled bugs broke Mistral thinking model support:

1. _detect_provider() had no mistral.ai host check, so all Mistral
   endpoints fell through to the generic 'openai' provider string.
   _provider_display_name() correctly identified them as 'Mistral',
   making any 'if provider == "Mistral"' check elsewhere dead code.

2. reasoning_effort parameter was never sent in the request payload,
   so Mistral never activated thinking mode even when the user
   configured a thinking-capable model (mistral-small-latest,
   mistral-medium-latest, magistral-*).

3. Mistral returns content as a typed array
   ([{"type":"thinking",...},{"type":"text",...}]) when
   reasoning is on, not as a plain string. Both the streaming and
   non-streaming parsers expected strings and silently dropped the
   thinking content.

4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-*
   model prefixes, so the frontend wouldn't tag reasoning output
   as thinking even after the above were fixed.

Fix:
- Add mistral.ai to _detect_provider() host checks
- Add a _normalize_mistral_content() helper that splits the typed
  array into (text, thinking) strings
- Inject payload["reasoning_effort"] = "high" when provider is
  Mistral and _supports_thinking(model) is true, in both stream_llm
  and llm_call_async payload construction
- Wire the normalizer into both response parsers
- Extend _THINKING_MODEL_PATTERNS to include magistral,
  mistral-small, mistral-medium, mistral-large

Tested on Docker install with mistral-small-latest +
reasoning_effort=high. Reasoning streams correctly into the
thinking panel after the fix.

Fixes #4678

* fix(llm): address review — lowercase provider id, configurable effort, tests

Addresses vdmkenny's review on PR #4698:

1. Removed duplicate 'if provider == "mistral"' block in stream_llm
   — two back-to-back copies, one was dead-redundant.

2. Dropped personal-context comment ('free-tier limits are generous
   for this user') and made reasoning_effort configurable via env var
   ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none).
   Default remains 'high' for backward compat with the tested behavior.

3. Recased provider id from 'Mistral' to 'mistral' to match the
   lowercase convention used by every other provider id in the file
   (openai, anthropic, ollama, copilot, ...). _provider_display_name()
   still returns the Title-Case 'Mistral' for UI labels — only the
   runtime id used in 'if provider == ...' checks was recased.

4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning
   _normalize_mistral_content()'s contract: string passthrough, the
   Mistral array format (thinking + text blocks), and edge cases
   (empty, garbage, None, wrong types, missing fields, string-vs-array
   inner thinking field).

Also fixed a gap the review didn't catch: the non-streaming paths
(llm_call sync + llm_call_async) were missing the reasoning_effort
injection entirely. Added the same injection to both, so Deep Research
and agent tool calls also activate Mistral thinking.

All 13 new tests pass. Existing reasoning/streaming/ollama-thinking
tests still pass (38 tests, no regressions).

Fixes #4678

* fix(chat): restore missing _explicit_web_intent definition (#5290)

chat_stream() references `_explicit_web_intent` in three places
(disabled-tools gating, global-disabled web allowance, and the
per-turn tool filter) but the assignment was dropped during a
branch merge. Every chat request raised

    NameError: name '_explicit_web_intent' is not defined

at routes/chat_routes.py, surfacing to the client as a bare
"Internal Server Error" before any LLM call was made — chat was
fully broken on dev and main.

Restore the original definition, computed from the already-derived
tool intent, immediately before its first use:

    _explicit_web_intent = bool(_tool_intent and _tool_intent.category == "web")

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 2d8177035b6a1dd3e9211c6cb5ba68443b826504)

* Parse local function_model tool wrappers

* Show fallback model in picker

* Nudge serve engine control up

* Move core serve memory fields farther left

* Move vllm block size left again

* Nudge vllm attention field right

* Raise unified llama context estimate

* fix(llm-core): prevent cache-affinity fields from reaching Cerebras

Recognize api.cerebras.ai as a Cerebras cloud provider so llama.cpp/LM Studio cache-affinity fields are not attached even when endpoint_kind is misconfigured as local. Add regression coverage for provider detection, self-hosted classification, and payload field exclusion.

* fix(models): accept bare-list /models responses (Together AI) (#4761)

* fix(api): handle varying response formats for model IDs from compatible providers

merge conflict for pr-2204 resolved

* fix(modal): keep body-portaled dropdowns above their tool modal at any stack depth (#4720) (#4724)

* fix(memory): keep the Brain memory item menu above the modal at any stack depth

The memory item "⋮" dropdown is portaled to <body> with a hardcoded
z-index of 10001. Tool modals, however, get a monotonically increasing
z-index from modalManager's bring-to-front counter (_modalTopZ), which
climbs unbounded as modals are opened/restored over a session. Once that
counter passes 10001, the Brain modal stacks above the body-portaled
dropdown, so the menu renders behind the panel — visible only where it
spills past the modal's edge (#4720).

Derive the dropdown's z-index from the owning modal's current z-index
(+1), keeping 10001 as a floor for the common low-counter case, so the
menu always sits just above its modal however high the counter has climbed.

Verified with document.elementFromPoint at the dropdown's location: with a
high modal z-index the old build returns the modal at every sampled point
(menu behind); the fixed build returns the dropdown (menu on top). The
default low-counter case is unchanged (z stays 10001).

* refactor(modal): route body-portaled dropdowns through a shared topPortalZ() helper

The hardcoded z-index:10001 the Brain memory menu used (#4720) is the same
literal shared by ~16 body-portaled dropdowns across calendar, cookbook,
cookbookServe, documentLibrary, emailLibrary, gallery, notes, emojiPicker and
memory — each renders behind its owning tool modal once modalManager's
bring-to-front counter climbs past the literal over a long session.

Promote the per-dropdown fix into a single topPortalZ() helper in
toolWindowZOrder.js — the existing source of truth for tool-window z, already
imported by modalManager's _bringToFront and notes.js — returning
max(topToolWindowZ(), dock-chip floor) + 1, so a portaled dropdown always sits
just above the live tool-window stack however high the counter has climbed.
Route all 16 sites through it. The slashCommands tour tooltips and the
cookbookServe VRAM dialog are intentionally left out (neither is a modal-owned
portaled dropdown).

Add tests/test_portal_dropdown_z_js.py covering the helper, including the #4720
scenario (modal counter at 99999 -> dropdown at 100000). Existing
test_notes_z_order_js.py stays green.

* fix(llm): detect mistral.ai provider and support reasoning_effort (#4698)

* fix(llm): detect mistral.ai provider and support reasoning_effort

Four coupled bugs broke Mistral thinking model support:

1. _detect_provider() had no mistral.ai host check, so all Mistral
   endpoints fell through to the generic 'openai' provider string.
   _provider_display_name() correctly identified them as 'Mistral',
   making any 'if provider == "Mistral"' check elsewhere dead code.

2. reasoning_effort parameter was never sent in the request payload,
   so Mistral never activated thinking mode even when the user
   configured a thinking-capable model (mistral-small-latest,
   mistral-medium-latest, magistral-*).

3. Mistral returns content as a typed array
   ([{"type":"thinking",...},{"type":"text",...}]) when
   reasoning is on, not as a plain string. Both the streaming and
   non-streaming parsers expected strings and silently dropped the
   thinking content.

4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-*
   model prefixes, so the frontend wouldn't tag reasoning output
   as thinking even after the above were fixed.

Fix:
- Add mistral.ai to _detect_provider() host checks
- Add a _normalize_mistral_content() helper that splits the typed
  array into (text, thinking) strings
- Inject payload["reasoning_effort"] = "high" when provider is
  Mistral and _supports_thinking(model) is true, in both stream_llm
  and llm_call_async payload construction
- Wire the normalizer into both response parsers
- Extend _THINKING_MODEL_PATTERNS to include magistral,
  mistral-small, mistral-medium, mistral-large

Tested on Docker install with mistral-small-latest +
reasoning_effort=high. Reasoning streams correctly into the
thinking panel after the fix.

Fixes #4678

* fix(llm): address review — lowercase provider id, configurable effort, tests

Addresses vdmkenny's review on PR #4698:

1. Removed duplicate 'if provider == "mistral"' block in stream_llm
   — two back-to-back copies, one was dead-redundant.

2. Dropped personal-context comment ('free-tier limits are generous
   for this user') and made reasoning_effort configurable via env var
   ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none).
   Default remains 'high' for backward compat with the tested behavior.

3. Recased provider id from 'Mistral' to 'mistral' to match the
   lowercase convention used by every other provider id in the file
   (openai, anthropic, ollama, copilot, ...). _provider_display_name()
   still returns the Title-Case 'Mistral' for UI labels — only the
   runtime id used in 'if provider == ...' checks was recased.

4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning
   _normalize_mistral_content()'s contract: string passthrough, the
   Mistral array format (thinking + text blocks), and edge cases
   (empty, garbage, None, wrong types, missing fields, string-vs-array
   inner thinking field).

Also fixed a gap the review didn't catch: the non-streaming paths
(llm_call sync + llm_call_async) were missing the reasoning_effort
injection entirely. Added the same injection to both, so Deep Research
and agent tool calls also activate Mistral thinking.

All 13 new tests pass. Existing reasoning/streaming/ollama-thinking
tests still pass (38 tests, no regressions).

Fixes #4678

* fix: Images cannot be seen by model that is vision capable (#4726)

* fix: Images cannot be seen by model that is vision capable

* fix: skip http(s) image_url for Ollama (images[] is base64-only)

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix(chat): strip executed email tool fences from the live stream (#3993) (#4275)

* fix(chat): strip executed email tool fences from the live stream (#3993)

The backend strips every fenced tool block from persisted text (the regex in
src/tool_parsing.py is built from the full TOOL_TAGS set, which includes the
email tools), so a reloaded session renders cleanly. The live frontend path
uses a separate hardcoded EXEC_FENCE_RE in static/js/chatRenderer.js that only
listed web_search/read_file/write_file/create_document/edit_document/
update_document — so executed email tool fences (list_emails, etc.) lingered as
raw code blocks in the live assistant bubble until the user reloaded.

Add the nine email tool tags to EXEC_FENCE_RE so the live render settles into
the same clean layout as the history reload. bash/python stay excluded on
purpose: those are languages a user may legitimately have asked the model to
show as code, not tool invocations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): single-source live exec-fence tool list from TOOL_TAGS (#3993)

Per review: EXEC_FENCE_RE was a second, hand-maintained copy of the
executable-tool list, so any tool not in it — and every future tool added to
TOOL_TAGS — would leave its executed fence lingering in the live bubble until
reload (the original #3993 bug, recurring one tool at a time).

EXEC_FENCE_RE is now built from an explicit EXEC_TOOL_TAGS list that mirrors
TOOL_TAGS (src/agent_tools/__init__.py) minus bash/python, which stay excluded
as legitimate code-example languages. A new regression test
(test_exec_fence_re_covers_all_executable_tools) extracts both lists from
source and fails if they drift, so the whole class is caught in CI instead of
by a user — the "minimum acceptable middle ground" from the review, made exact
(set equality, not just coverage).

Verified: pytest tests/test_live_strip_email_tool_fences.py (5 passed);
node --check static/js/chatRenderer.js; and a node run of the built regex
confirms email/generate_image/manage_memory/ls fences strip while
bash/python/sh are preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): build live exec-fence list from /api/tools at runtime (#3993)

Make TOOL_TAGS the single source for live exec-fence stripping. chatRenderer.js
no longer hard-codes a tool list; it fetches the backend's authoritative set
once from GET /api/tools (sorted(TOOL_TAGS)) and builds EXEC_FENCE_RE from it at
load, minus bash/python. No second list to drift, and a future tool added to
TOOL_TAGS is covered automatically — without touching the streaming path.

Until the fetch resolves EXEC_FENCE_RE is null and exec fences aren't stripped
(a sub-second window before the first stream); the backend already strips
persisted history, so a reload always renders clean.

Drop test_exec_fence_re_covers_all_executable_tools (no hand-maintained list to
guard) and add source-level guards: the frontend keeps no hard-coded list and
fetches /api/tools, and the endpoint serves the full sorted(TOOL_TAGS).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

* fix(chat): warn on /api/tools fetch failure instead of swallowing it (#3993)

A fresh-context review flagged that loadExecFenceRegex's catch silently
discarded errors: if the one-shot fetch fails, EXEC_FENCE_RE stays null for the
whole session and live exec fences go unstripped until reload, with zero signal.
console.warn it, and correct the comment to describe the failure mode honestly
(was understated as just a sub-second startup window).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): log and cleanly 500 on unreadable HTML page (#4637)

* fix(routes): serve 404 instead of 500 when an HTML page file is missing

_serve_html_with_nonce opened the HTML file with no error handling, and
callers such as /backgrounds and /login pass their paths in with no
existence check, so a missing or unreadable file raised an unhandled
OSError that surfaced as a 500. Wrap the read and raise HTTPException(404)
instead; the normal render path (CSP-nonce substitution) is unchanged.

Fixes #4594

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): distinguish missing page (404) from read failure (500)

The previous fix caught a broad OSError and returned 404 for every
failure, which masks real server-side problems (permission errors, I/O
failures) as "not found" and lets them slip past error alerting. Split
FileNotFoundError (genuine 404) from other OSError, which now logs the
exception and returns a generic 500 — without leaking the OS error
string or file path into the response body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): treat unreadable bundled HTML page as logged 500, not 404

Per PR #4637 review: every caller of the page-render helper serves a fixed,
server-owned template (index/login/backgrounds), never a client-supplied
path. So a missing or unreadable file is a server fault (broken deployment),
not a client "not found" — a 404 there mislabels a server error and hides a
missing core template from 5xx alerting, contradicting the OSError->500
rationale this PR is built on. Collapse both branches into a single logged,
leak-free 500.

Move the helper to src.app_helpers.serve_html_with_nonce so the behavior can
be unit-tested without importing the whole app (app.py is the slim
orchestrator; the test harness stubs src.database, so importing app in tests
is not viable). Add tests pinning missing/unreadable -> 500 (not 404) and
nonce injection on the happy path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(catalog): add Gemma 4 12B/QAT entries and RTX 3050 bandwidth (#4728)

Add official Gemma 4 12B-it plus QAT-INT4/INT8 catalog entries (with their
GGUF sources), QAT quantization support across the quant tables and the
prequantized-prefix list, and the missing RTX 3050 / 3050 Ti memory
bandwidth so speed estimates stop falling back to the generic cuda value.

* fix debugging on windows (#4679)

* fix: Real-ESRGAN install + Cookbook deps-panel crash on the Python 3.14 image (#4694)

* fix(docker): make Real-ESRGAN installable on the Python 3.14 image

realesrgan's deps basicsr/gfpgan/facexlib (unmaintained since 2022) read
their version in setup.py via `exec(...); locals()['__version__']`, which
raises KeyError on Python 3.13+ — PEP 667 made locals() in a function an
independent snapshot that exec() can no longer mutate. That fails the
Cookbook "install realesrgan" sdist build on the python:3.14 base.

Add a `realesrgan-wheels` builder stage that fetches the pinned sdists,
patches get_version() to exec into an explicit namespace dict, and builds
wheels; the final stage installs them --no-deps so a later
`pip install realesrgan` resolves from wheels instead of rebuilding the
broken sdists. torch stays a runtime pull to keep the base image lean.

Also add the runtime libs opencv-python (cv2) needs — libgl1,
libglib2.0-0t64, libxcb1 — which the slim base omits; without them the
install succeeds but `import cv2` dies with
`libxcb.so.1: cannot open shared object file`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cookbook): don't let a package's sys.exit() on import hang the deps panel

The local optional-dependency probe imports each package in-process and
catches ImportError / Exception. But a package can call sys.exit() at
import time — e.g. rembg does `sys.exit(1)` when no onnxruntime backend
loads. SystemExit is a BaseException, not Exception, so it escaped the
probe, propagated out of the list_packages endpoint, and hung the whole
Dependencies panel / worker (the UI loads forever).

Catch (Exception, SystemExit) so one broken optional package is reported
as not-usable instead of taking down the panel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): 500 (not 404) when the app-shell index.html is missing (#4791)

Follow-up to #4637. serve_index — the handler for / and the SPA deep-link
routes (/notes, /calendar, /cookbook, /email, /memory, /gallery, /tasks,
/library) — pre-checked os.path.exists and raised its own
HTTPException(404, "index.html not found") when the bundle was missing. So a
missing core template returned 404 before serve_html_with_nonce's 500 could
fire, the one inconsistency left after #4637.

index.html is a fixed, app-bundled template; a missing one is a broken
deployment (server fault), not a client "not found", so it should surface as a
logged 500 in 5xx alerting rather than a 404. Keep the static->root fallback,
drop the redundant existence guard and the dead-end 404, and let the shared
helper handle the missing case.

Verified against the running app: / and /notes return 200 with the bundle
present and a logged 500 when index.html is absent.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787)

setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv()
but never loaded .env, so on native Linux/macOS installs a password
pre-seeded in .env (documented in docs/setup.md and .env.example) was
silently ignored and a random one generated, breaking the first login.
Docker was unaffected because compose passes the vars into the container env.

Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(),
mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does
not override already-exported OS vars, so the existing precedence is kept.
python-dotenv is already a required dependency.

Adds a regression test that pre-seeds credentials only in .env (not the
shell) and asserts the stored bcrypt hash matches the pre-seeded password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: email poller marks calendar extraction processed on LLM failure (#4622)

Move calendar processed-marker insert into the LLM success path (else branch).
Previously, the INSERT ran even after a transient LLM failure, causing the
poller to skip retrying calendar extraction on subsequent runs.

Minimal change: only touches the try/except/else control flow in
_auto_summarize_pass_single() — preserves existing formatting and line endings.

* feat(ui): add toggle for padding around chat area (#4691)

* feat: Allow admins to choose if they want to share defaults (#4752)

* First bare fix

* Adding the option toggle

* toggle function fix

* Final fix, added missing /auth/

* Extended toggle text & added tests

* Comments change

* Description toggle change

* br tag fix

* description change based on suggestion

* fix(agent): parse misfenced read_file calls (#4799)

* fix: use atomic write in APIKeyManager.save() to prevent credential data loss (#4591) (#4597)

* fix: use atomic write in APIKeyManager.save() to prevent data loss

Opening api_keys.json with 'w' truncates the file before writing, so a
crash, disk-full, or mid-write error leaves all stored provider API keys
corrupted. Switch to atomic write (temp file + fsync + os.replace) so
the original file is always intact on any failure.

Fixes #4591

* chore: trigger CI re-run

* chore: update PR description

* chore: fix how-to-test section for description check

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* feat(discovery): detect llama.cpp servers and label local providers (#4729)

* feat(discovery): detect llama.cpp servers and label local providers

Scan port 8080 (llama-server) and 11435 (APFEL) during discovery, fingerprint
llama.cpp via its native /props endpoint, and label well-known local serving
ports (8080 llama.cpp, 8000 vLLM, 1234 LM Studio, 11434 Ollama) consistently
in both the Python provider helper and the JS endpoint UI. Adds a llama.cpp
hint to the /setup slash command.

* fix(discovery): don't infer the serving tool from the port alone

Per review: vLLM, SGLang, llama.cpp and plain OpenAI-compatible servers all
share 8000/8080, so labeling by port mislabels real setups (a vLLM box on 8080
shown as llama.cpp). Drop the port->tool assertions from _provider_label and
providerLabel; the authoritative signal is the /props fingerprint done during
discovery, which is unchanged. Loopback now reads a neutral 'local endpoint' /
'Local'. Tests updated to assert the neutral labels.

* refactor(tools): migrate config/integration admin tools to the registry (#4742)

Part of #3629 (the `admin_tools.py` bullet). Moves the config/integration admin
tools off the legacy elif dispatch chain in tool_implementations.py onto the
agent_tools registry:

  manage_endpoints, manage_mcp, manage_webhooks, manage_tokens, manage_settings

The do_* implementations (and manage_mcp's command-allowlist / RCE guard:
_validate_mcp_command, _mcp_allowed_commands, and the _MCP_* constants) move
verbatim into the new src/agent_tools/admin_tools.py. They register through a
single ADMIN_TOOL_HANDLERS map that TOOL_HANDLERS.update()s, and the five elif
branches plus their imports are dropped from tool_execution.py, so these tools
now flow through _direct_fallback like the other migrated clusters. The names
are re-exported from src.agent_tools for back-compat.

Dedup:
  - _parse_tool_args was duplicated in tool_implementations.py and
    document_tools.py. It now lives once in src.tool_utils (which imports nothing
    from the project beyond src.constants, so this introduces no cycle) and both
    call sites import it from there. The orphaned `import json` in document_tools
    is removed with it.
  - The five tools share one _owner_adapter(fn) factory that threads ctx["owner"]
    into the owner-taking do_* signature, instead of five near-identical wrappers.

Tests: new tests/test_admin_tools_registry.py pins the registration, the
re-export back-compat, the owner-threading adapter, and the single-source
_parse_tool_args (across admin_tools and document_tools). Existing MCP /
settings / webhook suites are repointed at the new module.

* refactor(exceptions): dedupe src/exceptions via core re-export (#4785)

src/exceptions.py was a byte-for-byte duplicate of the canonical
core/exceptions.py. Replace its class bodies with a re-export shim
(mirroring the core/constants.py -> src/constants.py pattern) so the
exception classes are defined in exactly one place. Also fix the stale
"# src/exceptions.py" header comment in core/exceptions.py.

No behavior change: both import paths resolve to the same class objects
(verified by identity), so `except SessionNotFoundError` works regardless
of which module it was imported from. Ran py_compile and
pytest tests/test_app.py (12 passed).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task
executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike
the chat path which stores build_chat_url(normalize_base(base)) on the
session. A task carrying an explicit bare OpenAI-compatible base such as
"http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the
agent loop swallows the empty body into "The model returned an empty
response" and marks the run success, so nothing surfaces the failure.

Tasks that omit an endpoint dodge this only because _resolve_defaults()
cribs an already-full URL from a recent chat session. The API/token path
(e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1")
hits it every time.

Fix: route every resolved task endpoint through _normalize_chat_endpoint()
at the three resolution sites (_execute_llm_task, the persona/research
session path, and _execute_research_task). The helper is idempotent
(strips any existing chat suffix, re-appends the correct one) and leaves
native-Ollama (/api...) and already-concrete URLs untouched, so other
providers are unaffected. Proven via isolated repro: ".../v1" -> 404 ->
empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output.

Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency,
and the native-Ollama/empty passthroughs.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-routes): harden _probe_endpoint against malformed model-list responses (#4789)

* fix(model-routes): harden _probe_endpoint against malformed model-list responses

_probe_endpoint parsed model lists with data.get(...) at four sites without
checking that data is a dict, and built the list with a truthiness-only
filter. A /models (or /api/tags) endpoint returning HTTP 200 with valid but
non-dict JSON ([], "x", null, 123) made data.get(...) raise AttributeError,
and a non-string id like 123 passed the filter and then hit .startswith() /
.lower() in the Z.AI/Kimi curated merge and _is_chat_model(). Both errors are
swallowed by the broad except Exception, but the comprehension dies mid-list
so the ENTIRE probed model list is discarded and the endpoint silently
degrades — masking a misconfigured/non-compliant upstream as "no models".

- Guard each data.get(...) with isinstance(data, dict) so a non-dict body
  falls through the existing `or []` default.
- Restrict the OpenAI and Ollama model-list comprehensions to non-empty str
  values, protecting the .startswith() merges and both _is_chat_model calls.
- Add an isinstance guard at the top of _is_chat_model (defense in depth for
  all four call sites).

No behavior change for well-formed {"data":[...]} / {"models":[...]}
responses. Adds regression tests (non-dict body via caplog, mixed/all
non-string ids, _is_chat_model boundary) that fail before the fix and pass
after.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-routes): extract _openai_model_ids / _ollama_model_names helpers

Per review on #4789: the malformed-response guards were inlined four times in
_probe_endpoint (two OpenAI-id comprehensions, two Ollama-name comprehensions).
Pull each into a small, directly-testable helper so the security-relevant
parsing lives in one place and a future malformed-shape fix doesn't have to be
applied in four spots (CONTRIBUTING flags repeated logic for this reason).

Behavior is unchanged. Adds direct unit tests for both helpers (non-dict body,
non-string ids, non-dict entries, name>model precedence).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cookbook): only block model launch on real port collisions (#4760)

* Fix #4507: only block model launch on real port collisions

Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so
every launch collided. Both pre-launch guards (serve panel + quick-run)
were count-based and fired regardless of port.

- quick-run now auto-assigns a free port (8080 for llama.cpp)
- both guards parse the new port and only prompt on a real overlap,
  stopping only the colliding serve
- dialog reports the actual port instead of a hardcoded 8000

* refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port

Addresses review on #4760:
- _taskPort regex now matches --port= as well as --port (space)
- _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex
- quick-run llama.cpp no longer pins 8080, so two can run concurrently

* fix(cookbook): _taskPort also parses -p; add port-parsing tests

Addresses review on #4760:
- _taskPort now matches -p <n> too, so it's the complete single reader
  (was missing the short flag that other readers already handle)
- add tests/test_cookbook_port_parsing_js.py covering the port forms,
  shared-reader reuse, and llama.cpp auto-assign

* test(cookbook): extract pure port helpers and test behavior

Addresses review on #4760: the prior tests only asserted source strings.
- extract portOf() and nextFreePort() into static/js/cookbookPorts.js
- cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate
- tests run the helpers via node and assert real behavior: all port forms
  (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the
  same-port-clash / different-port-coexist outcome

---------

Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(ui): route tasks.js + skills.js dropdowns through topPortalZ() (#4768)

Fixes #4767. #4724 routed 16 body-portaled dropdowns through the shared
topPortalZ() helper so they always render just above the currently-raised tool
modal, but two were missed and still used a hardcoded z-index, so they hit the
same #4720 bug once a modal's bring-to-front counter climbed past the literal:

  - tasks.js _showTaskDropdown(): inline z-index:100000 on .task-dropdown
  - skills.js kebab menu (.skill-kebab-menu): z-index:100002 in style.css

Both now set zIndex from topPortalZ() after they are appended to the body,
matching the other migrated sites. The dead CSS z-index on .skill-kebab-menu is
removed (the inline value always wins). test_portal_dropdown_z_js.py gains a
source guard asserting both files use topPortalZ() and that no hardcoded
100000/100002 portal literal survives in either file or style.css.

* do_list_models in ai_interaction.py dropped

---------

Co-authored-by: Max Hsu <maxmilian@users.noreply.github.com>
Co-authored-by: aubrey <kyuhex@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ahmed Dlshad <ahmed.dlshad.m@gmail.com>
Co-authored-by: Joel Alejandro Escareño Fernández <52678667+TheAlexz@users.noreply.github.com>
Co-authored-by: Kalin Stoyanov <kgs.void@gmail.com>
Co-authored-by: Pedro Barbosa <devpedrobarbosa@gmail.com>
Co-authored-by: Solanki Sumit <125974181+YAMRAJ13y@users.noreply.github.com>
Co-authored-by: Rudra Sarker <78224940+rudra496@users.noreply.github.com>
Co-authored-by: Skoh <101289702+SkohTV@users.noreply.github.com>
Co-authored-by: Jakub Grula <ramsters110@gmail.com>
Co-authored-by: Dividesbyzer0 <54127744+zoomdbz@users.noreply.github.com>
Co-authored-by: Kenny Van de Maele <kenny@kvandemaele.be>
Co-authored-by: Magiomakes <114195802+Magiomakes@users.noreply.github.com>
Co-authored-by: Samy <12219635+touzenesmy@users.noreply.github.com>
Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(upload): remove trailing whitespace

* fix(security): redact credential-bearing URLs and PII from logs (#4750)

* fix(security): redact credential-bearing URLs and PII from logs

Several log statements emitted sensitive data in clear text:

- model_routes / chat_routes / contacts_routes logged endpoint URLs raw.
  Admin-configured URLs can embed credentials in userinfo or query
  (e.g. https://user:pass@host, ?api_key=...). Route them through a
  shared core.log_safety.redact_url() that drops userinfo/query/fragment.
- note_routes / task_scheduler logged operator email addresses (smtp_user,
  recipient). Replaced with presence booleans, which keeps the diagnostic
  ("why didn't this send") without writing PII to logs.

model_routes already had a local redactor on its HTTPStatusError branch;
the generic except branch was missed, so reuse the existing helper there.

Clears CodeQL py/clear-text-logging-sensitive-data alerts 264, 317, 324,
325, 343, 344, 528.

* fix(security): re-bracket IPv6 hosts and single-source the URL redactor

Address review on #4750:
- redact_url now re-brackets IPv6 literals so host:port stays
  unambiguous (https://[2001:db8::1]:8443/v1, not the bracket-less
  ambiguous form).
- point model_routes._redact_url_for_log at the shared helper so the
  two redactors are single-sourced (also picks up the IPv6 fix).

* fix(chat): honor explicit web search denial

* fix(chat): require explicit web search enable

(cherry picked from commit 54f1d015b578832abbf608e4d367097a1b3def0e)

* fix(tasks): gate cookbook serve task execution (#5235)

* fix(security): scope owner-less email accounts to a mailbox match in route guards (#5238)

The HTTP email route guard `_assert_owns_account` and the explicit-account_id
path in `_get_email_config` gated cross-tenant access with
`if row.owner and row.owner != owner` -- which skips the check entirely when the
account row is owner-less (owner NULL or ""). `email_accounts` is the one
owner-scoped table left out of the legacy-owner migration backfill
(core/database.py), so such rows persist on multi-user deploys: an account
configured while auth was disabled, or an imported legacy row. Any authenticated
user could then pass that account's id to read/send/update-credentials/delete
another tenant's mailbox and read its decrypted IMAP/SMTP creds.

Both sibling paths already enforce the intended contract -- the same-file
`_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner`
gate (whose comment says it mirrors "the HTTP email route fallback") only expose
an owner-less account when its own mailbox (imap_user / from_address) is the
caller's. Factor that row-level predicate into `_account_visible_to_owner` and
use it in both guards, so owner-less accounts are visible only on a mailbox
match. Owned accounts, the legacy-claim path, and single-user mode (owner == "")
are unchanged.

Complements #5234 (which fixes the same class on the MCP tool layer); this is
the HTTP route layer it does not touch.

* refactor(routes): move contacts domain into routes/contacts/ subpackage (#5227)

Slice 2e of the route-domain reorganization (#4082/#4071, per
specs/architecture-runtime-inventory.md §6.3). Moves contacts_routes.py into
routes/contacts/, leaving a backward-compat sys.modules shim at the old path.
Pure file reorganization, no behavior change.

The shim uses sys.modules replacement (same pattern as the merged gallery
`import routes.contacts_routes`, `from routes.contacts_routes import X`,
`importlib.import_module(...)`, the string-targeted
`monkeypatch.setattr("routes.contacts_routes.SETTINGS_FILE", ...)` used by
test_carddav_password_encryption.py, and the `import ... as cr` +
`setattr(cr, ...)` pattern in test_contacts_add_null_name.py all operate on
the same module object the application uses. This also keeps the mutable
module state `_contact_cache` identical across import paths.

The canonical module does NOT depend on the shim — routes/contacts/
contacts_routes.py imports only from core/, src/, and stdlib (zero internal
routes/ coupling). The inbound edge from routes/email_helpers.py (imports
_fetch_contacts) keeps working through the shim.

Zero source-introspection landmines — no test reads this file by path.

Adds tests/test_contacts_routes_shim.py to pin the sys.modules shim contract
(same-object + string-targeted monkeypatch reach-through).

Verified: compileall clean; full suite 4485 passed, 3 skipped.

* fix(security): pin webhook delivery to the SSRF-validated IP (DNS rebinding) (#5147)

validate_webhook_url resolves the host to accept/reject, but the delivery
connect (httpx.AsyncClient.post) re-resolved independently — a DNS record
flipping between the two lookups (rebinding) could slip an internal IP
(127.0.0.1 / 169.254.169.254 / LAN) past the check and receive the signed
payload. The module docstring already flagged this as only a "partial
defense".

Resolve + validate once via _validated_public_ips, then pin the delivery
TCP connect to that approved IP with an async _PinnedAsyncTransport built
on the public httpcore/httpx APIs (mirrors the sync search-fetch pin from
validation and vhost routing still target the original hostname; only the
socket destination is pinned.

Delivery now uses a per-request pinned client instead of one shared client,
so close() is a no-op kept for API compatibility. Adds end-to-end tests that
drive the real transport against loopback servers, proving the connect
follows the pin rather than re-resolving the URL host.

Fixes #5146

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(security): apply sensitive-file deny-list to grep tool (#5011) (#5013)

The grep tool bypassed the sensitive-file deny-list that read_file,
write_file, and edit_file all respect. Two code paths fixed:

1. ripgrep path: adds --glob exclusion patterns for each entry in
   _SENSITIVE_FILE_PATTERNS (id_rsa, known_hosts, authorized_keys, etc.)
2. Pure-Python os.walk fallback: checks _is_sensitive_path() before
   opening each file, skipping files that match the deny-list

Fixes #5011

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix(chat): sanitize web search query to strip markdown and code blocks (#4863)

Layer a defensive cleanup on top of the generated-query web-search flow so the final selected query is sanitized before reaching comprehensive_web_search.

- remove fenced code blocks from the final search query
- preserve inline code as plain text
- collapse whitespace and cap query length
- cover generated-query success plus LLM failure/empty fallback paths

Partially addresses #4547.

* fix(calendar): honor list_events date range aliases (#3283)

* fix(calendar): honor list_events date range aliases

* fix(calendar): reject partially resolved loose range queries

* fix(integrations): don't append a trailing slash when api_call path is '/'

_join_integration_url built urljoin(base + '/', '') for a bare '/'
path — the minimum execute_api_call accepts — so every request against
a POST-to-base integration went to base_url + '/'. Discord webhook
URLs 404 ('Unknown Webhook') on the trailing-slash variant, which made
the integration look broken even though the stored base URL was
correct.

Resolve a bare '/' (or empty) path to the base URL itself and keep all
other paths joining exactly as before, including deliberate trailing
slashes inside non-empty paths (linkding /api/tags/, Home Assistant
/api/). The reminder webhook sender and the discord_webhook
connectivity test already posted to the bare base URL; execute_api_call
was the remaining path that re-added the slash.

Fixes #5138

* test: split service health tests (#4972)

* test: split service health tests

* test(service-health): preserve focus selector

* fix(tools): handle non-dict JSON values in _parse_tool_args (closes #5043) (#5064)

When an LLM generates a valid JSON string that parses to a native non-dict
type (like a list, int, or string), _parse_tool_args previously returned
that object. Callers expecting a dictionary would then crash with
AttributeError or KeyError when attempting to look up action keys.

- Update _parse_tool_args in src/tool_utils.py to explicitly type-check
  the parsed JSON object and return {} for non-dict objects.
- Add test coverage in tests/test_admin_tools_registry.py for lists,
  ints, and strings.

* test(integrations): drop redundant trailing-slash assertion

The exact-equality assert on the line above (requested_url == WEBHOOK_BASE)
already implies the URL has no trailing slash, so the endswith check adds
nothing.

* Move core serve memory fields further left

* Move vllm attention farther right

* Color cookbook context fit notes

* fix(parser): parse Gemma 3/4 custom tool calling tokens (#5033)

* fix: parse Gemma 3/4 custom tool calling tokens in parser

* test: cover Gemma tool call parsing

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix(agent): preserve bare email tool parity (#5075)

* fix(agent): confine glob literal lookups to the search root (#5010)

GlobTool resolves its search root through _resolve_search_root (which
confines it to the workspace or default allowlist), but the literal
fast-path joined the model-supplied pattern onto that root without
re-confining it. os.path.join lets an absolute pattern or one containing
../ escape the root, and normpath collapsed the .. segments, so glob
returned the absolute path of arbitrary host files once they existed --
an existence/path oracle that bypasses the confinement read_file,
write_file, grep, and ls all enforce.

Keep the literal lookup inside the root via a commonpath containment
check; an escaping literal falls through to the os.walk matcher, which
only ever yields paths under the root. Wildcard matching was already
confined.

* fix(agent): skip deny-listed sensitive files in glob (#5094)

* fix(llm): normalize OpenAI-compatible chat URLs

Normalize OpenAI-compatible chat URL shapes so base /v1 endpoints route to /v1/chat/completions while already-full chat endpoints remain idempotent.

Preserve native local Ollama routing for bare localhost:11434 endpoints, keep localhost:11434/v1 as OpenAI-compatible, and add focused regression coverage for provider detection, chat target URLs, and model listing from /v1.

Part of #541.

* fix(search): use generated query for chat mode web search #4547 (#4557)

* fix(search): use generated query for chat mode web search #4547

* style(search): tidy query generation call

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix(email): validate IMAP/SMTP ports instead of crashing with 500 (#4464)

The email-account endpoints coerced user-supplied ports with a bare int(data.get("imap_port") or 993), so a non-numeric port (e.g. "imap") raised ValueError and surfaced as an HTTP 500 in the create, update, and test-config endpoints.

Add a _coerce_port(value, default) -> (port, error) helper and use it in all three endpoints, returning the endpoints standard {"ok": False, "error": ...} response (matching the existing "name required" validation) instead of crashing. A blank or missing port still falls back to the default (993/465).

* fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236,
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix

* Support mobile enter for queued agent prompts

* Keep open document context for section edits

* Add bulk email attachment downloads

* Add hover labels to mini sidebar buttons

* fix(cookbook): treat local Windows as Windows for serve commands (#3975)

* fix(cookbook): prefer native llama-server on local Windows

* fix(cookbook): harden local llama-server launch commands

* fix(cookbook): build serve commands for selected target

* fix(cookbook): only block model launch on real port collisions (#4760)

* Fix #4507: only block model launch on real port collisions

Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so
every launch collided. Both pre-launch guards (serve panel + quick-run)
were count-based and fired regardless of port.

- quick-run now auto-assigns a free port (8080 for llama.cpp)
- both guards parse the new port and only prompt on a real overlap,
  stopping only the colliding serve
- dialog reports the actual port instead of a hardcoded 8000

* refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port

Addresses review on #4760:
- _taskPort regex now matches --port= as well as --port (space)
- _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex
- quick-run llama.cpp no longer pins 8080, so two can run concurrently

* fix(cookbook): _taskPort also parses -p; add port-parsing tests

Addresses review on #4760:
- _taskPort now matches -p <n> too, so it's the complete single reader
  (was missing the short flag that other readers already handle)
- add tests/test_cookbook_port_parsing_js.py covering the port forms,
  shared-reader reuse, and llama.cpp auto-assign

* test(cookbook): extract pure port helpers and test behavior

Addresses review on #4760: the prior tests only asserted source strings.
- extract portOf() and nextFreePort() into static/js/cookbookPorts.js
- cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate
- tests run the helpers via node and assert real behavior: all port forms
  (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the
  same-port-clash / different-port-coexist outcome

---------

Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(agent): execute fenced tool calls with inline args and route bare email tool names (#3681)

* fix(agent): execute fenced tool calls with inline args and bare email tool names

Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:

1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
   tool call with args on the same line ("```list_email_accounts {}")
…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants