feat(security): support optional encrypted-at-rest secrets via SOPS (re #233)#236
feat(security): support optional encrypted-at-rest secrets via SOPS (re #233)#236nickorlabs wants to merge 1 commit into
Conversation
1b9f506 to
9d4c8e2
Compare
End-to-end validation on a Linux/Docker serverStood the full stack up on an Ubuntu 24.04 / Docker 29.4.3 host using this branch — closing the maintainer-checkbox in the test plan. Setup performed (matches the SECURITY.md workflow exactly)git clone <fork> && git checkout feat/sops-encrypted-secrets
age-keygen -o ~/.config/sops/age/keys.txt && chmod 600 ~/.config/sops/age/keys.txt
# Paste the public key into .sops.yaml
cp secrets.env.example secrets.env
$EDITOR secrets.env # added ODYSSEUS_TEST_SECRET=hello-from-sops-2026
sops -e -i secrets.env # encrypt in place
docker compose up -d --build
Six gates, all green1. PID 1 inside the container is 2. Sentinel secret IS in uvicorn's process environment (read via 3. 4. Plaintext value of the sentinel appears NOWHERE on the container filesystem 5. Secrets are scope-isolated to uvicorn's child tree — a fresh This is a nice bonus property: an attacker who lands a container exec shell after boot can't lift the SOPS-injected env from 6. The app boots and serves normally Compatibility note — re: #511 (Secure by default uplift, merged today)Checked: zero file overlap between this PR and #511. The PR currently reports
They cover complementary surfaces. Ready for review whenever convenient. Happy to rebase or adjust scope (e.g. gate |
9d4c8e2 to
ff6ae71
Compare
Rebased on current
|
ff6ae71 to
9e53b1e
Compare
Rebased on current
|
74e8a87 to
298e3fb
Compare
a439e06 to
843b2be
Compare
|
Thanks for the detailed write-up and validation. I did a focused local audit. From the local checks, I did not find a concrete merge blocker:
That said, I do not think I should approve or merge this without maintainer/security direction from @pewdiepie-archdaemon. This changes deployment and secret-management policy, including Docker image contents, entrypoint behavior, backup/key-management expectations, and the intended workflow around committable encrypted A few points I would want maintainer confirmation on before merge:
Technically promising, but I’d leave final approval/merge to @pewdiepie-archdaemon because this is security/deployment policy, not just implementation correctness. Best regards, |
843b2be to
a7cf1e1
Compare
|
Quick workflow question now that My read is |
|
Thanks for asking. Yes, I think this should target The PR metadata already shows Could you please rebuild/rebase from current Expected final scope:
Also please update the PR body checklist from Once that is clean, I can re-review the actual SOPS/security-policy questions separately. |
a7cf1e1 to
5d2a78e
Compare
|
Retargeted to Branch HEAD: |
|
Thanks, the rebase/scope issue is resolved now. The branch is clean against I still have one blocker before merge: the Dockerfile downloads the SOPS release binary with Since this PR adds a secrets/deployment path, I think the install path should verify the downloaded binary in the same PR. Once that is added, this should be straightforward to approve. |
|
Thanks will look at this later today and make updates. |
5d2a78e to
9fd4184
Compare
|
Thanks for the re-scope. Most of the previous security/deployment concerns look resolved now: checksum verification is present, the mixed Compose workflow is removed, Docker-only scope is explicit, SEARXNG_SECRET is out of scope, and the internal token name is now ODYSSEUS_INTERNAL_TOKEN. I found one remaining small blocker before approval: .sops.yaml still says users can copy the config to .sops.yaml.local, but the documented encryption command is plain sops -e -i secrets.env. SOPS will discover .sops.yaml by default, not .sops.yaml.local, unless users pass an explicit config path. This reintroduces the config-discovery confusion from the earlier review. Could you remove the .sops.yaml.local suggestion and keep the guidance aligned with editing .sops.yaml in place? |
1853f71 to
7ff17f8
Compare
RaresKeY
left a comment
There was a problem hiding this comment.
Current-head follow-up review. This is not replacing or restating the earlier feedback: the earlier .sops.yaml.local config-discovery blocker looks fixed on this head, so I’m only adding the non-duplicative issues I found on the current head.
Findings
issue (runtime): Keep the app as the Docker stop signal target in SOPS mode
-
Problem: The non-SOPS path ends by
execing the app throughgosu, so Docker stop signals reach the application process directly. The SOPS path insteadexecsgosu ... sops exec-env ..., and SOPS then starts the final command in its default child-process mode. The shell's finalexec$cmdreplaces only SOPS' shell child, not the SOPS process that Docker targets as PID 1. -
Impact: SOPS-enabled containers can skip the normal graceful shutdown path. In a SOPS 3.13.1 probe with the current command shape, plain
exec-envleft the final child running after SIGTERM, while the same command with--same-processmade the child own the SOPS PID and receive SIGTERM. -
Ask: Preserve the no-SOPS signal contract in the SOPS branch, for example by using
sops exec-env --same-process "$SECRETS_FILE" "python /app/setup.py || true; exec$cmd"or an equivalent structure where the final app process is the signal target. -
Location:
docker/entrypoint.sh:119
issue (docs): Quote the native
exec-env command as one shell command string
-
Problem:
SECURITY.mdtells native users to wrap startup withsops exec-env secrets.env -- uvicorn app:app ..., but SOPS v3.13.1exec-envtakes exactly two positional arguments and runs the second as a shell command string. It is not argv passthrough. -
Impact: Native users following the documented workaround fail before Odysseus starts, which undermines the Docker-only scope guidance added by this PR.
-
Ask: Change the example to a single quoted command string, such as
sops exec-env secrets.env 'uvicorn app:app ...', or otherwise document the exact SOPS-supported syntax. -
Location:
SECURITY.md:30
Validation
Ran:
git diff --check origin/dev...HEADsh -n docker/entrypoint.shdocker compose --env-file /dev/null config- verified SOPS v3.13.1
exec-envparser behavior - reproduced the SOPS
exec-envsignal behavior with a throwaway encrypted dotenv
Not run:
- Full Docker build.
- Live container boot with mounted encrypted
secrets.envand age key. - Full local pytest.
|
@alteixeira20 — good catch, thanks. That was a genuine miss on my end: Pushed # SOPS encryption configuration for Odysseus secrets at rest.
#
# Replace the placeholder below with your own age public key, then
# encrypt with: sops -e -i secrets.envNo more |
alteixeira20
left a comment
There was a problem hiding this comment.
Thanks, the .sops.yaml.local issue is fixed.
I still would not approve this yet because two current-head issues remain:
-
In
docker/entrypoint.sh, please preserve the normal Docker stop-signal/shutdown contract in SOPS mode. The currentsops exec-envpath still leaves SOPS as the process supervising the app command instead of replacing itself with the final app process. Please usesops exec-env --same-processor an equivalent structure so the final Odysseus process receives Docker stop signals the same way it does in the non-SOPS path. -
In
SECURITY.md, please update the native wrapper example.sops exec-envexpects the secrets file plus one command string, not argv-style passthrough. The documented example should quote the command, for example:sops exec-env secrets.env 'uvicorn app:app --host 0.0.0.0 --port 7000'
Once those are fixed, I can re-review the narrowed SOPS-only diff.
7ff17f8 to
48cae76
Compare
|
Thanks @RaresKeY. Both findings fixed on N6 (P1 — SIGTERM not reaching the app in SOPS mode)Added exec gosu "$PUID:$PGID" sops exec-env --same-process "$SECRETS_FILE" \
"python /app/setup.py || true; exec$cmd"Verified end-to-end in a throwaway container with a real encrypted
N7 (P3 — wrong native
|
Adds opt-in SOPS support so deployers who want it can encrypt true
secrets (OPENAI_API_KEY, ODYSSEUS_ADMIN_PASSWORD, ODYSSEUS_INTERNAL_TOKEN,
MCP OAuth secrets, IMAP passwords, etc.) at rest instead of leaving them
plaintext in .env. Non-secret config (APP_PORT, LLM_HOST, etc.) stays
in .env so PR diffs remain reviewable.
Uses SOPS's idiomatic in-place encryption convention: secrets.env is
encrypted in place with 'sops -e -i secrets.env' and the encrypted file
itself is safe to commit. To edit later: 'sops secrets.env' opens
$EDITOR with decrypted contents and re-encrypts on save.
The feature is only active when /app/secrets.env is present at
container start; existing deployments behave exactly as before. If
secrets.env is present but NOT SOPS-encrypted (detected via the
sops_version marker), the entrypoint refuses to start — that shape is
almost always a packaging mistake.
Scope: the hook lives in docker/entrypoint.sh, so this only wraps the
Docker entrypoint path. Native startups (uvicorn directly, systemd,
launchd) read process env / .env as usual. Cross-service secrets like
SEARXNG_SECRET (consumed by a separate container via compose
interpolation) are also out of scope here — tracked as a follow-up.
Build-context and .gitignore hardening for plaintext/decrypt artifacts
ships as a small separate PR so it can land independently.
- Dockerfile: pin sops v3.13.1 (~10MB static Go binary), arch-aware
via dpkg, per-arch SHA256 verified before chmod +x so a CDN
substitution can't ship an unverified binary
- docker/entrypoint.sh: wrap gosu with 'sops exec-env' when encrypted
secrets.env exists; setup.py runs INSIDE the wrap so the encrypted
ODYSSEUS_ADMIN_PASSWORD reaches the admin-seed step; fail loudly if
sops is missing, no SOPS_AGE_KEY[_FILE] is set, or the file is
plaintext
- .sops.yaml: creation rule with placeholder age public key
- secrets.env.example: documents the in-place encryption workflow and
the app-internal scope
- SECURITY.md: new 'Encrypting Secrets At Rest' section with the
age-keygen -> 'sops -e -i' -> compose-mount workflow and scope notes
Verified end-to-end with the entrypoint exec-ing a sentinel-loaded
container:
positive (encrypted file mounted, key bound) -> sentinel in env,
setup.py sees seed
negative (no secrets.env) -> entrypoint unchanged
refuse (plaintext secrets.env present) -> exit 1 with hint
Proposed in odysseus-dev#233. Naming chosen to avoid colliding with the existing
runtime Bitwarden integration at routes/vault_routes.py.
48cae76 to
8b5d229
Compare
|
@alteixeira20 — the two concerns in your latest review are exactly the same as @RaresKeY's N6 + N7, which I'd already fixed in the prior head Rebased onto current exec "$GOSU_BIN" "$PUID:$PGID" sops exec-env --same-process "$SECRETS_FILE" \
"$PYTHON_BIN /app/setup.py || true; exec$cmd"Re-ran the same end-to-end signal + decryption test against the rebased image: So: PID 1 = the final shell (running as
Diff stat unchanged at 5 files / +177 / -3. Ready for re-review when you have time. |
|
@alteixeira20 @RaresKeY — when you have a moment, could you take another look? Both of your "Request Changes" reviews above are against commit
No further code changes pending — happy to address anything new the latest head surfaces, but the two items you flagged should both be resolved now. |
|
Sorry for the delay, tomorrow I'll take a couple of hours to get up to speed, I'm not home at the moment |
|
@alteixeira20 @RaresKeY — when you have a moment, could you take another look? The current head
No further changes are pending from my side — happy to address anything new the latest head surfaces. Thanks! |
|
Thanks for the updates. Before I re-review this, could you please refresh/rebase the branch against current The visible scope still looks SOPS-only ( |
* fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236, #524. The _XML_OPEN_TOOL_CALL_RE alerts (#234, #477) are false positives (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix
…us-dev#4704) * fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236, odysseus-dev#524. The _XML_OPEN_TOOL_CALL_RE alerts (odysseus-dev#234, odysseus-dev#477) are false positives (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix
…us-dev#4704) * fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236, (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix
…us-dev#4704) * fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236, (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix (cherry picked from commit c098355)
…us-dev#4704) * fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236, odysseus-dev#524. The _XML_OPEN_TOOL_CALL_RE alerts (odysseus-dev#234, odysseus-dev#477) are false positives (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix
…us-dev#4704) * fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236, (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix
… resolve code quality items (#1) * Code quality: security, deps, error handling, tests, frontend, database Phase 0 — Security (5/5): - Fix bare except in require_admin() (middleware.py) - Replace shell=True with list-form subprocess (builtin_actions.py) - Add .catch() to 3 dynamic imports (app.js) - Encrypt Webhook.secret with EncryptedText - Verify XSS safety in chat.js Phase 1 — Deps/Config (7/7): - Pin all 28 deps; move test deps to requirements-dev.txt - Set asyncio_loop_scope = module, trim CORS whitespace - Pin Docker to python:3.13-slim - Replace __import__ with standard imports Phase 2 — Error Handling (3/3): - Create safe_create_task() helper; fix 13 fire-and-forget tasks - Lock _response_cache with threading.Lock() - Narrow bare excepts in subprocess_tools, builtin_actions, agent_loop, task_scheduler (24 instances fixed) Phase 3 — Tests (2/2): - Fix test_session_manager explicit assertion - Remove src.database MagicMock fallback in conftest.py Phase 4 — Frontend (5/5): - Scope MutationObservers to #app; add beforeunload cleanup - Create shared API_BASE module (14 files updated) - Fix admin.js undefined API_BASE bug - Fix chatRenderer.js window.API_BASE fallback - Fix _defaultChat race, rename guard in app.js Phase 5 — Database (3/3): - Defer init_db() from module import to app startup - Add .limit() safety caps to 11 unbounded queries - Extract 26 ORM models to core/orm_models.py (709 lines) - Add index=True to 6 FK columns Phase 6 — Cross-cutting (3/4): - Add Docker socket security warning - Fix dead-host cache: move _clear_host_dead after status check - Cache EncryptedText import to avoid per-call overhead - Add -> None to note_model_activity * making ulises multilang * finished multilang base & tests * fix(security): scope owner-less email accounts to a mailbox match in route guards (#5238) The HTTP email route guard `_assert_owns_account` and the explicit-account_id path in `_get_email_config` gated cross-tenant access with `if row.owner and row.owner != owner` -- which skips the check entirely when the account row is owner-less (owner NULL or ""). `email_accounts` is the one owner-scoped table left out of the legacy-owner migration backfill (core/database.py), so such rows persist on multi-user deploys: an account configured while auth was disabled, or an imported legacy row. Any authenticated user could then pass that account's id to read/send/update-credentials/delete another tenant's mailbox and read its decrypted IMAP/SMTP creds. Both sibling paths already enforce the intended contract -- the same-file `_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner` gate (whose comment says it mirrors "the HTTP email route fallback") only expose an owner-less account when its own mailbox (imap_user / from_address) is the caller's. Factor that row-level predicate into `_account_visible_to_owner` and use it in both guards, so owner-less accounts are visible only on a mailbox match. Owned accounts, the legacy-claim path, and single-user mode (owner == "") are unchanged. Complements #5234 (which fixes the same class on the MCP tool layer); this is the HTTP route layer it does not touch. * fix(agent): fall back to keyword tool selection when retrieval times out The retrieval-timeout branch hard-coded ALWAYS_AVAILABLE, silently skipping the deterministic keyword hints whenever the embedding backend was slow (e.g. a remote endpoint cold-loading its model). Queries that named email or calendar outright lost those tools and the model concluded the integrations did not exist. Let the timeout fall through to the existing keyword fallback instead — same baseline, plus the hints. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(search): pin httpx connection to resolved IP to block DNS rebinding (#704) * fix(search): pin DNS-validated fetch connections Rebase the DNS-rebinding SSRF fix onto current dev after search content moved behind the services.search.content canonical module. Integrate the pinned httpcore NetworkBackend/BaseTransport approach with the current size-capped Client.stream fetch path, preserving Host/SNI semantics while forcing TCP connect to the already validated resolved IP. Keep src.search.content as the compatibility wrapper and preserve existing OG-image http(s) behavior; this avoids reintroducing the unrelated scope changes that previously blocked review. Add the explicit httpcore>=1.0,<2.0 requirement used by the public httpcore NetworkBackend and ConnectionPool APIs. * test(search): restore and rebase DNS rebinding regressions Keep the current security regression coverage that the stale PR branch had deleted, including auth-disabled localhost bypass and Ollama cookbook hardening tests. Carry forward the DNS-rebinding coverage for private resolve blocking, pinned TCP connect behavior, Host header preservation, redirect revalidation, and the BaseTransport/public-httpcore static guard. Update redirect tests to mock the current Client.stream-based capped fetch path rather than the older httpx.stream/get path. * test(search): adapt size-cap fetch tests to pinned client stream The DNS-rebinding repair moved _get_public_url from the module-level httpx.stream shortcut to httpx.Client(...).stream(...) so the fetch can use the pinned transport. Keep the existing size-cap test fakes by routing Client.stream through the monkeypatched httpx.stream only when a test has installed that fake; otherwise fall back to a real Client. This fixes the CI failures in tests/test_web_fetch_size_caps.py without touching unrelated upload-handler atomicity behavior, which is already flaky on clean origin/dev. --------- Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com> * fix: resolve RAG manager search signature TypeError (#4994) * fix: resolve RAG manager search signature TypeError and adjust similarity threshold * fix: revert similarity threshold change to keep PR focused * test(rag): remove trailing whitespace --------- Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com> * fix(security): scope send_to_session to an exact session owner * fix(tools): handle non-dict JSON values in _parse_tool_args (closes #5043) (#5064) * fix(search): extract non-ASCII capitalized names in _extract_entities * fix(agent): skip deny-listed sensitive files in glob (#5094) * fix(security): match the sensitive-file deny-list case-insensitively (#5097) * fix(integrations): don't append a trailing slash when api_call path is '/' * fix(calendar): honor list_events date range aliases (#3283) * fix: auto-spam move/delete targets the wrong message (seqnum vs UID) (#1874) * fix(session): use utcnow_naive across session routes (#1116) (#5003) * fix(cookbook): stop Ollama runner from executing the install one-liner (#3926) * fix(security): apply the webhook SSRF guard to the reminder ntfy sender (#5142) * fix(security): pin webhook delivery to the SSRF-validated IP (DNS rebinding) (#5147) * fix(security): validate integration api_call URLs with the outbound SSRF guard (#5145) * fix(security): match grep's rg sensitive-file exclusions case-insensitively (#5189) * Fix Cookbook download runner for Python 3.11 * fix: add grace period to document tidy to prevent deleting new documents (#5036) * fix(email): enforce MCP account owner scope (#5234) * fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704) * fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619) * fix(security): prevent ReDoS in XML and args tool-call parsers (#4941) * fix(chat): sanitize web search query to strip markdown and code blocks (#4863) * fix(security): batch A — 7 security fixes * fix(bugfix): batch B — 16 critical bugfixes * fix(chat): batch C — web search denial + explicit intent * test(ci): batch D — test splits + CI guidance * feat(refactor): batch E — features, refactors, catalog * fix(infra): batch F — 16 infrastructure + bugfixes * feat(docs): batch G — models, documents, docs updates * fix(ux): batch H — email, gallery, cookbook UX fixes * feat(models): batch I — models hardening + OCR captions * ui: batch J1 — cookbook/llama model UI spacing * ui: batch J2 — cookbook engine tint, MTP, vllm spacing * feat(background): batch J3 — background tasks, CLI sanitization, info-budget fixes * chore(release): bump version to 1.0.2 * fix(ui): lower cookbook serve top row (cherry-pick fc6a6dc) * fix(chat): honor explicit web search denial * fix(mobile): stack the model-comparison grid into one column on phones (#4979) The comparison grid hard-codes 2-4 equal columns with no phone breakpoint, so at 390px two models get ~178px columns and four get ~88px columns. Each column is a full scrolling chat, so content is unreadably over-wrapped and clipped. On phones (<=768px), stack the panes into a single scrollable column. Desktop is unaffected. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(docs): correct broken backup-restore link in setup.md Fixes #4926 - the link used docs/backup-restore.md from within docs/setup.md, which resolved to docs/docs/backup-restore.md (404). Changed to same-directory relative path. * docs(setup): add a self-host troubleshooting cookbook of common traps (#4834) ROADMAP "Self-host troubleshooting cookbook" asks to document the weird 30-second fixes that otherwise become 30-minute searches. Adds a "Common self-host traps" subsection under Troubleshooting covering: the UTF-8 BOM .env gotcha (app.py loads with utf-8-sig), macOS AirPlay holding port 7000 (the start script uses 7860), the plain-HTTP Tailscale/LAN clipboard limitation, self-hosted ntfy delivery (NTFY_BIND/NTFY_BASE_URL + the ntfy Android Instant-delivery toggle), Dovecot cleartext-auth on LAN mail stacks, and Radicale full-collection-URL sync. Docs only; grounded in existing repo behavior (.env.example NTFY_* block, app.py utf-8-sig loader, start-macos.sh port choice). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619) Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike the chat path which stores build_chat_url(normalize_base(base)) on the session. A task carrying an explicit bare OpenAI-compatible base such as "http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the agent loop swallows the empty body into "The model returned an empty response" and marks the run success, so nothing surfaces the failure. Tasks that omit an endpoint dodge this only because _resolve_defaults() cribs an already-full URL from a recent chat session. The API/token path (e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1") hits it every time. Fix: route every resolved task endpoint through _normalize_chat_endpoint() at the three resolution sites (_execute_llm_task, the persona/research session path, and _execute_research_task). The helper is idempotent (strips any existing chat suffix, re-appends the correct one) and leaves native-Ollama (/api...) and already-concrete URLs untouched, so other providers are unaffected. Proven via isolated repro: ".../v1" -> 404 -> empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output. Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency, and the native-Ollama/empty passthroughs. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787) setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv() but never loaded .env, so on native Linux/macOS installs a password pre-seeded in .env (documented in docs/setup.md and .env.example) was silently ignored and a random one generated, breaking the first login. Docker was unaffected because compose passes the vars into the container env. Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(), mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does not override already-exported OS vars, so the existing precedence is kept. python-dotenv is already a required dependency. Adds a regression test that pre-seeds credentials only in .env (not the shell) and asserts the stored bcrypt hash matches the pre-seeded password. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(llm): detect mistral.ai provider and support reasoning_effort (#4698) * fix(llm): detect mistral.ai provider and support reasoning_effort Four coupled bugs broke Mistral thinking model support: 1. _detect_provider() had no mistral.ai host check, so all Mistral endpoints fell through to the generic 'openai' provider string. _provider_display_name() correctly identified them as 'Mistral', making any 'if provider == "Mistral"' check elsewhere dead code. 2. reasoning_effort parameter was never sent in the request payload, so Mistral never activated thinking mode even when the user configured a thinking-capable model (mistral-small-latest, mistral-medium-latest, magistral-*). 3. Mistral returns content as a typed array ([{"type":"thinking",...},{"type":"text",...}]) when reasoning is on, not as a plain string. Both the streaming and non-streaming parsers expected strings and silently dropped the thinking content. 4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-* model prefixes, so the frontend wouldn't tag reasoning output as thinking even after the above were fixed. Fix: - Add mistral.ai to _detect_provider() host checks - Add a _normalize_mistral_content() helper that splits the typed array into (text, thinking) strings - Inject payload["reasoning_effort"] = "high" when provider is Mistral and _supports_thinking(model) is true, in both stream_llm and llm_call_async payload construction - Wire the normalizer into both response parsers - Extend _THINKING_MODEL_PATTERNS to include magistral, mistral-small, mistral-medium, mistral-large Tested on Docker install with mistral-small-latest + reasoning_effort=high. Reasoning streams correctly into the thinking panel after the fix. Fixes #4678 * fix(llm): address review — lowercase provider id, configurable effort, tests Addresses vdmkenny's review on PR #4698: 1. Removed duplicate 'if provider == "mistral"' block in stream_llm — two back-to-back copies, one was dead-redundant. 2. Dropped personal-context comment ('free-tier limits are generous for this user') and made reasoning_effort configurable via env var ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none). Default remains 'high' for backward compat with the tested behavior. 3. Recased provider id from 'Mistral' to 'mistral' to match the lowercase convention used by every other provider id in the file (openai, anthropic, ollama, copilot, ...). _provider_display_name() still returns the Title-Case 'Mistral' for UI labels — only the runtime id used in 'if provider == ...' checks was recased. 4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning _normalize_mistral_content()'s contract: string passthrough, the Mistral array format (thinking + text blocks), and edge cases (empty, garbage, None, wrong types, missing fields, string-vs-array inner thinking field). Also fixed a gap the review didn't catch: the non-streaming paths (llm_call sync + llm_call_async) were missing the reasoning_effort injection entirely. Added the same injection to both, so Deep Research and agent tool calls also activate Mistral thinking. All 13 new tests pass. Existing reasoning/streaming/ollama-thinking tests still pass (38 tests, no regressions). Fixes #4678 * fix(chat): restore missing _explicit_web_intent definition (#5290) chat_stream() references `_explicit_web_intent` in three places (disabled-tools gating, global-disabled web allowance, and the per-turn tool filter) but the assignment was dropped during a branch merge. Every chat request raised NameError: name '_explicit_web_intent' is not defined at routes/chat_routes.py, surfacing to the client as a bare "Internal Server Error" before any LLM call was made — chat was fully broken on dev and main. Restore the original definition, computed from the already-derived tool intent, immediately before its first use: _explicit_web_intent = bool(_tool_intent and _tool_intent.category == "web") Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 2d8177035b6a1dd3e9211c6cb5ba68443b826504) * Parse local function_model tool wrappers * Show fallback model in picker * Nudge serve engine control up * Move core serve memory fields farther left * Move vllm block size left again * Nudge vllm attention field right * Raise unified llama context estimate * fix(llm-core): prevent cache-affinity fields from reaching Cerebras Recognize api.cerebras.ai as a Cerebras cloud provider so llama.cpp/LM Studio cache-affinity fields are not attached even when endpoint_kind is misconfigured as local. Add regression coverage for provider detection, self-hosted classification, and payload field exclusion. * fix(models): accept bare-list /models responses (Together AI) (#4761) * fix(api): handle varying response formats for model IDs from compatible providers merge conflict for pr-2204 resolved * fix(modal): keep body-portaled dropdowns above their tool modal at any stack depth (#4720) (#4724) * fix(memory): keep the Brain memory item menu above the modal at any stack depth The memory item "⋮" dropdown is portaled to <body> with a hardcoded z-index of 10001. Tool modals, however, get a monotonically increasing z-index from modalManager's bring-to-front counter (_modalTopZ), which climbs unbounded as modals are opened/restored over a session. Once that counter passes 10001, the Brain modal stacks above the body-portaled dropdown, so the menu renders behind the panel — visible only where it spills past the modal's edge (#4720). Derive the dropdown's z-index from the owning modal's current z-index (+1), keeping 10001 as a floor for the common low-counter case, so the menu always sits just above its modal however high the counter has climbed. Verified with document.elementFromPoint at the dropdown's location: with a high modal z-index the old build returns the modal at every sampled point (menu behind); the fixed build returns the dropdown (menu on top). The default low-counter case is unchanged (z stays 10001). * refactor(modal): route body-portaled dropdowns through a shared topPortalZ() helper The hardcoded z-index:10001 the Brain memory menu used (#4720) is the same literal shared by ~16 body-portaled dropdowns across calendar, cookbook, cookbookServe, documentLibrary, emailLibrary, gallery, notes, emojiPicker and memory — each renders behind its owning tool modal once modalManager's bring-to-front counter climbs past the literal over a long session. Promote the per-dropdown fix into a single topPortalZ() helper in toolWindowZOrder.js — the existing source of truth for tool-window z, already imported by modalManager's _bringToFront and notes.js — returning max(topToolWindowZ(), dock-chip floor) + 1, so a portaled dropdown always sits just above the live tool-window stack however high the counter has climbed. Route all 16 sites through it. The slashCommands tour tooltips and the cookbookServe VRAM dialog are intentionally left out (neither is a modal-owned portaled dropdown). Add tests/test_portal_dropdown_z_js.py covering the helper, including the #4720 scenario (modal counter at 99999 -> dropdown at 100000). Existing test_notes_z_order_js.py stays green. * fix(llm): detect mistral.ai provider and support reasoning_effort (#4698) * fix(llm): detect mistral.ai provider and support reasoning_effort Four coupled bugs broke Mistral thinking model support: 1. _detect_provider() had no mistral.ai host check, so all Mistral endpoints fell through to the generic 'openai' provider string. _provider_display_name() correctly identified them as 'Mistral', making any 'if provider == "Mistral"' check elsewhere dead code. 2. reasoning_effort parameter was never sent in the request payload, so Mistral never activated thinking mode even when the user configured a thinking-capable model (mistral-small-latest, mistral-medium-latest, magistral-*). 3. Mistral returns content as a typed array ([{"type":"thinking",...},{"type":"text",...}]) when reasoning is on, not as a plain string. Both the streaming and non-streaming parsers expected strings and silently dropped the thinking content. 4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-* model prefixes, so the frontend wouldn't tag reasoning output as thinking even after the above were fixed. Fix: - Add mistral.ai to _detect_provider() host checks - Add a _normalize_mistral_content() helper that splits the typed array into (text, thinking) strings - Inject payload["reasoning_effort"] = "high" when provider is Mistral and _supports_thinking(model) is true, in both stream_llm and llm_call_async payload construction - Wire the normalizer into both response parsers - Extend _THINKING_MODEL_PATTERNS to include magistral, mistral-small, mistral-medium, mistral-large Tested on Docker install with mistral-small-latest + reasoning_effort=high. Reasoning streams correctly into the thinking panel after the fix. Fixes #4678 * fix(llm): address review — lowercase provider id, configurable effort, tests Addresses vdmkenny's review on PR #4698: 1. Removed duplicate 'if provider == "mistral"' block in stream_llm — two back-to-back copies, one was dead-redundant. 2. Dropped personal-context comment ('free-tier limits are generous for this user') and made reasoning_effort configurable via env var ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none). Default remains 'high' for backward compat with the tested behavior. 3. Recased provider id from 'Mistral' to 'mistral' to match the lowercase convention used by every other provider id in the file (openai, anthropic, ollama, copilot, ...). _provider_display_name() still returns the Title-Case 'Mistral' for UI labels — only the runtime id used in 'if provider == ...' checks was recased. 4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning _normalize_mistral_content()'s contract: string passthrough, the Mistral array format (thinking + text blocks), and edge cases (empty, garbage, None, wrong types, missing fields, string-vs-array inner thinking field). Also fixed a gap the review didn't catch: the non-streaming paths (llm_call sync + llm_call_async) were missing the reasoning_effort injection entirely. Added the same injection to both, so Deep Research and agent tool calls also activate Mistral thinking. All 13 new tests pass. Existing reasoning/streaming/ollama-thinking tests still pass (38 tests, no regressions). Fixes #4678 * fix: Images cannot be seen by model that is vision capable (#4726) * fix: Images cannot be seen by model that is vision capable * fix: skip http(s) image_url for Ollama (images[] is base64-only) --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix(chat): strip executed email tool fences from the live stream (#3993) (#4275) * fix(chat): strip executed email tool fences from the live stream (#3993) The backend strips every fenced tool block from persisted text (the regex in src/tool_parsing.py is built from the full TOOL_TAGS set, which includes the email tools), so a reloaded session renders cleanly. The live frontend path uses a separate hardcoded EXEC_FENCE_RE in static/js/chatRenderer.js that only listed web_search/read_file/write_file/create_document/edit_document/ update_document — so executed email tool fences (list_emails, etc.) lingered as raw code blocks in the live assistant bubble until the user reloaded. Add the nine email tool tags to EXEC_FENCE_RE so the live render settles into the same clean layout as the history reload. bash/python stay excluded on purpose: those are languages a user may legitimately have asked the model to show as code, not tool invocations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(chat): single-source live exec-fence tool list from TOOL_TAGS (#3993) Per review: EXEC_FENCE_RE was a second, hand-maintained copy of the executable-tool list, so any tool not in it — and every future tool added to TOOL_TAGS — would leave its executed fence lingering in the live bubble until reload (the original #3993 bug, recurring one tool at a time). EXEC_FENCE_RE is now built from an explicit EXEC_TOOL_TAGS list that mirrors TOOL_TAGS (src/agent_tools/__init__.py) minus bash/python, which stay excluded as legitimate code-example languages. A new regression test (test_exec_fence_re_covers_all_executable_tools) extracts both lists from source and fails if they drift, so the whole class is caught in CI instead of by a user — the "minimum acceptable middle ground" from the review, made exact (set equality, not just coverage). Verified: pytest tests/test_live_strip_email_tool_fences.py (5 passed); node --check static/js/chatRenderer.js; and a node run of the built regex confirms email/generate_image/manage_memory/ls fences strip while bash/python/sh are preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(chat): build live exec-fence list from /api/tools at runtime (#3993) Make TOOL_TAGS the single source for live exec-fence stripping. chatRenderer.js no longer hard-codes a tool list; it fetches the backend's authoritative set once from GET /api/tools (sorted(TOOL_TAGS)) and builds EXEC_FENCE_RE from it at load, minus bash/python. No second list to drift, and a future tool added to TOOL_TAGS is covered automatically — without touching the streaming path. Until the fetch resolves EXEC_FENCE_RE is null and exec fences aren't stripped (a sub-second window before the first stream); the backend already strips persisted history, so a reload always renders clean. Drop test_exec_fence_re_covers_all_executable_tools (no hand-maintained list to guard) and add source-level guards: the frontend keeps no hard-coded list and fetches /api/tools, and the endpoint serves the full sorted(TOOL_TAGS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL * fix(chat): warn on /api/tools fetch failure instead of swallowing it (#3993) A fresh-context review flagged that loadExecFenceRegex's catch silently discarded errors: if the one-shot fetch fails, EXEC_FENCE_RE stays null for the whole session and live exec fences go unstripped until reload, with zero signal. console.warn it, and correct the comment to describe the failure mode honestly (was understated as just a sub-second startup window). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): log and cleanly 500 on unreadable HTML page (#4637) * fix(routes): serve 404 instead of 500 when an HTML page file is missing _serve_html_with_nonce opened the HTML file with no error handling, and callers such as /backgrounds and /login pass their paths in with no existence check, so a missing or unreadable file raised an unhandled OSError that surfaced as a 500. Wrap the read and raise HTTPException(404) instead; the normal render path (CSP-nonce substitution) is unchanged. Fixes #4594 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(routes): distinguish missing page (404) from read failure (500) The previous fix caught a broad OSError and returned 404 for every failure, which masks real server-side problems (permission errors, I/O failures) as "not found" and lets them slip past error alerting. Split FileNotFoundError (genuine 404) from other OSError, which now logs the exception and returns a generic 500 — without leaking the OS error string or file path into the response body. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(routes): treat unreadable bundled HTML page as logged 500, not 404 Per PR #4637 review: every caller of the page-render helper serves a fixed, server-owned template (index/login/backgrounds), never a client-supplied path. So a missing or unreadable file is a server fault (broken deployment), not a client "not found" — a 404 there mislabels a server error and hides a missing core template from 5xx alerting, contradicting the OSError->500 rationale this PR is built on. Collapse both branches into a single logged, leak-free 500. Move the helper to src.app_helpers.serve_html_with_nonce so the behavior can be unit-tested without importing the whole app (app.py is the slim orchestrator; the test harness stubs src.database, so importing app in tests is not viable). Add tests pinning missing/unreadable -> 500 (not 404) and nonce injection on the happy path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(catalog): add Gemma 4 12B/QAT entries and RTX 3050 bandwidth (#4728) Add official Gemma 4 12B-it plus QAT-INT4/INT8 catalog entries (with their GGUF sources), QAT quantization support across the quant tables and the prequantized-prefix list, and the missing RTX 3050 / 3050 Ti memory bandwidth so speed estimates stop falling back to the generic cuda value. * fix debugging on windows (#4679) * fix: Real-ESRGAN install + Cookbook deps-panel crash on the Python 3.14 image (#4694) * fix(docker): make Real-ESRGAN installable on the Python 3.14 image realesrgan's deps basicsr/gfpgan/facexlib (unmaintained since 2022) read their version in setup.py via `exec(...); locals()['__version__']`, which raises KeyError on Python 3.13+ — PEP 667 made locals() in a function an independent snapshot that exec() can no longer mutate. That fails the Cookbook "install realesrgan" sdist build on the python:3.14 base. Add a `realesrgan-wheels` builder stage that fetches the pinned sdists, patches get_version() to exec into an explicit namespace dict, and builds wheels; the final stage installs them --no-deps so a later `pip install realesrgan` resolves from wheels instead of rebuilding the broken sdists. torch stays a runtime pull to keep the base image lean. Also add the runtime libs opencv-python (cv2) needs — libgl1, libglib2.0-0t64, libxcb1 — which the slim base omits; without them the install succeeds but `import cv2` dies with `libxcb.so.1: cannot open shared object file`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cookbook): don't let a package's sys.exit() on import hang the deps panel The local optional-dependency probe imports each package in-process and catches ImportError / Exception. But a package can call sys.exit() at import time — e.g. rembg does `sys.exit(1)` when no onnxruntime backend loads. SystemExit is a BaseException, not Exception, so it escaped the probe, propagated out of the list_packages endpoint, and hung the whole Dependencies panel / worker (the UI loads forever). Catch (Exception, SystemExit) so one broken optional package is reported as not-usable instead of taking down the panel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(routes): 500 (not 404) when the app-shell index.html is missing (#4791) Follow-up to #4637. serve_index — the handler for / and the SPA deep-link routes (/notes, /calendar, /cookbook, /email, /memory, /gallery, /tasks, /library) — pre-checked os.path.exists and raised its own HTTPException(404, "index.html not found") when the bundle was missing. So a missing core template returned 404 before serve_html_with_nonce's 500 could fire, the one inconsistency left after #4637. index.html is a fixed, app-bundled template; a missing one is a broken deployment (server fault), not a client "not found", so it should surface as a logged 500 in 5xx alerting rather than a 404. Keep the static->root fallback, drop the redundant existence guard and the dead-end 404, and let the shared helper handle the missing case. Verified against the running app: / and /notes return 200 with the bundle present and a logged 500 when index.html is absent. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787) setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv() but never loaded .env, so on native Linux/macOS installs a password pre-seeded in .env (documented in docs/setup.md and .env.example) was silently ignored and a random one generated, breaking the first login. Docker was unaffected because compose passes the vars into the container env. Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(), mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does not override already-exported OS vars, so the existing precedence is kept. python-dotenv is already a required dependency. Adds a regression test that pre-seeds credentials only in .env (not the shell) and asserts the stored bcrypt hash matches the pre-seeded password. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: email poller marks calendar extraction processed on LLM failure (#4622) Move calendar processed-marker insert into the LLM success path (else branch). Previously, the INSERT ran even after a transient LLM failure, causing the poller to skip retrying calendar extraction on subsequent runs. Minimal change: only touches the try/except/else control flow in _auto_summarize_pass_single() — preserves existing formatting and line endings. * feat(ui): add toggle for padding around chat area (#4691) * feat: Allow admins to choose if they want to share defaults (#4752) * First bare fix * Adding the option toggle * toggle function fix * Final fix, added missing /auth/ * Extended toggle text & added tests * Comments change * Description toggle change * br tag fix * description change based on suggestion * fix(agent): parse misfenced read_file calls (#4799) * fix: use atomic write in APIKeyManager.save() to prevent credential data loss (#4591) (#4597) * fix: use atomic write in APIKeyManager.save() to prevent data loss Opening api_keys.json with 'w' truncates the file before writing, so a crash, disk-full, or mid-write error leaves all stored provider API keys corrupted. Switch to atomic write (temp file + fsync + os.replace) so the original file is always intact on any failure. Fixes #4591 * chore: trigger CI re-run * chore: update PR description * chore: fix how-to-test section for description check --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * feat(discovery): detect llama.cpp servers and label local providers (#4729) * feat(discovery): detect llama.cpp servers and label local providers Scan port 8080 (llama-server) and 11435 (APFEL) during discovery, fingerprint llama.cpp via its native /props endpoint, and label well-known local serving ports (8080 llama.cpp, 8000 vLLM, 1234 LM Studio, 11434 Ollama) consistently in both the Python provider helper and the JS endpoint UI. Adds a llama.cpp hint to the /setup slash command. * fix(discovery): don't infer the serving tool from the port alone Per review: vLLM, SGLang, llama.cpp and plain OpenAI-compatible servers all share 8000/8080, so labeling by port mislabels real setups (a vLLM box on 8080 shown as llama.cpp). Drop the port->tool assertions from _provider_label and providerLabel; the authoritative signal is the /props fingerprint done during discovery, which is unchanged. Loopback now reads a neutral 'local endpoint' / 'Local'. Tests updated to assert the neutral labels. * refactor(tools): migrate config/integration admin tools to the registry (#4742) Part of #3629 (the `admin_tools.py` bullet). Moves the config/integration admin tools off the legacy elif dispatch chain in tool_implementations.py onto the agent_tools registry: manage_endpoints, manage_mcp, manage_webhooks, manage_tokens, manage_settings The do_* implementations (and manage_mcp's command-allowlist / RCE guard: _validate_mcp_command, _mcp_allowed_commands, and the _MCP_* constants) move verbatim into the new src/agent_tools/admin_tools.py. They register through a single ADMIN_TOOL_HANDLERS map that TOOL_HANDLERS.update()s, and the five elif branches plus their imports are dropped from tool_execution.py, so these tools now flow through _direct_fallback like the other migrated clusters. The names are re-exported from src.agent_tools for back-compat. Dedup: - _parse_tool_args was duplicated in tool_implementations.py and document_tools.py. It now lives once in src.tool_utils (which imports nothing from the project beyond src.constants, so this introduces no cycle) and both call sites import it from there. The orphaned `import json` in document_tools is removed with it. - The five tools share one _owner_adapter(fn) factory that threads ctx["owner"] into the owner-taking do_* signature, instead of five near-identical wrappers. Tests: new tests/test_admin_tools_registry.py pins the registration, the re-export back-compat, the owner-threading adapter, and the single-source _parse_tool_args (across admin_tools and document_tools). Existing MCP / settings / webhook suites are repointed at the new module. * refactor(exceptions): dedupe src/exceptions via core re-export (#4785) src/exceptions.py was a byte-for-byte duplicate of the canonical core/exceptions.py. Replace its class bodies with a re-export shim (mirroring the core/constants.py -> src/constants.py pattern) so the exception classes are defined in exactly one place. Also fix the stale "# src/exceptions.py" header comment in core/exceptions.py. No behavior change: both import paths resolve to the same class objects (verified by identity), so `except SessionNotFoundError` works regardless of which module it was imported from. Ran py_compile and pytest tests/test_app.py (12 passed). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619) Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike the chat path which stores build_chat_url(normalize_base(base)) on the session. A task carrying an explicit bare OpenAI-compatible base such as "http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the agent loop swallows the empty body into "The model returned an empty response" and marks the run success, so nothing surfaces the failure. Tasks that omit an endpoint dodge this only because _resolve_defaults() cribs an already-full URL from a recent chat session. The API/token path (e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1") hits it every time. Fix: route every resolved task endpoint through _normalize_chat_endpoint() at the three resolution sites (_execute_llm_task, the persona/research session path, and _execute_research_task). The helper is idempotent (strips any existing chat suffix, re-appends the correct one) and leaves native-Ollama (/api...) and already-concrete URLs untouched, so other providers are unaffected. Proven via isolated repro: ".../v1" -> 404 -> empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output. Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency, and the native-Ollama/empty passthroughs. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model-routes): harden _probe_endpoint against malformed model-list responses (#4789) * fix(model-routes): harden _probe_endpoint against malformed model-list responses _probe_endpoint parsed model lists with data.get(...) at four sites without checking that data is a dict, and built the list with a truthiness-only filter. A /models (or /api/tags) endpoint returning HTTP 200 with valid but non-dict JSON ([], "x", null, 123) made data.get(...) raise AttributeError, and a non-string id like 123 passed the filter and then hit .startswith() / .lower() in the Z.AI/Kimi curated merge and _is_chat_model(). Both errors are swallowed by the broad except Exception, but the comprehension dies mid-list so the ENTIRE probed model list is discarded and the endpoint silently degrades — masking a misconfigured/non-compliant upstream as "no models". - Guard each data.get(...) with isinstance(data, dict) so a non-dict body falls through the existing `or []` default. - Restrict the OpenAI and Ollama model-list comprehensions to non-empty str values, protecting the .startswith() merges and both _is_chat_model calls. - Add an isinstance guard at the top of _is_chat_model (defense in depth for all four call sites). No behavior change for well-formed {"data":[...]} / {"models":[...]} responses. Adds regression tests (non-dict body via caplog, mixed/all non-string ids, _is_chat_model boundary) that fail before the fix and pass after. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(model-routes): extract _openai_model_ids / _ollama_model_names helpers Per review on #4789: the malformed-response guards were inlined four times in _probe_endpoint (two OpenAI-id comprehensions, two Ollama-name comprehensions). Pull each into a small, directly-testable helper so the security-relevant parsing lives in one place and a future malformed-shape fix doesn't have to be applied in four spots (CONTRIBUTING flags repeated logic for this reason). Behavior is unchanged. Adds direct unit tests for both helpers (non-dict body, non-string ids, non-dict entries, name>model precedence). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cookbook): only block model launch on real port collisions (#4760) * Fix #4507: only block model launch on real port collisions Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so every launch collided. Both pre-launch guards (serve panel + quick-run) were count-based and fired regardless of port. - quick-run now auto-assigns a free port (8080 for llama.cpp) - both guards parse the new port and only prompt on a real overlap, stopping only the colliding serve - dialog reports the actual port instead of a hardcoded 8000 * refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port Addresses review on #4760: - _taskPort regex now matches --port= as well as --port (space) - _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex - quick-run llama.cpp no longer pins 8080, so two can run concurrently * fix(cookbook): _taskPort also parses -p; add port-parsing tests Addresses review on #4760: - _taskPort now matches -p <n> too, so it's the complete single reader (was missing the short flag that other readers already handle) - add tests/test_cookbook_port_parsing_js.py covering the port forms, shared-reader reuse, and llama.cpp auto-assign * test(cookbook): extract pure port helpers and test behavior Addresses review on #4760: the prior tests only asserted source strings. - extract portOf() and nextFreePort() into static/js/cookbookPorts.js - cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate - tests run the helpers via node and assert real behavior: all port forms (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the same-port-clash / different-port-coexist outcome --------- Co-authored-by: samy <samy@odysseus.boukouro.com> * fix(ui): route tasks.js + skills.js dropdowns through topPortalZ() (#4768) Fixes #4767. #4724 routed 16 body-portaled dropdowns through the shared topPortalZ() helper so they always render just above the currently-raised tool modal, but two were missed and still used a hardcoded z-index, so they hit the same #4720 bug once a modal's bring-to-front counter climbed past the literal: - tasks.js _showTaskDropdown(): inline z-index:100000 on .task-dropdown - skills.js kebab menu (.skill-kebab-menu): z-index:100002 in style.css Both now set zIndex from topPortalZ() after they are appended to the body, matching the other migrated sites. The dead CSS z-index on .skill-kebab-menu is removed (the inline value always wins). test_portal_dropdown_z_js.py gains a source guard asserting both files use topPortalZ() and that no hardcoded 100000/100002 portal literal survives in either file or style.css. * do_list_models in ai_interaction.py dropped --------- Co-authored-by: Max Hsu <maxmilian@users.noreply.github.com> Co-authored-by: aubrey <kyuhex@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ahmed Dlshad <ahmed.dlshad.m@gmail.com> Co-authored-by: Joel Alejandro Escareño Fernández <52678667+TheAlexz@users.noreply.github.com> Co-authored-by: Kalin Stoyanov <kgs.void@gmail.com> Co-authored-by: Pedro Barbosa <devpedrobarbosa@gmail.com> Co-authored-by: Solanki Sumit <125974181+YAMRAJ13y@users.noreply.github.com> Co-authored-by: Rudra Sarker <78224940+rudra496@users.noreply.github.com> Co-authored-by: Skoh <101289702+SkohTV@users.noreply.github.com> Co-authored-by: Jakub Grula <ramsters110@gmail.com> Co-authored-by: Dividesbyzer0 <54127744+zoomdbz@users.noreply.github.com> Co-authored-by: Kenny Van de Maele <kenny@kvandemaele.be> Co-authored-by: Magiomakes <114195802+Magiomakes@users.noreply.github.com> Co-authored-by: Samy <12219635+touzenesmy@users.noreply.github.com> Co-authored-by: samy <samy@odysseus.boukouro.com> * fix(upload): remove trailing whitespace * fix(security): redact credential-bearing URLs and PII from logs (#4750) * fix(security): redact credential-bearing URLs and PII from logs Several log statements emitted sensitive data in clear text: - model_routes / chat_routes / contacts_routes logged endpoint URLs raw. Admin-configured URLs can embed credentials in userinfo or query (e.g. https://user:pass@host, ?api_key=...). Route them through a shared core.log_safety.redact_url() that drops userinfo/query/fragment. - note_routes / task_scheduler logged operator email addresses (smtp_user, recipient). Replaced with presence booleans, which keeps the diagnostic ("why didn't this send") without writing PII to logs. model_routes already had a local redactor on its HTTPStatusError branch; the generic except branch was missed, so reuse the existing helper there. Clears CodeQL py/clear-text-logging-sensitive-data alerts 264, 317, 324, 325, 343, 344, 528. * fix(security): re-bracket IPv6 hosts and single-source the URL redactor Address review on #4750: - redact_url now re-brackets IPv6 literals so host:port stays unambiguous (https://[2001:db8::1]:8443/v1, not the bracket-less ambiguous form). - point model_routes._redact_url_for_log at the shared helper so the two redactors are single-sourced (also picks up the IPv6 fix). * fix(chat): honor explicit web search denial * fix(chat): require explicit web search enable (cherry picked from commit 54f1d015b578832abbf608e4d367097a1b3def0e) * fix(tasks): gate cookbook serve task execution (#5235) * fix(security): scope owner-less email accounts to a mailbox match in route guards (#5238) The HTTP email route guard `_assert_owns_account` and the explicit-account_id path in `_get_email_config` gated cross-tenant access with `if row.owner and row.owner != owner` -- which skips the check entirely when the account row is owner-less (owner NULL or ""). `email_accounts` is the one owner-scoped table left out of the legacy-owner migration backfill (core/database.py), so such rows persist on multi-user deploys: an account configured while auth was disabled, or an imported legacy row. Any authenticated user could then pass that account's id to read/send/update-credentials/delete another tenant's mailbox and read its decrypted IMAP/SMTP creds. Both sibling paths already enforce the intended contract -- the same-file `_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner` gate (whose comment says it mirrors "the HTTP email route fallback") only expose an owner-less account when its own mailbox (imap_user / from_address) is the caller's. Factor that row-level predicate into `_account_visible_to_owner` and use it in both guards, so owner-less accounts are visible only on a mailbox match. Owned accounts, the legacy-claim path, and single-user mode (owner == "") are unchanged. Complements #5234 (which fixes the same class on the MCP tool layer); this is the HTTP route layer it does not touch. * refactor(routes): move contacts domain into routes/contacts/ subpackage (#5227) Slice 2e of the route-domain reorganization (#4082/#4071, per specs/architecture-runtime-inventory.md §6.3). Moves contacts_routes.py into routes/contacts/, leaving a backward-compat sys.modules shim at the old path. Pure file reorganization, no behavior change. The shim uses sys.modules replacement (same pattern as the merged gallery `import routes.contacts_routes`, `from routes.contacts_routes import X`, `importlib.import_module(...)`, the string-targeted `monkeypatch.setattr("routes.contacts_routes.SETTINGS_FILE", ...)` used by test_carddav_password_encryption.py, and the `import ... as cr` + `setattr(cr, ...)` pattern in test_contacts_add_null_name.py all operate on the same module object the application uses. This also keeps the mutable module state `_contact_cache` identical across import paths. The canonical module does NOT depend on the shim — routes/contacts/ contacts_routes.py imports only from core/, src/, and stdlib (zero internal routes/ coupling). The inbound edge from routes/email_helpers.py (imports _fetch_contacts) keeps working through the shim. Zero source-introspection landmines — no test reads this file by path. Adds tests/test_contacts_routes_shim.py to pin the sys.modules shim contract (same-object + string-targeted monkeypatch reach-through). Verified: compileall clean; full suite 4485 passed, 3 skipped. * fix(security): pin webhook delivery to the SSRF-validated IP (DNS rebinding) (#5147) validate_webhook_url resolves the host to accept/reject, but the delivery connect (httpx.AsyncClient.post) re-resolved independently — a DNS record flipping between the two lookups (rebinding) could slip an internal IP (127.0.0.1 / 169.254.169.254 / LAN) past the check and receive the signed payload. The module docstring already flagged this as only a "partial defense". Resolve + validate once via _validated_public_ips, then pin the delivery TCP connect to that approved IP with an async _PinnedAsyncTransport built on the public httpcore/httpx APIs (mirrors the sync search-fetch pin from validation and vhost routing still target the original hostname; only the socket destination is pinned. Delivery now uses a per-request pinned client instead of one shared client, so close() is a no-op kept for API compatibility. Adds end-to-end tests that drive the real transport against loopback servers, proving the connect follows the pin rather than re-resolving the URL host. Fixes #5146 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(security): apply sensitive-file deny-list to grep tool (#5011) (#5013) The grep tool bypassed the sensitive-file deny-list that read_file, write_file, and edit_file all respect. Two code paths fixed: 1. ripgrep path: adds --glob exclusion patterns for each entry in _SENSITIVE_FILE_PATTERNS (id_rsa, known_hosts, authorized_keys, etc.) 2. Pure-Python os.walk fallback: checks _is_sensitive_path() before opening each file, skipping files that match the deny-list Fixes #5011 Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix(chat): sanitize web search query to strip markdown and code blocks (#4863) Layer a defensive cleanup on top of the generated-query web-search flow so the final selected query is sanitized before reaching comprehensive_web_search. - remove fenced code blocks from the final search query - preserve inline code as plain text - collapse whitespace and cap query length - cover generated-query success plus LLM failure/empty fallback paths Partially addresses #4547. * fix(calendar): honor list_events date range aliases (#3283) * fix(calendar): honor list_events date range aliases * fix(calendar): reject partially resolved loose range queries * fix(integrations): don't append a trailing slash when api_call path is '/' _join_integration_url built urljoin(base + '/', '') for a bare '/' path — the minimum execute_api_call accepts — so every request against a POST-to-base integration went to base_url + '/'. Discord webhook URLs 404 ('Unknown Webhook') on the trailing-slash variant, which made the integration look broken even though the stored base URL was correct. Resolve a bare '/' (or empty) path to the base URL itself and keep all other paths joining exactly as before, including deliberate trailing slashes inside non-empty paths (linkding /api/tags/, Home Assistant /api/). The reminder webhook sender and the discord_webhook connectivity test already posted to the bare base URL; execute_api_call was the remaining path that re-added the slash. Fixes #5138 * test: split service health tests (#4972) * test: split service health tests * test(service-health): preserve focus selector * fix(tools): handle non-dict JSON values in _parse_tool_args (closes #5043) (#5064) When an LLM generates a valid JSON string that parses to a native non-dict type (like a list, int, or string), _parse_tool_args previously returned that object. Callers expecting a dictionary would then crash with AttributeError or KeyError when attempting to look up action keys. - Update _parse_tool_args in src/tool_utils.py to explicitly type-check the parsed JSON object and return {} for non-dict objects. - Add test coverage in tests/test_admin_tools_registry.py for lists, ints, and strings. * test(integrations): drop redundant trailing-slash assertion The exact-equality assert on the line above (requested_url == WEBHOOK_BASE) already implies the URL has no trailing slash, so the endswith check adds nothing. * Move core serve memory fields further left * Move vllm attention farther right * Color cookbook context fit notes * fix(parser): parse Gemma 3/4 custom tool calling tokens (#5033) * fix: parse Gemma 3/4 custom tool calling tokens in parser * test: cover Gemma tool call parsing --------- Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com> * fix(agent): preserve bare email tool parity (#5075) * fix(agent): confine glob literal lookups to the search root (#5010) GlobTool resolves its search root through _resolve_search_root (which confines it to the workspace or default allowlist), but the literal fast-path joined the model-supplied pattern onto that root without re-confining it. os.path.join lets an absolute pattern or one containing ../ escape the root, and normpath collapsed the .. segments, so glob returned the absolute path of arbitrary host files once they existed -- an existence/path oracle that bypasses the confinement read_file, write_file, grep, and ls all enforce. Keep the literal lookup inside the root via a commonpath containment check; an escaping literal falls through to the os.walk matcher, which only ever yields paths under the root. Wildcard matching was already confined. * fix(agent): skip deny-listed sensitive files in glob (#5094) * fix(llm): normalize OpenAI-compatible chat URLs Normalize OpenAI-compatible chat URL shapes so base /v1 endpoints route to /v1/chat/completions while already-full chat endpoints remain idempotent. Preserve native local Ollama routing for bare localhost:11434 endpoints, keep localhost:11434/v1 as OpenAI-compatible, and add focused regression coverage for provider detection, chat target URLs, and model listing from /v1. Part of #541. * fix(search): use generated query for chat mode web search #4547 (#4557) * fix(search): use generated query for chat mode web search #4547 * style(search): tidy query generation call --------- Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com> * fix(email): validate IMAP/SMTP ports instead of crashing with 500 (#4464) The email-account endpoints coerced user-supplied ports with a bare int(data.get("imap_port") or 993), so a non-numeric port (e.g. "imap") raised ValueError and surfaced as an HTTP 500 in the create, update, and test-config endpoints. Add a _coerce_port(value, default) -> (port, error) helper and use it in all three endpoints, returning the endpoints standard {"ok": False, "error": ...} response (matching the existing "name required" validation) instead of crashing. A blank or missing port still falls back to the default (993/465). * fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704) * fix(security): prevent ReDoS in LLM-output tool/think parsers The regexes that parse untrusted model output in text_helpers.py and tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole response, they degrade to O(n^2) when the closing delimiter is absent: the engine rescans to end-of-string from every opener. Model output is untrusted, so a prompt-injected or malicious model can stall the agent loop with many unclosed openers (measured ~25s on a 60KB <thought flood). - text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the Gemma <|channel>...<channel|> subs when no <channel|> closer is present. - tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check for their closing delimiter. With no closer the regex cannot match, so skipping is equivalent; only the wasted O(n^2) rescan is removed. Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236, (its greedy [\s\S]*\Z is linear) and left untouched. * fix(security): close ReDoS gaps in tool/think parsers from review Addresses two review findings on the closer-guard approach: - Whole-string "closer exists?" checks were bypassable: a stale closer before an opener flood, or a closer with no reachable inner `}`, kept the guard true while every opener still rescanned to end-of-string (O(n^2)). Replace the substring guards with `_iter_delimited`, a forward-only scan that pairs each opener with a *later* closer and stops once none is reachable (O(n)). `parse_tool_blocks` and `strip_tool_blocks` (via `_strip_delimited`) both use it for the [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats. Verified equivalent to the original regexes on well-formed inputs. - `<thought([^>]*)>` dropped the tag-name boundary and corrupted unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`: the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*` overlap) while restoring the boundary; capture is byte-for-byte identical for real `<thought ...>` openers. Adds regressions for stale-closer-before-opener, closer-present-without- inner-brace, and the <thoughtful>/<thoughts> passthrough. * fix(security): close Gemma channel ReDoS guard flagged in review vdmkenny noted the same bypassable whole-string guard remained in text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma thought/response channel subs. A stale `<channel|>` before a `<|channel>thought` opener flood keeps the guard true while every opener still rescans to end-of-string (measured ~7.3s at 4k openers). Replace it with `_sub_delimited`, the same forward-only scan used for the tool-call parsers: pair each opener with a later closer, stop when none is reachable (O(n)). Verified output-equivalent to the original capture regexes on well-formed multi-channel inputs; the stale-closer case now runs in <2ms. Adds a regression for stale-closer-before-opener on the Gemma path. * fix(security): harden strip_think() think-tag ReDoS flagged in review The earlier fixes hardened normalize_thinking_markup and the delimiter scanners, but the production entrypoint strip_think() still ran _THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag _THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to end-of-string from every opener on a "many openers, no closer" flood. On the prior head, malformed `<think` / `<thinking` / `<thought` floods took 6-14s through strip_think(). The shipped `<thought>` normalization had the same residual: the single-opener case was linear but an opener flood was still O(n^2) (~4.4s). - Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing forward-only _sub_delimited scan (pair each opener with the first reachable closer, stop when none is reachable). One pass collapses sequential and nested blocks as before. - Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`) so a no-`>` opener flood can't drive a single match attempt to end-of-string. Identical capture for well-formed think/thought tags. - email_helpers._strip_think: compute had_think from the single linear _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which had the same O(n^2) on the email reply/summary/extraction paths. All flood variants now finish in <10ms (were 6-14s). Output verified byte-for-byte identical to the prior implementation over a 34-case corpus (nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds strip_think() timing regressions for malformed openers, opener floods (all three tag names), the closed-opener flood, and the malformed-closer flood. * docs: trim verbose comments in think-tag ReDoS fix * Support mobile enter for queued agent prompts * Keep open document context for section edits * Add bulk email attachment downloads * Add hover labels to mini sidebar buttons * fix(cookbook): treat local Windows as Windows for serve commands (#3975) * fix(cookbook): prefer native llama-server on local Windows * fix(cookbook): harden local llama-server launch commands * fix(cookbook): build serve commands for selected target * fix(cookbook): only block model launch on real port collisions (#4760) * Fix #4507: only block model launch on real port collisions Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so every launch collided. Both pre-launch guards (serve panel + quick-run) were count-based and fired regardless of port. - quick-run now auto-assigns a free port (8080 for llama.cpp) - both guards parse the new port and only prompt on a real overlap, stopping only the colliding serve - dialog reports the actual port instead of a hardcoded 8000 * refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port Addresses review on #4760: - _taskPort regex now matches --port= as well as --port (space) - _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex - quick-run llama.cpp no longer pins 8080, so two can run concurrently * fix(cookbook): _taskPort also parses -p; add port-parsing tests Addresses review on #4760: - _taskPort now matches -p <n> too, so it's the complete single reader (was missing the short flag that other readers already handle) - add tests/test_cookbook_port_parsing_js.py covering the port forms, shared-reader reuse, and llama.cpp auto-assign * test(cookbook): extract pure port helpers and test behavior Addresses review on #4760: the prior tests only asserted source strings. - extract portOf() and nextFreePort() into static/js/cookbookPorts.js - cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate - tests run the helpers via node and assert real behavior: all port forms (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the same-port-clash / different-port-coexist outcome --------- Co-authored-by: samy <samy@odysseus.boukouro.com> * fix(agent): execute fenced tool calls with inline args and route bare email tool names (#3681) * fix(agent): execute fenced tool calls with inline args and bare email tool names Two bugs made local (Ollama) models unable to use email tools, leaving raw fences like ```list_email_accounts {}``` in the chat: 1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a tool call with args on the same line ("```list_email_accounts {}") …
Summary
Adds opt-in SOPS support so deployers who want it can encrypt true secrets (
OPENAI_API_KEY,ODYSSEUS_ADMIN_PASSWORD,ODYSSEUS_INTERNAL_TOKEN, MCP OAuth secrets, IMAP passwords, etc.) at rest instead of leaving them plaintext in.env. The feature is only active when an encryptedsecrets.envis mounted at container start; existing deployments behave bit-for-bit identically.Linked Issue
Fixes #233
Scope (narrowed per reviewer feedback)
ODYSSEUS_ADMIN_PASSWORDreaches the admin seed.SEARXNG_SECRETetc., which need host-side compose interpolation) — see the linked tracking issue..gitignore/.dockerignorehardening for plaintext-and-decrypt artifacts — shipping as a small standalone PR so it can land independently.uvicorn app:app, systemd, launchd) startup hooks — SECURITY.md states the scope; native users wrap their own launch.Type of Change
Checklist
devdocker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.How to Test
End-to-end verified on an Ubuntu 24.04 / Docker 29.4.3 host; reproducible in <5 minutes.
.sops.yamlin the repo root: replace theage1REPLACE...placeholder with the public key from step 1.docker-compose.override.yml:docker compose up -d --builddocker exec odysseus-odysseus-1 ps -ef | head -3— PID 1 should besops exec-env /app/secrets.env ...docker exec -u 1000 odysseus-odysseus-1 sh -c "tr '\0' '\n' < /proc/<uvicorn-pid>/environ" | grep ODYSSEUS_TEST_SECRET— should print the plaintext valueInitial admin user created (admin)with no random-password notice — confirmsODYSSEUS_ADMIN_PASSWORDreachedsetup.pyinside the SOPS wrapdocker exec odysseus-odysseus-1 head -2 /app/secrets.env— showsENC[AES256_GCM,...]markers, never plaintextdocker exec odysseus-odysseus-1 grep -rI hello-from-sops /app /tmp 2>/dev/null— empty (plaintext never written to disk)docker exec odysseus-odysseus-1 curl -s -o /dev/null -w "%{http_code}\n" http://localhost:7000/→302secrets.envmount,docker compose up -d. Container boots identically to currentdev; entrypoint takes the existingexec gosupath with no SOPS-related logic touched.secrets.envat/app/secrets.env. Container exits 1 with a helpful error pointing atsops -e -i secrets.env— prevents the silent-shipping-plaintext-secrets footgun.Addressing the four surfaces called out on #233
The maintainer noted on #233: "secret storage touches setup, backups, Docker/native installs, and migration behavior." Each:
Setup:
setup.pyruns inside thesops exec-envwrap when an encryptedsecrets.envis present, so the encryptedODYSSEUS_ADMIN_PASSWORDreaches the admin-seed step. In the non-SOPS path setup.py runs as today. The script remains idempotent and runs as the dropped user via gosu.Backups: This is the design point that makes SOPS attractive vs Fernet. The encrypted
secrets.envfile is safe to commit and back up alongside the repo; the age private key (~/.config/sops/age/keys.txt, 0600) is the ONLY thing that must be backed up separately. SECURITY.md says this explicitly. There's no parallel "back updata/.app_key" footgun the way Fernet-based DB encryption has, because the encrypted file is content-addressable and recoverable from any backup including git.Docker vs native installs: This PR is Docker-only — only
docker/entrypoint.shinvokessops exec-env. Native (uvicorn app:app) installs are unaffected; native users wrap their own launch withsops exec-env. SECURITY.md documents this explicitly. A first-class native hook is out of scope here and can be a separate PR if requested.Migration: One-step, no breakage. Existing users keep their plaintext
.envunchanged. To move to encrypted: move the secret-flavored lines out of.envintosecrets.env, runsops -e -i secrets.env, restart with the mount in place. The PR'ssecrets.env.examplelists exactly which keys are expected as secrets and notes which ones are out of scope (e.g.SEARXNG_SECRET).Naming hazard
The codebase already has
routes/vault_routes.py(Bitwarden/Vaultwarden integration — a runtime feature). To avoid collision this PR uses "encrypted secrets at rest" / "SOPS" /secrets.envthroughout — never "vault."Diff
No application code touched. Image grows ~10MB (one static Go binary for
sops).Visual / UI changes
No UI changes. All edits are infrastructure (Dockerfile, entrypoint, docs).