Skip to content

feat(security): support optional encrypted-at-rest secrets via SOPS (re #233)#236

Open
nickorlabs wants to merge 1 commit into
odysseus-dev:devfrom
nickorlabs:feat/sops-encrypted-secrets
Open

feat(security): support optional encrypted-at-rest secrets via SOPS (re #233)#236
nickorlabs wants to merge 1 commit into
odysseus-dev:devfrom
nickorlabs:feat/sops-encrypted-secrets

Conversation

@nickorlabs

@nickorlabs nickorlabs commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds opt-in SOPS support so deployers who want it can encrypt true secrets (OPENAI_API_KEY, ODYSSEUS_ADMIN_PASSWORD, ODYSSEUS_INTERNAL_TOKEN, MCP OAuth secrets, IMAP passwords, etc.) at rest instead of leaving them plaintext in .env. The feature is only active when an encrypted secrets.env is mounted at container start; existing deployments behave bit-for-bit identically.

Linked Issue

Fixes #233

Scope (narrowed per reviewer feedback)

  • In scope: Docker entrypoint SOPS hook for app-internal secrets read by the wrapped odysseus process. setup.py runs inside the wrap so the encrypted ODYSSEUS_ADMIN_PASSWORD reaches the admin seed.
  • Out of scope, deferred to follow-ups:
    • Cross-service compose secrets (SEARXNG_SECRET etc., which need host-side compose interpolation) — see the linked tracking issue.
    • Build-context / .gitignore / .dockerignore hardening for plaintext-and-decrypt artifacts — shipping as a small standalone PR so it can land independently.
    • Native (uvicorn app:app, systemd, launchd) startup hooks — SECURITY.md states the scope; native users wrap their own launch.

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)
  • New feature (non-breaking — adds new behaviour)
  • Breaking change (changes or removes existing behaviour)
  • Refactor / cleanup (behaviour unchanged)
  • Documentation only
  • CI / tooling / configuration

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up) and verified the change works end-to-end. Type-checks and unit tests are not enough.

How to Test

End-to-end verified on an Ubuntu 24.04 / Docker 29.4.3 host; reproducible in <5 minutes.

  1. Generate an age key on the host:
    mkdir -p ~/.config/sops/age
    age-keygen -o ~/.config/sops/age/keys.txt && chmod 600 ~/.config/sops/age/keys.txt
    age-keygen -y < ~/.config/sops/age/keys.txt    # prints the public key
  2. Edit .sops.yaml in the repo root: replace the age1REPLACE... placeholder with the public key from step 1.
  3. Create + encrypt your secrets:
    cp secrets.env.example secrets.env
    $EDITOR secrets.env                       # add ODYSSEUS_TEST_SECRET=hello-from-sops (or real values)
    sops -e -i secrets.env                    # encrypt in place
  4. Mount the encrypted file and the age key into the container via docker-compose.override.yml:
    services:
      odysseus:
        volumes:
          - ./secrets.env:/app/secrets.env:ro
          - ~/.config/sops/age/keys.txt:/run/secrets/sops-age-key:ro
        environment:
          - SOPS_AGE_KEY_FILE=/run/secrets/sops-age-key
  5. Boot: docker compose up -d --build
  6. Verify:
    • docker exec odysseus-odysseus-1 ps -ef | head -3 — PID 1 should be sops exec-env /app/secrets.env ...
    • docker exec -u 1000 odysseus-odysseus-1 sh -c "tr '\0' '\n' < /proc/<uvicorn-pid>/environ" | grep ODYSSEUS_TEST_SECRET — should print the plaintext value
    • First-boot admin seed: container logs Initial admin user created (admin) with no random-password notice — confirms ODYSSEUS_ADMIN_PASSWORD reached setup.py inside the SOPS wrap
    • docker exec odysseus-odysseus-1 head -2 /app/secrets.env — shows ENC[AES256_GCM,...] markers, never plaintext
    • docker exec odysseus-odysseus-1 grep -rI hello-from-sops /app /tmp 2>/dev/null — empty (plaintext never written to disk)
    • HTTP serves: docker exec odysseus-odysseus-1 curl -s -o /dev/null -w "%{http_code}\n" http://localhost:7000/302
  7. Negative path: remove the secrets.env mount, docker compose up -d. Container boots identically to current dev; entrypoint takes the existing exec gosu path with no SOPS-related logic touched.
  8. Refuse-to-start path: mount a plaintext (non-SOPS) secrets.env at /app/secrets.env. Container exits 1 with a helpful error pointing at sops -e -i secrets.env — prevents the silent-shipping-plaintext-secrets footgun.

Addressing the four surfaces called out on #233

The maintainer noted on #233: "secret storage touches setup, backups, Docker/native installs, and migration behavior." Each:

Setup: setup.py runs inside the sops exec-env wrap when an encrypted secrets.env is present, so the encrypted ODYSSEUS_ADMIN_PASSWORD reaches the admin-seed step. In the non-SOPS path setup.py runs as today. The script remains idempotent and runs as the dropped user via gosu.

Backups: This is the design point that makes SOPS attractive vs Fernet. The encrypted secrets.env file is safe to commit and back up alongside the repo; the age private key (~/.config/sops/age/keys.txt, 0600) is the ONLY thing that must be backed up separately. SECURITY.md says this explicitly. There's no parallel "back up data/.app_key" footgun the way Fernet-based DB encryption has, because the encrypted file is content-addressable and recoverable from any backup including git.

Docker vs native installs: This PR is Docker-only — only docker/entrypoint.sh invokes sops exec-env. Native (uvicorn app:app) installs are unaffected; native users wrap their own launch with sops exec-env. SECURITY.md documents this explicitly. A first-class native hook is out of scope here and can be a separate PR if requested.

Migration: One-step, no breakage. Existing users keep their plaintext .env unchanged. To move to encrypted: move the secret-flavored lines out of .env into secrets.env, run sops -e -i secrets.env, restart with the mount in place. The PR's secrets.env.example lists exactly which keys are expected as secrets and notes which ones are out of scope (e.g. SEARXNG_SECRET).

Naming hazard

The codebase already has routes/vault_routes.py (Bitwarden/Vaultwarden integration — a runtime feature). To avoid collision this PR uses "encrypted secrets at rest" / "SOPS" / secrets.env throughout — never "vault."

Diff

 .sops.yaml           |  13 +++++++++++++
 Dockerfile           |  21 +++++++++++++++++++--
 SECURITY.md          |  56 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 docker/entrypoint.sh |  47 ++++++++++++++++++++++++++++++++++++++++---
 secrets.env.example  |  36 +++++++++++++++++++++++++++++++++
 5 files changed, 170 insertions(+), 3 deletions(-)

No application code touched. Image grows ~10MB (one static Go binary for sops).

Visual / UI changes

No UI changes. All edits are infrastructure (Dockerfile, entrypoint, docs).

@nickorlabs

Copy link
Copy Markdown
Contributor Author

End-to-end validation on a Linux/Docker server

Stood the full stack up on an Ubuntu 24.04 / Docker 29.4.3 host using this branch — closing the maintainer-checkbox in the test plan.

Setup performed (matches the SECURITY.md workflow exactly)

git clone <fork> && git checkout feat/sops-encrypted-secrets
age-keygen -o ~/.config/sops/age/keys.txt && chmod 600 ~/.config/sops/age/keys.txt
# Paste the public key into .sops.yaml
cp secrets.env.example secrets.env
$EDITOR secrets.env              # added ODYSSEUS_TEST_SECRET=hello-from-sops-2026
sops -e -i secrets.env           # encrypt in place
docker compose up -d --build

docker-compose.override.yml was used to bind-mount the age private key (/run/secrets/sops-age-key) and set SOPS_AGE_KEY_FILE — same shape as the README example.

Six gates, all green

1. PID 1 inside the container is sops exec-env wrapping uvicorn

UID         PID  PPID  CMD
odysseus      1     0  sops exec-env /app/secrets.env exec 'uvicorn' 'app:app' '--host' '0.0.0.0' '--port' '7000'
odysseus     84     1  /usr/local/bin/python3.12 /usr/local/bin/uvicorn app:app --host 0.0.0.0 --port 7000

2. Sentinel secret IS in uvicorn's process environment (read via /proc/<pid>/environ as the owning uid)

ODYSSEUS_TEST_SECRET=hello-from-sops-2026
PGID=1000
PUID=1000

3. secrets.env on disk is encrypted (file mounted from the image)

#ENC[AES256_GCM,data:myZekD5n4QtX...,iv:LvPGVuqQ7Svw...,tag:UiQWcgFX8l2pX8FE2cS61g==,type:comment]
ODYSSEUS_TEST_SECRET=ENC[AES256_GCM,data:yl/5B+J5o3kU...,iv:VbW99X/r/j1yA0gqv6dU4rqx...]

4. Plaintext value of the sentinel appears NOWHERE on the container filesystem

$ docker exec odysseus-odysseus-1 grep -rI hello-from-sops-2026 /app /tmp 2>/dev/null
(empty — no leak)

5. Secrets are scope-isolated to uvicorn's child tree — a fresh docker exec env does NOT see them

$ docker exec odysseus-odysseus-1 env | grep ODYSSEUS_TEST_SECRET
(empty — pass)

This is a nice bonus property: an attacker who lands a container exec shell after boot can't lift the SOPS-injected env from env//proc/self/environ. They'd need to read /proc/<uvicorn-pid>/environ as uid 1000.

6. The app boots and serves normally

$ docker exec odysseus-odysseus-1 curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://localhost:7000/
HTTP 302
# Logs show clean ChromaDB connect, 58 tools indexed, 5 built-in MCP servers up, NPX Browser MCP started, /docs reachable.

Compatibility note — re: #511 (Secure by default uplift, merged today)

Checked: zero file overlap between this PR and #511. The PR currently reports mergeStateStatus: CLEAN, mergeable: MERGEABLE. The two changes encrypt different layers:

They cover complementary surfaces.

Ready for review whenever convenient. Happy to rebase or adjust scope (e.g. gate sops install behind a build ARG so non-users don't pull the ~10MB binary, or swap age for GPG/cloud KMS) on request.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from 9d4c8e2 to ff6ae71 Compare June 1, 2026 23:37
@nickorlabs

Copy link
Copy Markdown
Contributor Author

Rebased on current main (post-#511) + re-validated

Pulled #511 (and the other 13 commits since this branch forked) and rebased. Diff stat unchanged: still 6 files, +141/-0.

 .gitignore           |  8 ++++++++
 .sops.yaml           | 13 +++++++++++++
 Dockerfile           |  9 +++++++++
 SECURITY.md          | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 docker/entrypoint.sh | 33 +++++++++++++++++++++++++++++++++
 secrets.env.example  | 27 +++++++++++++++++++++++++++

Three trivial 3-way merges happened on rebase (in your favor):

  • docker/entrypoint.sh — your new VLLM_USE_FLASHINFER_SAMPLER + extra CUDA wheel path block landed above my SOPS block cleanly
  • .gitignore — your docs/windows-port/ addition + mine
  • Dockerfile — your services/cache/search mkdir + mine

No semantic conflicts. PR status: mergeStateStatus: CLEAN, mergeable: MERGEABLE.

Re-validated on the same Ubuntu/Docker host, all 6 gates green

Gate 1  PID 1 is `sops exec-env` wrapping uvicorn  ✓
Gate 2  Sentinel in /proc/<uvicorn>/environ        ✓  ODYSSEUS_TEST_SECRET=hello-from-sops-2026
Gate 3  /app/secrets.env on disk is encrypted      ✓  ENC[AES256_GCM,...] markers
Gate 4  No plaintext value anywhere on disk        ✓  grep -rI on /app /tmp returns nothing
Gate 5  Fresh `docker exec env` sees no secrets    ✓  scope-isolated to uvicorn child tree
Gate 6  HTTP 302 from / (login redirect)           ✓  including across Tailscale

Compatibility with #511's new APP_BIND

Set APP_BIND=0.0.0.0 in the deployer's .env to keep the container reachable on the LAN/tailnet. Default loopback bind from #511 is preserved — SOPS doesn't interact with it.

Ready for review.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from ff6ae71 to 9e53b1e Compare June 2, 2026 13:28
@nickorlabs

Copy link
Copy Markdown
Contributor Author

Rebased on current main

Upstream moved while this PR was waiting; conflicts surfaced and I just rebased.

  • New base: c075abc Search: consolidate core and provider implementations
  • One auto-merge needed: docker/entrypoint.sh — your PR Expose Cookbook user-install CLIs in Docker #887 added an export PATH="/app/.local/bin:$PATH" line right after the VLLM_USE_FLASHINFER_SAMPLER block, and this PR's SOPS block targets the same spot. Resolution: keep both, in the order PATH first, then SOPS block (PATH always runs; SOPS only fires if secrets.env is present).
  • .gitignore and SECURITY.md auto-merged.

Diff stat unchanged: still 6 files, +141/-0, same shape, same semantics.

Status: MERGEABLE / CLEAN. Ready when convenient.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch 5 times, most recently from 74e8a87 to 298e3fb Compare June 3, 2026 16:22
@github-actions github-actions Bot added needs work PR description incomplete — please update before review ready for review Description complete — ready for maintainer review and removed needs work PR description incomplete — please update before review labels Jun 3, 2026
@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch 2 times, most recently from a439e06 to 843b2be Compare June 4, 2026 14:57
@alteixeira20
alteixeira20 changed the base branch from main to dev June 4, 2026 23:05
@alteixeira20

Copy link
Copy Markdown
Collaborator

Thanks for the detailed write-up and validation.

I did a focused local audit. From the local checks, I did not find a concrete merge blocker:

  • scope is coherent and limited to SOPS/encrypted-at-rest secrets support;

  • no application runtime code is touched;

  • git diff --check passed;

  • focused security-related tests passed:

    • tests/test_api_chat_security.py
    • tests/test_backup_cli_security.py
    • tests/test_email_smtp_security.py
    • tests/test_security_regressions.py
    • result: 112 passed;
  • merge probe into current upstream/dev was clean.

That said, I do not think I should approve or merge this without maintainer/security direction from @pewdiepie-archdaemon. This changes deployment and secret-management policy, including Docker image contents, entrypoint behavior, backup/key-management expectations, and the intended workflow around committable encrypted secrets.env files.

A few points I would want maintainer confirmation on before merge:

  • whether SOPS should be installed into every Docker image by default, or gated behind a build arg/optional image path;
  • whether downloading the SOPS binary in the Dockerfile needs checksum/signature verification;
  • whether the tracked .sops.yaml placeholder workflow is the desired UX, or whether .sops.yaml.local / documented local config is safer;
  • whether “encrypted secrets.env is committable” is acceptable project guidance for this repo’s expected users;
  • whether Docker-only support is the intended first slice, with native/systemd/launchd wrappers left for follow-up.

Technically promising, but I’d leave final approval/merge to @pewdiepie-archdaemon because this is security/deployment policy, not just implementation correctness.

Best regards,
Alexandre.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from 843b2be to a7cf1e1 Compare June 6, 2026 04:36
@nickorlabs

Copy link
Copy Markdown
Contributor Author

Quick workflow question now that dev is live alongside main: should this PR retarget to dev?

My read is main is still the right base — SOPS is infra-only (Dockerfile, entrypoint, gitignore, SECURITY.md), no feature code that needs dev's bake time, and the diff doesn't conflict with anything in dev. But if dev is now the default base for all incoming PRs regardless of category, happy to rebase onto it. Either way is one rebase.

@alteixeira20

Copy link
Copy Markdown
Collaborator

Thanks for asking.

Yes, I think this should target dev now.

The PR metadata already shows dev, but the current diff still includes unrelated main-side files (routes/document_routes.py, routes/gallery_routes.py, routes/task_routes.py, and src/caldav_sync.py), and GitHub reports it as DIRTY / CONFLICTING.

Could you please rebuild/rebase from current dev so the diff is SOPS-only again?

Expected final scope:

  • .gitignore
  • .sops.yaml
  • Dockerfile
  • SECURITY.md
  • docker/entrypoint.sh
  • secrets.env.example

Also please update the PR body checklist from main to dev.

Once that is clean, I can re-review the actual SOPS/security-policy questions separately.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from a7cf1e1 to 5d2a78e Compare June 7, 2026 15:30
@nickorlabs

Copy link
Copy Markdown
Contributor Author

Retargeted to dev. Rebased with --onto to keep only the SOPS commit (so the 4 main-only auth/security backports don't get replayed on top of dev). Two-dot diff is still the same 6 files, +141/-0.

$ git diff --name-only origin/dev..HEAD
.gitignore
.sops.yaml
Dockerfile
SECURITY.md
docker/entrypoint.sh
secrets.env.example

$ git diff --stat origin/dev..HEAD
 6 files changed, 141 insertions(+)

Branch HEAD: 5d2a78e. Ready for review against dev.

@alteixeira20

Copy link
Copy Markdown
Collaborator

Thanks, the rebase/scope issue is resolved now. The branch is clean against dev, checks are green, and the diff is back to the expected six SOPS-only files.

I still have one blocker before merge: the Dockerfile downloads the SOPS release binary with curl and makes it executable, but does not verify a checksum or signature.

Since this PR adds a secrets/deployment path, I think the install path should verify the downloaded binary in the same PR. Once that is added, this should be straightforward to approve.

@RaresKeY

Copy link
Copy Markdown
Collaborator

In addition to the already-raised checksum/signature blocker for the downloaded SOPS binary, I found two deployment-contract issues in the encrypted env workflow.

Findings

P1 Badge issue (security): Keep plaintext secret files out of the Docker build context

  • Problem: The new runtime check refuses a plaintext /app/secrets.env, but the Dockerfile has already copied the full build context into the image by then. .dockerignore excludes .env but not secrets.env or the new plaintext transient names, so a forgotten plaintext file or decrypt artifact can be baked into image layers before the entrypoint exits.

  • Impact: This defeats the new plaintext-at-rest guarantee and can leak real deployment secrets through local image cache or any pushed image.

  • Ask: Keep secrets.env* out of the image build context and provide the encrypted file at runtime instead, or add an equivalent design that prevents plaintext files from entering Docker layers before validation runs.

  • Location: Dockerfile:42

P2 Badge issue (security/deployment): The SOPS wrapper does not reach every secret consumer listed in the docs

  • Problem: The docs/example list ODYSSEUS_ADMIN_PASSWORD and SEARXNG_SECRET as values to move into encrypted secrets.env, but setup reads the admin password before sops exec-env, and SearXNG is a separate service outside the wrapped Odysseus process.

  • Impact: Users can follow the migration guide and still not have those secrets applied through the encrypted path.

  • Ask: Either wire those consumers into the SOPS-supported path, or narrow the docs/example to variables actually consumed by the wrapped Odysseus process.

  • Location: docker/entrypoint.sh:86

Validation

Ran git diff --check, sh -n docker/entrypoint.sh, and focused setup/security tests. Docker build/compose E2E was not run.

@nickorlabs

Copy link
Copy Markdown
Contributor Author

Thanks will look at this later today and make updates.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from 5d2a78e to 9fd4184 Compare June 11, 2026 21:13
@alteixeira20

Copy link
Copy Markdown
Collaborator

Thanks for the re-scope. Most of the previous security/deployment concerns look resolved now: checksum verification is present, the mixed Compose workflow is removed, Docker-only scope is explicit, SEARXNG_SECRET is out of scope, and the internal token name is now ODYSSEUS_INTERNAL_TOKEN.

I found one remaining small blocker before approval: .sops.yaml still says users can copy the config to .sops.yaml.local, but the documented encryption command is plain sops -e -i secrets.env. SOPS will discover .sops.yaml by default, not .sops.yaml.local, unless users pass an explicit config path. This reintroduces the config-discovery confusion from the earlier review.

Could you remove the .sops.yaml.local suggestion and keep the guidance aligned with editing .sops.yaml in place?

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch 2 times, most recently from 1853f71 to 7ff17f8 Compare June 15, 2026 16:51

@RaresKeY RaresKeY left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head follow-up review. This is not replacing or restating the earlier feedback: the earlier .sops.yaml.local config-discovery blocker looks fixed on this head, so I’m only adding the non-duplicative issues I found on the current head.

Findings

P1 Badge issue (runtime): Keep the app as the Docker stop signal target in SOPS mode

  • Problem: The non-SOPS path ends by execing the app through gosu, so Docker stop signals reach the application process directly. The SOPS path instead execs gosu ... sops exec-env ..., and SOPS then starts the final command in its default child-process mode. The shell's final exec$cmd replaces only SOPS' shell child, not the SOPS process that Docker targets as PID 1.

  • Impact: SOPS-enabled containers can skip the normal graceful shutdown path. In a SOPS 3.13.1 probe with the current command shape, plain exec-env left the final child running after SIGTERM, while the same command with --same-process made the child own the SOPS PID and receive SIGTERM.

  • Ask: Preserve the no-SOPS signal contract in the SOPS branch, for example by using sops exec-env --same-process "$SECRETS_FILE" "python /app/setup.py || true; exec$cmd" or an equivalent structure where the final app process is the signal target.

  • Location: docker/entrypoint.sh:119

P3 Badge issue (docs): Quote the native exec-env command as one shell command string

  • Problem: SECURITY.md tells native users to wrap startup with sops exec-env secrets.env -- uvicorn app:app ..., but SOPS v3.13.1 exec-env takes exactly two positional arguments and runs the second as a shell command string. It is not argv passthrough.

  • Impact: Native users following the documented workaround fail before Odysseus starts, which undermines the Docker-only scope guidance added by this PR.

  • Ask: Change the example to a single quoted command string, such as sops exec-env secrets.env 'uvicorn app:app ...', or otherwise document the exact SOPS-supported syntax.

  • Location: SECURITY.md:30

Validation

Ran:

  • git diff --check origin/dev...HEAD
  • sh -n docker/entrypoint.sh
  • docker compose --env-file /dev/null config
  • verified SOPS v3.13.1 exec-env parser behavior
  • reproduced the SOPS exec-env signal behavior with a throwaway encrypted dotenv

Not run:

  • Full Docker build.
  • Live container boot with mounted encrypted secrets.env and age key.
  • Full local pytest.

@nickorlabs

Copy link
Copy Markdown
Contributor Author

@alteixeira20 — good catch, thanks. That was a genuine miss on my end: .sops.yaml had skip-worktree set on my dev host (keeping local recipient keys out of git), so the rewrite of that file silently never reached the index when I amended for the narrowing. Apologies for the noise.

Pushed 7ff17f8 with the actual fix — the comment block now reads:

# SOPS encryption configuration for Odysseus secrets at rest.
#
# Replace the placeholder below with your own age public key, then
# encrypt with: sops -e -i secrets.env

No more .sops.yaml.local reference; the discovery path matches the documented sops -e -i secrets.env command. Verified via raw URL after the push so we don't re-litigate the same hidden-file mistake.

@alteixeira20 alteixeira20 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the .sops.yaml.local issue is fixed.

I still would not approve this yet because two current-head issues remain:

  1. In docker/entrypoint.sh, please preserve the normal Docker stop-signal/shutdown contract in SOPS mode. The current sops exec-env path still leaves SOPS as the process supervising the app command instead of replacing itself with the final app process. Please use sops exec-env --same-process or an equivalent structure so the final Odysseus process receives Docker stop signals the same way it does in the non-SOPS path.

  2. In SECURITY.md, please update the native wrapper example. sops exec-env expects the secrets file plus one command string, not argv-style passthrough. The documented example should quote the command, for example:

    sops exec-env secrets.env 'uvicorn app:app --host 0.0.0.0 --port 7000'

Once those are fixed, I can re-review the narrowed SOPS-only diff.

@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from 7ff17f8 to 48cae76 Compare June 16, 2026 03:33
@github-actions github-actions Bot added merge conflict Conflicts with the base branch; needs a rebase before review. ready for review Description complete — ready for maintainer review and removed ready for review Description complete — ready for maintainer review labels Jun 16, 2026
@nickorlabs

Copy link
Copy Markdown
Contributor Author

Thanks @RaresKeY. Both findings fixed on 48cae76.

N6 (P1 — SIGTERM not reaching the app in SOPS mode)

Added --same-process to the sops exec-env invocation:

exec gosu "$PUID:$PGID" sops exec-env --same-process "$SECRETS_FILE" \
    "python /app/setup.py || true; exec$cmd"

Verified end-to-end in a throwaway container with a real encrypted secrets.env + age key mount:

  • PID 1 ownership: ps -ef inside the container shows the chain's final sh -c (running as odysseus, UID 1000) as PID 1 — sops is no longer in the process tree.
  • Graceful shutdown: docker stop --timeout 10 against a CMD that traps SIGTERM and exits 0 completed in 0.59s, with the trap firing on PID 1:
    READY_AT_PID_$$
    GOT_SIGTERM_AT_PID_1
    
    Matches the no-SOPS branch's signal contract.
  • SOPS injection still works: same container with a CMD that dumps env: TEST_SECRET=... and ADMIN_PW=... both injected from the encrypted file, and setup.py logged Initial admin user created (admin) (the seeded password, not a random one) — proving the encrypted ODYSSEUS_ADMIN_PASSWORD still reaches setup.py under --same-process.

N7 (P3 — wrong native exec-env syntax in docs)

The Scope paragraph in SECURITY.md was using sops exec-env secrets.env -- uvicorn app:app ... (argv-passthrough), which v3.13.1 exec-env doesn't accept. Replaced with the single shell-string form, and noted the --same-process flag for the same signal-reaching reason:

sops exec-env --same-process secrets.env 'uvicorn app:app --host 0.0.0.0 --port 7000'

Diff (5 files, +177/-3)

 .sops.yaml           |  13 +++++++++++++
 Dockerfile           |  21 +++++++++++++++++++--
 SECURITY.md          |  56 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 docker/entrypoint.sh |  54 +++++++++++++++++++++++++++++++++++++++++++++++++---
 secrets.env.example  |  36 +++++++++++++++++++++++++++++++++

Adds opt-in SOPS support so deployers who want it can encrypt true
secrets (OPENAI_API_KEY, ODYSSEUS_ADMIN_PASSWORD, ODYSSEUS_INTERNAL_TOKEN,
MCP OAuth secrets, IMAP passwords, etc.) at rest instead of leaving them
plaintext in .env. Non-secret config (APP_PORT, LLM_HOST, etc.) stays
in .env so PR diffs remain reviewable.

Uses SOPS's idiomatic in-place encryption convention: secrets.env is
encrypted in place with 'sops -e -i secrets.env' and the encrypted file
itself is safe to commit. To edit later: 'sops secrets.env' opens
$EDITOR with decrypted contents and re-encrypts on save.

The feature is only active when /app/secrets.env is present at
container start; existing deployments behave exactly as before. If
secrets.env is present but NOT SOPS-encrypted (detected via the
sops_version marker), the entrypoint refuses to start — that shape is
almost always a packaging mistake.

Scope: the hook lives in docker/entrypoint.sh, so this only wraps the
Docker entrypoint path. Native startups (uvicorn directly, systemd,
launchd) read process env / .env as usual. Cross-service secrets like
SEARXNG_SECRET (consumed by a separate container via compose
interpolation) are also out of scope here — tracked as a follow-up.
Build-context and .gitignore hardening for plaintext/decrypt artifacts
ships as a small separate PR so it can land independently.

- Dockerfile: pin sops v3.13.1 (~10MB static Go binary), arch-aware
  via dpkg, per-arch SHA256 verified before chmod +x so a CDN
  substitution can't ship an unverified binary
- docker/entrypoint.sh: wrap gosu with 'sops exec-env' when encrypted
  secrets.env exists; setup.py runs INSIDE the wrap so the encrypted
  ODYSSEUS_ADMIN_PASSWORD reaches the admin-seed step; fail loudly if
  sops is missing, no SOPS_AGE_KEY[_FILE] is set, or the file is
  plaintext
- .sops.yaml: creation rule with placeholder age public key
- secrets.env.example: documents the in-place encryption workflow and
  the app-internal scope
- SECURITY.md: new 'Encrypting Secrets At Rest' section with the
  age-keygen -> 'sops -e -i' -> compose-mount workflow and scope notes

Verified end-to-end with the entrypoint exec-ing a sentinel-loaded
container:
  positive (encrypted file mounted, key bound)   -> sentinel in env,
                                                    setup.py sees seed
  negative (no secrets.env)                      -> entrypoint unchanged
  refuse   (plaintext secrets.env present)       -> exit 1 with hint

Proposed in odysseus-dev#233. Naming chosen to avoid colliding with the existing
runtime Bitwarden integration at routes/vault_routes.py.
@nickorlabs
nickorlabs force-pushed the feat/sops-encrypted-secrets branch from 48cae76 to 8b5d229 Compare June 16, 2026 18:37
@nickorlabs

Copy link
Copy Markdown
Contributor Author

@alteixeira20 — the two concerns in your latest review are exactly the same as @RaresKeY's N6 + N7, which I'd already fixed in the prior head 48cae76 (your review timestamp was ~2h before that push landed, so you were reading 7ff17f80).

Rebased onto current dev and resolved the conflict with #4349 — head is now 8b5d229. The SOPS branch now uses the $GOSU_BIN / $PYTHON_BIN variables introduced by #4349 instead of calling gosu / python directly, so the entrypoint is internally consistent:

exec "$GOSU_BIN" "$PUID:$PGID" sops exec-env --same-process "$SECRETS_FILE" \
    "$PYTHON_BIN /app/setup.py || true; exec$cmd"

Re-ran the same end-to-end signal + decryption test against the rebased image:

--- PID 1 ---
odysseus  1  0 ... sh -c trap "echo TRAPPED..." TERM; echo READY...; sleep 60 & wait
--- docker stop --timeout 10 ---
real    0m1.206s
--- logs ---
[ok] Initial admin user created (admin)
READY_PID_$$_SECRET=$ODYSSEUS_TEST_SECRET
TRAPPED_PID_1_SECRET=postrebase-2026-06-16

So: PID 1 = the final shell (running as odysseus / UID 1000, not as sops), docker stop exits in ~1.2s, the SIGTERM trap fires on PID 1 with the decrypted secret in the env, and setup.py still sees the encrypted ODYSSEUS_ADMIN_PASSWORD (no random password). All four invariants match the no-SOPS contract.

SECURITY.md's native example is the single quoted shell-string form you cited:

sops exec-env --same-process secrets.env 'uvicorn app:app --host 0.0.0.0 --port 7000'

Diff stat unchanged at 5 files / +177 / -3. Ready for re-review when you have time.

@github-actions github-actions Bot removed the merge conflict Conflicts with the base branch; needs a rebase before review. label Jun 16, 2026
@nickorlabs

Copy link
Copy Markdown
Contributor Author

@alteixeira20 @RaresKeY — when you have a moment, could you take another look? Both of your "Request Changes" reviews above are against commit 7ff17f80; the current head is 8b5d229 and contains the fixes you each asked for. A quick recap so you don't have to scroll the thread:

  • --same-process for SIGTERM forwardingdocker/entrypoint.sh:159 now reads exec "$GOSU_BIN" "$PUID:$PGID" sops exec-env --same-process "$SECRETS_FILE" "$PYTHON_BIN /app/setup.py || true; exec$cmd". Validated end-to-end on a real encrypted container: PID 1 = the final shell (UID 1000, not sops), docker stop --timeout 10 exits in 1.21s with the SIGTERM trap firing on PID 1, and setup.py still gets the encrypted ODYSSEUS_ADMIN_PASSWORD (logs Initial admin user created (admin), not random).
  • Native exec-env shell-string syntaxSECURITY.md:30 now reads sops exec-env --same-process secrets.env 'uvicorn app:app --host 0.0.0.0 --port 7000' (single quoted command string, not -- argv passthrough).
  • Rebase onto current dev — picked up $GOSU_BIN / $PYTHON_BIN from fix(devops): harden docker config defaults #4349 and threaded them through the SOPS branch, so the entrypoint is internally consistent. Diff stat 5 files / +177 / -3.

No further code changes pending — happy to address anything new the latest head surfaces, but the two items you flagged should both be resolved now.

@alteixeira20

Copy link
Copy Markdown
Collaborator

Sorry for the delay, tomorrow I'll take a couple of hours to get up to speed, I'm not home at the moment

@nickorlabs

Copy link
Copy Markdown
Contributor Author

@alteixeira20 @RaresKeY — when you have a moment, could you take another look? The current head 8b5d229 has the two items you each flagged resolved:

  • --same-process is now passed to sops exec-env so SIGTERM reaches PID 1 and docker stop behaves like the non-SOPS path.
  • The native exec-env example in SECURITY.md uses the correct single quoted shell-string form.
  • Rebased onto current dev and picked up the $GOSU_BIN / $PYTHON_BIN variables from fix(devops): harden docker config defaults #4349.

No further changes are pending from my side — happy to address anything new the latest head surfaces. Thanks!

@RaresKeY

Copy link
Copy Markdown
Collaborator

Thanks for the updates. Before I re-review this, could you please refresh/rebase the branch against current dev? GitHub currently reports the PR as DIRTY / CONFLICTING (mergeable: false, rebaseable: false) on head 8b5d229, so I can’t verify the latest SOPS diff cleanly against the current base yet.

The visible scope still looks SOPS-only (.sops.yaml, Dockerfile, SECURITY.md, docker/entrypoint.sh, secrets.env.example). After the rebase/conflict fix, please confirm the diff is still limited to that scope and rerun the relevant Docker/SOPS validation. Then I can re-review the updated head.

nopoz added a commit that referenced this pull request Jun 27, 2026
* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236,
#524. The _XML_OPEN_TOOL_CALL_RE alerts (#234, #477) are false positives
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix
Chocotunda pushed a commit to Chocotunda/odysseus that referenced this pull request Jun 28, 2026
…us-dev#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236,
odysseus-dev#524. The _XML_OPEN_TOOL_CALL_RE alerts (odysseus-dev#234, odysseus-dev#477) are false positives
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix
buluma pushed a commit to buluma/odysseus that referenced this pull request Jul 1, 2026
…us-dev#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236,
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix
buluma pushed a commit to buluma/odysseus that referenced this pull request Jul 2, 2026
…us-dev#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236,
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix

(cherry picked from commit c098355)
entitycs pushed a commit to entitycs/odysseus that referenced this pull request Jul 5, 2026
…us-dev#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236,
odysseus-dev#524. The _XML_OPEN_TOOL_CALL_RE alerts (odysseus-dev#234, odysseus-dev#477) are false positives
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix
samooth pushed a commit to samooth/ulises that referenced this pull request Jul 13, 2026
…us-dev#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos odysseus-dev#230, odysseus-dev#231, odysseus-dev#232, odysseus-dev#233, odysseus-dev#235, odysseus-dev#236,
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix
ChicoCifrado added a commit to digitalsovereignsociety/ulises that referenced this pull request Jul 20, 2026
… resolve code quality items (#1)

* Code quality: security, deps, error handling, tests, frontend, database

Phase 0 — Security (5/5):
  - Fix bare except in require_admin() (middleware.py)
  - Replace shell=True with list-form subprocess (builtin_actions.py)
  - Add .catch() to 3 dynamic imports (app.js)
  - Encrypt Webhook.secret with EncryptedText
  - Verify XSS safety in chat.js

Phase 1 — Deps/Config (7/7):
  - Pin all 28 deps; move test deps to requirements-dev.txt
  - Set asyncio_loop_scope = module, trim CORS whitespace
  - Pin Docker to python:3.13-slim
  - Replace __import__ with standard imports

Phase 2 — Error Handling (3/3):
  - Create safe_create_task() helper; fix 13 fire-and-forget tasks
  - Lock _response_cache with threading.Lock()
  - Narrow bare excepts in subprocess_tools, builtin_actions,
    agent_loop, task_scheduler (24 instances fixed)

Phase 3 — Tests (2/2):
  - Fix test_session_manager explicit assertion
  - Remove src.database MagicMock fallback in conftest.py

Phase 4 — Frontend (5/5):
  - Scope MutationObservers to #app; add beforeunload cleanup
  - Create shared API_BASE module (14 files updated)
  - Fix admin.js undefined API_BASE bug
  - Fix chatRenderer.js window.API_BASE fallback
  - Fix _defaultChat race, rename guard in app.js

Phase 5 — Database (3/3):
  - Defer init_db() from module import to app startup
  - Add .limit() safety caps to 11 unbounded queries
  - Extract 26 ORM models to core/orm_models.py (709 lines)
  - Add index=True to 6 FK columns

Phase 6 — Cross-cutting (3/4):
  - Add Docker socket security warning
  - Fix dead-host cache: move _clear_host_dead after status check
  - Cache EncryptedText import to avoid per-call overhead
  - Add -> None to note_model_activity

* making ulises multilang

* finished multilang base & tests

* fix(security): scope owner-less email accounts to a mailbox match in route guards (#5238)
The HTTP email route guard `_assert_owns_account` and the explicit-account_id
path in `_get_email_config` gated cross-tenant access with
`if row.owner and row.owner != owner` -- which skips the check entirely when the
account row is owner-less (owner NULL or ""). `email_accounts` is the one
owner-scoped table left out of the legacy-owner migration backfill
(core/database.py), so such rows persist on multi-user deploys: an account
configured while auth was disabled, or an imported legacy row. Any authenticated
user could then pass that account's id to read/send/update-credentials/delete
another tenant's mailbox and read its decrypted IMAP/SMTP creds.

Both sibling paths already enforce the intended contract -- the same-file
`_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner`
gate (whose comment says it mirrors "the HTTP email route fallback") only expose
an owner-less account when its own mailbox (imap_user / from_address) is the
caller's. Factor that row-level predicate into `_account_visible_to_owner` and
use it in both guards, so owner-less accounts are visible only on a mailbox
match. Owned accounts, the legacy-claim path, and single-user mode (owner == "")
are unchanged.

Complements #5234 (which fixes the same class on the MCP tool layer); this is
the HTTP route layer it does not touch.

* fix(agent): fall back to keyword tool selection when retrieval times out
The retrieval-timeout branch hard-coded ALWAYS_AVAILABLE, silently skipping
the deterministic keyword hints whenever the embedding backend was slow
(e.g. a remote endpoint cold-loading its model). Queries that named email
or calendar outright lost those tools and the model concluded the
integrations did not exist. Let the timeout fall through to the existing
keyword fallback instead — same baseline, plus the hints.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(search): pin httpx connection to resolved IP to block DNS rebinding (#704)
* fix(search): pin DNS-validated fetch connections

Rebase the DNS-rebinding SSRF fix onto current dev after search content moved behind the services.search.content canonical module.

Integrate the pinned httpcore NetworkBackend/BaseTransport approach with the current size-capped Client.stream fetch path, preserving Host/SNI semantics while forcing TCP connect to the already validated resolved IP.

Keep src.search.content as the compatibility wrapper and preserve existing OG-image http(s) behavior; this avoids reintroducing the unrelated scope changes that previously blocked review.

Add the explicit httpcore>=1.0,<2.0 requirement used by the public httpcore NetworkBackend and ConnectionPool APIs.

* test(search): restore and rebase DNS rebinding regressions

Keep the current security regression coverage that the stale PR branch had deleted, including auth-disabled localhost bypass and Ollama cookbook hardening tests.

Carry forward the DNS-rebinding coverage for private resolve blocking, pinned TCP connect behavior, Host header preservation, redirect revalidation, and the BaseTransport/public-httpcore static guard.

Update redirect tests to mock the current Client.stream-based capped fetch path rather than the older httpx.stream/get path.

* test(search): adapt size-cap fetch tests to pinned client stream

The DNS-rebinding repair moved _get_public_url from the module-level httpx.stream shortcut to httpx.Client(...).stream(...) so the fetch can use the pinned transport.

Keep the existing size-cap test fakes by routing Client.stream through the monkeypatched httpx.stream only when a test has installed that fake; otherwise fall back to a real Client.

This fixes the CI failures in tests/test_web_fetch_size_caps.py without touching unrelated upload-handler atomicity behavior, which is already flaky on clean origin/dev.

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix: resolve RAG manager search signature TypeError (#4994)
* fix: resolve RAG manager search signature TypeError and adjust similarity threshold

* fix: revert similarity threshold change to keep PR focused

* test(rag): remove trailing whitespace

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix(security): scope send_to_session to an exact session owner

* fix(tools): handle non-dict JSON values in _parse_tool_args (closes #5043) (#5064)

* fix(search): extract non-ASCII capitalized names in _extract_entities

* fix(agent): skip deny-listed sensitive files in glob (#5094)

* fix(security): match the sensitive-file deny-list case-insensitively (#5097)

* fix(integrations): don't append a trailing slash when api_call path is '/'

* fix(calendar): honor list_events date range aliases (#3283)

* fix: auto-spam move/delete targets the wrong message (seqnum vs UID) (#1874)

* fix(session): use utcnow_naive across session routes (#1116) (#5003)

* fix(cookbook): stop Ollama runner from executing the install one-liner (#3926)

* fix(security): apply the webhook SSRF guard to the reminder ntfy sender (#5142)

* fix(security): pin webhook delivery to the SSRF-validated IP (DNS rebinding) (#5147)

* fix(security): validate integration api_call URLs with the outbound SSRF guard (#5145)

* fix(security): match grep's rg sensitive-file exclusions case-insensitively (#5189)

* Fix Cookbook download runner for Python 3.11

* fix: add grace period to document tidy to prevent deleting new documents (#5036)

* fix(email): enforce MCP account owner scope (#5234)

* fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704)

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

* fix(security): prevent ReDoS in XML and args tool-call parsers (#4941)

* fix(chat): sanitize web search query to strip markdown and code blocks (#4863)

* fix(security): batch A — 7 security fixes

* fix(bugfix): batch B — 16 critical bugfixes

* fix(chat): batch C — web search denial + explicit intent

* test(ci): batch D — test splits + CI guidance

* feat(refactor): batch E — features, refactors, catalog

* fix(infra): batch F — 16 infrastructure + bugfixes

* feat(docs): batch G — models, documents, docs updates

* fix(ux): batch H — email, gallery, cookbook UX fixes

* feat(models): batch I — models hardening + OCR captions

* ui: batch J1 — cookbook/llama model UI spacing

* ui: batch J2 — cookbook engine tint, MTP, vllm spacing

* feat(background): batch J3 — background tasks, CLI sanitization, info-budget fixes

* chore(release): bump version to 1.0.2

* fix(ui): lower cookbook serve top row (cherry-pick fc6a6dc)

* fix(chat): honor explicit web search denial

* fix(mobile): stack the model-comparison grid into one column on phones (#4979)

The comparison grid hard-codes 2-4 equal columns with no phone breakpoint, so at
390px two models get ~178px columns and four get ~88px columns. Each column is a
full scrolling chat, so content is unreadably over-wrapped and clipped. On
phones (<=768px), stack the panes into a single scrollable column. Desktop is
unaffected.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(docs): correct broken backup-restore link in setup.md

Fixes #4926 - the link used docs/backup-restore.md from within docs/setup.md, which resolved to docs/docs/backup-restore.md (404). Changed to same-directory relative path.

* docs(setup): add a self-host troubleshooting cookbook of common traps (#4834)

ROADMAP "Self-host troubleshooting cookbook" asks to document the weird
30-second fixes that otherwise become 30-minute searches. Adds a "Common
self-host traps" subsection under Troubleshooting covering: the UTF-8 BOM
.env gotcha (app.py loads with utf-8-sig), macOS AirPlay holding port 7000
(the start script uses 7860), the plain-HTTP Tailscale/LAN clipboard
limitation, self-hosted ntfy delivery (NTFY_BIND/NTFY_BASE_URL + the ntfy
Android Instant-delivery toggle), Dovecot cleartext-auth on LAN mail stacks,
and Radicale full-collection-URL sync.

Docs only; grounded in existing repo behavior (.env.example NTFY_* block,
app.py utf-8-sig loader, start-macos.sh port choice).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task
executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike
the chat path which stores build_chat_url(normalize_base(base)) on the
session. A task carrying an explicit bare OpenAI-compatible base such as
"http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the
agent loop swallows the empty body into "The model returned an empty
response" and marks the run success, so nothing surfaces the failure.

Tasks that omit an endpoint dodge this only because _resolve_defaults()
cribs an already-full URL from a recent chat session. The API/token path
(e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1")
hits it every time.

Fix: route every resolved task endpoint through _normalize_chat_endpoint()
at the three resolution sites (_execute_llm_task, the persona/research
session path, and _execute_research_task). The helper is idempotent
(strips any existing chat suffix, re-appends the correct one) and leaves
native-Ollama (/api...) and already-concrete URLs untouched, so other
providers are unaffected. Proven via isolated repro: ".../v1" -> 404 ->
empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output.

Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency,
and the native-Ollama/empty passthroughs.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787)

setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv()
but never loaded .env, so on native Linux/macOS installs a password
pre-seeded in .env (documented in docs/setup.md and .env.example) was
silently ignored and a random one generated, breaking the first login.
Docker was unaffected because compose passes the vars into the container env.

Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(),
mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does
not override already-exported OS vars, so the existing precedence is kept.
python-dotenv is already a required dependency.

Adds a regression test that pre-seeds credentials only in .env (not the
shell) and asserts the stored bcrypt hash matches the pre-seeded password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(llm): detect mistral.ai provider and support reasoning_effort (#4698)

* fix(llm): detect mistral.ai provider and support reasoning_effort

Four coupled bugs broke Mistral thinking model support:

1. _detect_provider() had no mistral.ai host check, so all Mistral
   endpoints fell through to the generic 'openai' provider string.
   _provider_display_name() correctly identified them as 'Mistral',
   making any 'if provider == "Mistral"' check elsewhere dead code.

2. reasoning_effort parameter was never sent in the request payload,
   so Mistral never activated thinking mode even when the user
   configured a thinking-capable model (mistral-small-latest,
   mistral-medium-latest, magistral-*).

3. Mistral returns content as a typed array
   ([{"type":"thinking",...},{"type":"text",...}]) when
   reasoning is on, not as a plain string. Both the streaming and
   non-streaming parsers expected strings and silently dropped the
   thinking content.

4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-*
   model prefixes, so the frontend wouldn't tag reasoning output
   as thinking even after the above were fixed.

Fix:
- Add mistral.ai to _detect_provider() host checks
- Add a _normalize_mistral_content() helper that splits the typed
  array into (text, thinking) strings
- Inject payload["reasoning_effort"] = "high" when provider is
  Mistral and _supports_thinking(model) is true, in both stream_llm
  and llm_call_async payload construction
- Wire the normalizer into both response parsers
- Extend _THINKING_MODEL_PATTERNS to include magistral,
  mistral-small, mistral-medium, mistral-large

Tested on Docker install with mistral-small-latest +
reasoning_effort=high. Reasoning streams correctly into the
thinking panel after the fix.

Fixes #4678

* fix(llm): address review — lowercase provider id, configurable effort, tests

Addresses vdmkenny's review on PR #4698:

1. Removed duplicate 'if provider == "mistral"' block in stream_llm
   — two back-to-back copies, one was dead-redundant.

2. Dropped personal-context comment ('free-tier limits are generous
   for this user') and made reasoning_effort configurable via env var
   ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none).
   Default remains 'high' for backward compat with the tested behavior.

3. Recased provider id from 'Mistral' to 'mistral' to match the
   lowercase convention used by every other provider id in the file
   (openai, anthropic, ollama, copilot, ...). _provider_display_name()
   still returns the Title-Case 'Mistral' for UI labels — only the
   runtime id used in 'if provider == ...' checks was recased.

4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning
   _normalize_mistral_content()'s contract: string passthrough, the
   Mistral array format (thinking + text blocks), and edge cases
   (empty, garbage, None, wrong types, missing fields, string-vs-array
   inner thinking field).

Also fixed a gap the review didn't catch: the non-streaming paths
(llm_call sync + llm_call_async) were missing the reasoning_effort
injection entirely. Added the same injection to both, so Deep Research
and agent tool calls also activate Mistral thinking.

All 13 new tests pass. Existing reasoning/streaming/ollama-thinking
tests still pass (38 tests, no regressions).

Fixes #4678

* fix(chat): restore missing _explicit_web_intent definition (#5290)

chat_stream() references `_explicit_web_intent` in three places
(disabled-tools gating, global-disabled web allowance, and the
per-turn tool filter) but the assignment was dropped during a
branch merge. Every chat request raised

    NameError: name '_explicit_web_intent' is not defined

at routes/chat_routes.py, surfacing to the client as a bare
"Internal Server Error" before any LLM call was made — chat was
fully broken on dev and main.

Restore the original definition, computed from the already-derived
tool intent, immediately before its first use:

    _explicit_web_intent = bool(_tool_intent and _tool_intent.category == "web")

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 2d8177035b6a1dd3e9211c6cb5ba68443b826504)

* Parse local function_model tool wrappers

* Show fallback model in picker

* Nudge serve engine control up

* Move core serve memory fields farther left

* Move vllm block size left again

* Nudge vllm attention field right

* Raise unified llama context estimate

* fix(llm-core): prevent cache-affinity fields from reaching Cerebras

Recognize api.cerebras.ai as a Cerebras cloud provider so llama.cpp/LM Studio cache-affinity fields are not attached even when endpoint_kind is misconfigured as local. Add regression coverage for provider detection, self-hosted classification, and payload field exclusion.

* fix(models): accept bare-list /models responses (Together AI) (#4761)

* fix(api): handle varying response formats for model IDs from compatible providers

merge conflict for pr-2204 resolved

* fix(modal): keep body-portaled dropdowns above their tool modal at any stack depth (#4720) (#4724)

* fix(memory): keep the Brain memory item menu above the modal at any stack depth

The memory item "⋮" dropdown is portaled to <body> with a hardcoded
z-index of 10001. Tool modals, however, get a monotonically increasing
z-index from modalManager's bring-to-front counter (_modalTopZ), which
climbs unbounded as modals are opened/restored over a session. Once that
counter passes 10001, the Brain modal stacks above the body-portaled
dropdown, so the menu renders behind the panel — visible only where it
spills past the modal's edge (#4720).

Derive the dropdown's z-index from the owning modal's current z-index
(+1), keeping 10001 as a floor for the common low-counter case, so the
menu always sits just above its modal however high the counter has climbed.

Verified with document.elementFromPoint at the dropdown's location: with a
high modal z-index the old build returns the modal at every sampled point
(menu behind); the fixed build returns the dropdown (menu on top). The
default low-counter case is unchanged (z stays 10001).

* refactor(modal): route body-portaled dropdowns through a shared topPortalZ() helper

The hardcoded z-index:10001 the Brain memory menu used (#4720) is the same
literal shared by ~16 body-portaled dropdowns across calendar, cookbook,
cookbookServe, documentLibrary, emailLibrary, gallery, notes, emojiPicker and
memory — each renders behind its owning tool modal once modalManager's
bring-to-front counter climbs past the literal over a long session.

Promote the per-dropdown fix into a single topPortalZ() helper in
toolWindowZOrder.js — the existing source of truth for tool-window z, already
imported by modalManager's _bringToFront and notes.js — returning
max(topToolWindowZ(), dock-chip floor) + 1, so a portaled dropdown always sits
just above the live tool-window stack however high the counter has climbed.
Route all 16 sites through it. The slashCommands tour tooltips and the
cookbookServe VRAM dialog are intentionally left out (neither is a modal-owned
portaled dropdown).

Add tests/test_portal_dropdown_z_js.py covering the helper, including the #4720
scenario (modal counter at 99999 -> dropdown at 100000). Existing
test_notes_z_order_js.py stays green.

* fix(llm): detect mistral.ai provider and support reasoning_effort (#4698)

* fix(llm): detect mistral.ai provider and support reasoning_effort

Four coupled bugs broke Mistral thinking model support:

1. _detect_provider() had no mistral.ai host check, so all Mistral
   endpoints fell through to the generic 'openai' provider string.
   _provider_display_name() correctly identified them as 'Mistral',
   making any 'if provider == "Mistral"' check elsewhere dead code.

2. reasoning_effort parameter was never sent in the request payload,
   so Mistral never activated thinking mode even when the user
   configured a thinking-capable model (mistral-small-latest,
   mistral-medium-latest, magistral-*).

3. Mistral returns content as a typed array
   ([{"type":"thinking",...},{"type":"text",...}]) when
   reasoning is on, not as a plain string. Both the streaming and
   non-streaming parsers expected strings and silently dropped the
   thinking content.

4. _THINKING_MODEL_PATTERNS didn't include magistral or mistral-*
   model prefixes, so the frontend wouldn't tag reasoning output
   as thinking even after the above were fixed.

Fix:
- Add mistral.ai to _detect_provider() host checks
- Add a _normalize_mistral_content() helper that splits the typed
  array into (text, thinking) strings
- Inject payload["reasoning_effort"] = "high" when provider is
  Mistral and _supports_thinking(model) is true, in both stream_llm
  and llm_call_async payload construction
- Wire the normalizer into both response parsers
- Extend _THINKING_MODEL_PATTERNS to include magistral,
  mistral-small, mistral-medium, mistral-large

Tested on Docker install with mistral-small-latest +
reasoning_effort=high. Reasoning streams correctly into the
thinking panel after the fix.

Fixes #4678

* fix(llm): address review — lowercase provider id, configurable effort, tests

Addresses vdmkenny's review on PR #4698:

1. Removed duplicate 'if provider == "mistral"' block in stream_llm
   — two back-to-back copies, one was dead-redundant.

2. Dropped personal-context comment ('free-tier limits are generous
   for this user') and made reasoning_effort configurable via env var
   ODYSSEUS_MISTRAL_REASONING_EFFORT (high / medium / low / none).
   Default remains 'high' for backward compat with the tested behavior.

3. Recased provider id from 'Mistral' to 'mistral' to match the
   lowercase convention used by every other provider id in the file
   (openai, anthropic, ollama, copilot, ...). _provider_display_name()
   still returns the Title-Case 'Mistral' for UI labels — only the
   runtime id used in 'if provider == ...' checks was recased.

4. Added tests/test_llm_core_mistral_content.py with 13 tests pinning
   _normalize_mistral_content()'s contract: string passthrough, the
   Mistral array format (thinking + text blocks), and edge cases
   (empty, garbage, None, wrong types, missing fields, string-vs-array
   inner thinking field).

Also fixed a gap the review didn't catch: the non-streaming paths
(llm_call sync + llm_call_async) were missing the reasoning_effort
injection entirely. Added the same injection to both, so Deep Research
and agent tool calls also activate Mistral thinking.

All 13 new tests pass. Existing reasoning/streaming/ollama-thinking
tests still pass (38 tests, no regressions).

Fixes #4678

* fix: Images cannot be seen by model that is vision capable (#4726)

* fix: Images cannot be seen by model that is vision capable

* fix: skip http(s) image_url for Ollama (images[] is base64-only)

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix(chat): strip executed email tool fences from the live stream (#3993) (#4275)

* fix(chat): strip executed email tool fences from the live stream (#3993)

The backend strips every fenced tool block from persisted text (the regex in
src/tool_parsing.py is built from the full TOOL_TAGS set, which includes the
email tools), so a reloaded session renders cleanly. The live frontend path
uses a separate hardcoded EXEC_FENCE_RE in static/js/chatRenderer.js that only
listed web_search/read_file/write_file/create_document/edit_document/
update_document — so executed email tool fences (list_emails, etc.) lingered as
raw code blocks in the live assistant bubble until the user reloaded.

Add the nine email tool tags to EXEC_FENCE_RE so the live render settles into
the same clean layout as the history reload. bash/python stay excluded on
purpose: those are languages a user may legitimately have asked the model to
show as code, not tool invocations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): single-source live exec-fence tool list from TOOL_TAGS (#3993)

Per review: EXEC_FENCE_RE was a second, hand-maintained copy of the
executable-tool list, so any tool not in it — and every future tool added to
TOOL_TAGS — would leave its executed fence lingering in the live bubble until
reload (the original #3993 bug, recurring one tool at a time).

EXEC_FENCE_RE is now built from an explicit EXEC_TOOL_TAGS list that mirrors
TOOL_TAGS (src/agent_tools/__init__.py) minus bash/python, which stay excluded
as legitimate code-example languages. A new regression test
(test_exec_fence_re_covers_all_executable_tools) extracts both lists from
source and fails if they drift, so the whole class is caught in CI instead of
by a user — the "minimum acceptable middle ground" from the review, made exact
(set equality, not just coverage).

Verified: pytest tests/test_live_strip_email_tool_fences.py (5 passed);
node --check static/js/chatRenderer.js; and a node run of the built regex
confirms email/generate_image/manage_memory/ls fences strip while
bash/python/sh are preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(chat): build live exec-fence list from /api/tools at runtime (#3993)

Make TOOL_TAGS the single source for live exec-fence stripping. chatRenderer.js
no longer hard-codes a tool list; it fetches the backend's authoritative set
once from GET /api/tools (sorted(TOOL_TAGS)) and builds EXEC_FENCE_RE from it at
load, minus bash/python. No second list to drift, and a future tool added to
TOOL_TAGS is covered automatically — without touching the streaming path.

Until the fetch resolves EXEC_FENCE_RE is null and exec fences aren't stripped
(a sub-second window before the first stream); the backend already strips
persisted history, so a reload always renders clean.

Drop test_exec_fence_re_covers_all_executable_tools (no hand-maintained list to
guard) and add source-level guards: the frontend keeps no hard-coded list and
fetches /api/tools, and the endpoint serves the full sorted(TOOL_TAGS).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

* fix(chat): warn on /api/tools fetch failure instead of swallowing it (#3993)

A fresh-context review flagged that loadExecFenceRegex's catch silently
discarded errors: if the one-shot fetch fails, EXEC_FENCE_RE stays null for the
whole session and live exec fences go unstripped until reload, with zero signal.
console.warn it, and correct the comment to describe the failure mode honestly
(was understated as just a sub-second startup window).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CVCKth4g8pWh7pwFDVm4iL

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): log and cleanly 500 on unreadable HTML page (#4637)

* fix(routes): serve 404 instead of 500 when an HTML page file is missing

_serve_html_with_nonce opened the HTML file with no error handling, and
callers such as /backgrounds and /login pass their paths in with no
existence check, so a missing or unreadable file raised an unhandled
OSError that surfaced as a 500. Wrap the read and raise HTTPException(404)
instead; the normal render path (CSP-nonce substitution) is unchanged.

Fixes #4594

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): distinguish missing page (404) from read failure (500)

The previous fix caught a broad OSError and returned 404 for every
failure, which masks real server-side problems (permission errors, I/O
failures) as "not found" and lets them slip past error alerting. Split
FileNotFoundError (genuine 404) from other OSError, which now logs the
exception and returns a generic 500 — without leaking the OS error
string or file path into the response body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): treat unreadable bundled HTML page as logged 500, not 404

Per PR #4637 review: every caller of the page-render helper serves a fixed,
server-owned template (index/login/backgrounds), never a client-supplied
path. So a missing or unreadable file is a server fault (broken deployment),
not a client "not found" — a 404 there mislabels a server error and hides a
missing core template from 5xx alerting, contradicting the OSError->500
rationale this PR is built on. Collapse both branches into a single logged,
leak-free 500.

Move the helper to src.app_helpers.serve_html_with_nonce so the behavior can
be unit-tested without importing the whole app (app.py is the slim
orchestrator; the test harness stubs src.database, so importing app in tests
is not viable). Add tests pinning missing/unreadable -> 500 (not 404) and
nonce injection on the happy path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(catalog): add Gemma 4 12B/QAT entries and RTX 3050 bandwidth (#4728)

Add official Gemma 4 12B-it plus QAT-INT4/INT8 catalog entries (with their
GGUF sources), QAT quantization support across the quant tables and the
prequantized-prefix list, and the missing RTX 3050 / 3050 Ti memory
bandwidth so speed estimates stop falling back to the generic cuda value.

* fix debugging on windows (#4679)

* fix: Real-ESRGAN install + Cookbook deps-panel crash on the Python 3.14 image (#4694)

* fix(docker): make Real-ESRGAN installable on the Python 3.14 image

realesrgan's deps basicsr/gfpgan/facexlib (unmaintained since 2022) read
their version in setup.py via `exec(...); locals()['__version__']`, which
raises KeyError on Python 3.13+ — PEP 667 made locals() in a function an
independent snapshot that exec() can no longer mutate. That fails the
Cookbook "install realesrgan" sdist build on the python:3.14 base.

Add a `realesrgan-wheels` builder stage that fetches the pinned sdists,
patches get_version() to exec into an explicit namespace dict, and builds
wheels; the final stage installs them --no-deps so a later
`pip install realesrgan` resolves from wheels instead of rebuilding the
broken sdists. torch stays a runtime pull to keep the base image lean.

Also add the runtime libs opencv-python (cv2) needs — libgl1,
libglib2.0-0t64, libxcb1 — which the slim base omits; without them the
install succeeds but `import cv2` dies with
`libxcb.so.1: cannot open shared object file`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cookbook): don't let a package's sys.exit() on import hang the deps panel

The local optional-dependency probe imports each package in-process and
catches ImportError / Exception. But a package can call sys.exit() at
import time — e.g. rembg does `sys.exit(1)` when no onnxruntime backend
loads. SystemExit is a BaseException, not Exception, so it escaped the
probe, propagated out of the list_packages endpoint, and hung the whole
Dependencies panel / worker (the UI loads forever).

Catch (Exception, SystemExit) so one broken optional package is reported
as not-usable instead of taking down the panel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(routes): 500 (not 404) when the app-shell index.html is missing (#4791)

Follow-up to #4637. serve_index — the handler for / and the SPA deep-link
routes (/notes, /calendar, /cookbook, /email, /memory, /gallery, /tasks,
/library) — pre-checked os.path.exists and raised its own
HTTPException(404, "index.html not found") when the bundle was missing. So a
missing core template returned 404 before serve_html_with_nonce's 500 could
fire, the one inconsistency left after #4637.

index.html is a fixed, app-bundled template; a missing one is a broken
deployment (server fault), not a client "not found", so it should surface as a
logged 500 in 5xx alerting rather than a 404. Keep the static->root fallback,
drop the redundant existence guard and the dead-end 404, and let the shared
helper handle the missing case.

Verified against the running app: / and /notes return 200 with the bundle
present and a logged 500 when index.html is absent.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(setup): load .env so a pre-seeded admin password is honored on native installs (#4787)

setup.py read ODYSSEUS_ADMIN_USER / ODYSSEUS_ADMIN_PASSWORD via os.getenv()
but never loaded .env, so on native Linux/macOS installs a password
pre-seeded in .env (documented in docs/setup.md and .env.example) was
silently ignored and a random one generated, breaking the first login.
Docker was unaffected because compose passes the vars into the container env.

Call load_dotenv(BASE_DIR/.env, encoding="utf-8-sig") at the top of main(),
mirroring app.py (utf-8-sig tolerates a Notepad UTF-8 BOM). load_dotenv does
not override already-exported OS vars, so the existing precedence is kept.
python-dotenv is already a required dependency.

Adds a regression test that pre-seeds credentials only in .env (not the
shell) and asserts the stored bcrypt hash matches the pre-seeded password.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: email poller marks calendar extraction processed on LLM failure (#4622)

Move calendar processed-marker insert into the LLM success path (else branch).
Previously, the INSERT ran even after a transient LLM failure, causing the
poller to skip retrying calendar extraction on subsequent runs.

Minimal change: only touches the try/except/else control flow in
_auto_summarize_pass_single() — preserves existing formatting and line endings.

* feat(ui): add toggle for padding around chat area (#4691)

* feat: Allow admins to choose if they want to share defaults (#4752)

* First bare fix

* Adding the option toggle

* toggle function fix

* Final fix, added missing /auth/

* Extended toggle text & added tests

* Comments change

* Description toggle change

* br tag fix

* description change based on suggestion

* fix(agent): parse misfenced read_file calls (#4799)

* fix: use atomic write in APIKeyManager.save() to prevent credential data loss (#4591) (#4597)

* fix: use atomic write in APIKeyManager.save() to prevent data loss

Opening api_keys.json with 'w' truncates the file before writing, so a
crash, disk-full, or mid-write error leaves all stored provider API keys
corrupted. Switch to atomic write (temp file + fsync + os.replace) so
the original file is always intact on any failure.

Fixes #4591

* chore: trigger CI re-run

* chore: update PR description

* chore: fix how-to-test section for description check

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* feat(discovery): detect llama.cpp servers and label local providers (#4729)

* feat(discovery): detect llama.cpp servers and label local providers

Scan port 8080 (llama-server) and 11435 (APFEL) during discovery, fingerprint
llama.cpp via its native /props endpoint, and label well-known local serving
ports (8080 llama.cpp, 8000 vLLM, 1234 LM Studio, 11434 Ollama) consistently
in both the Python provider helper and the JS endpoint UI. Adds a llama.cpp
hint to the /setup slash command.

* fix(discovery): don't infer the serving tool from the port alone

Per review: vLLM, SGLang, llama.cpp and plain OpenAI-compatible servers all
share 8000/8080, so labeling by port mislabels real setups (a vLLM box on 8080
shown as llama.cpp). Drop the port->tool assertions from _provider_label and
providerLabel; the authoritative signal is the /props fingerprint done during
discovery, which is unchanged. Loopback now reads a neutral 'local endpoint' /
'Local'. Tests updated to assert the neutral labels.

* refactor(tools): migrate config/integration admin tools to the registry (#4742)

Part of #3629 (the `admin_tools.py` bullet). Moves the config/integration admin
tools off the legacy elif dispatch chain in tool_implementations.py onto the
agent_tools registry:

  manage_endpoints, manage_mcp, manage_webhooks, manage_tokens, manage_settings

The do_* implementations (and manage_mcp's command-allowlist / RCE guard:
_validate_mcp_command, _mcp_allowed_commands, and the _MCP_* constants) move
verbatim into the new src/agent_tools/admin_tools.py. They register through a
single ADMIN_TOOL_HANDLERS map that TOOL_HANDLERS.update()s, and the five elif
branches plus their imports are dropped from tool_execution.py, so these tools
now flow through _direct_fallback like the other migrated clusters. The names
are re-exported from src.agent_tools for back-compat.

Dedup:
  - _parse_tool_args was duplicated in tool_implementations.py and
    document_tools.py. It now lives once in src.tool_utils (which imports nothing
    from the project beyond src.constants, so this introduces no cycle) and both
    call sites import it from there. The orphaned `import json` in document_tools
    is removed with it.
  - The five tools share one _owner_adapter(fn) factory that threads ctx["owner"]
    into the owner-taking do_* signature, instead of five near-identical wrappers.

Tests: new tests/test_admin_tools_registry.py pins the registration, the
re-export back-compat, the owner-threading adapter, and the single-source
_parse_tool_args (across admin_tools and document_tools). Existing MCP /
settings / webhook suites are repointed at the new module.

* refactor(exceptions): dedupe src/exceptions via core re-export (#4785)

src/exceptions.py was a byte-for-byte duplicate of the canonical
core/exceptions.py. Replace its class bodies with a re-export shim
(mirroring the core/constants.py -> src/constants.py pattern) so the
exception classes are defined in exactly one place. Also fix the stale
"# src/exceptions.py" header comment in core/exceptions.py.

No behavior change: both import paths resolve to the same class objects
(verified by identity), so `except SessionNotFoundError` works regardless
of which module it was imported from. Ran py_compile and
pytest tests/test_app.py (12 passed).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(tasks): normalize task endpoint URL to /chat/completions before model call (#4619)

Upstream bug (present in pewdiepie-archdaemon/odysseus main): the task
executor passes task.endpoint_url VERBATIM to the model HTTP call, unlike
the chat path which stores build_chat_url(normalize_base(base)) on the
session. A task carrying an explicit bare OpenAI-compatible base such as
"http://host:11434/v1" therefore POSTs to a 404 ("page not found"); the
agent loop swallows the empty body into "The model returned an empty
response" and marks the run success, so nothing surfaces the failure.

Tasks that omit an endpoint dodge this only because _resolve_defaults()
cribs an already-full URL from a recent chat session. The API/token path
(e.g. an external client that POSTs /api/tasks with endpoint_url=".../v1")
hits it every time.

Fix: route every resolved task endpoint through _normalize_chat_endpoint()
at the three resolution sites (_execute_llm_task, the persona/research
session path, and _execute_research_task). The helper is idempotent
(strips any existing chat suffix, re-appends the correct one) and leaves
native-Ollama (/api...) and already-concrete URLs untouched, so other
providers are unaffected. Proven via isolated repro: ".../v1" -> 404 ->
empty; ".../v1/chat/completions" -> 200 -> real gemma4:31b output.

Regression test asserts the bare-/v1 -> full-chat-URL mapping, idempotency,
and the native-Ollama/empty passthroughs.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(model-routes): harden _probe_endpoint against malformed model-list responses (#4789)

* fix(model-routes): harden _probe_endpoint against malformed model-list responses

_probe_endpoint parsed model lists with data.get(...) at four sites without
checking that data is a dict, and built the list with a truthiness-only
filter. A /models (or /api/tags) endpoint returning HTTP 200 with valid but
non-dict JSON ([], "x", null, 123) made data.get(...) raise AttributeError,
and a non-string id like 123 passed the filter and then hit .startswith() /
.lower() in the Z.AI/Kimi curated merge and _is_chat_model(). Both errors are
swallowed by the broad except Exception, but the comprehension dies mid-list
so the ENTIRE probed model list is discarded and the endpoint silently
degrades — masking a misconfigured/non-compliant upstream as "no models".

- Guard each data.get(...) with isinstance(data, dict) so a non-dict body
  falls through the existing `or []` default.
- Restrict the OpenAI and Ollama model-list comprehensions to non-empty str
  values, protecting the .startswith() merges and both _is_chat_model calls.
- Add an isinstance guard at the top of _is_chat_model (defense in depth for
  all four call sites).

No behavior change for well-formed {"data":[...]} / {"models":[...]}
responses. Adds regression tests (non-dict body via caplog, mixed/all
non-string ids, _is_chat_model boundary) that fail before the fix and pass
after.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(model-routes): extract _openai_model_ids / _ollama_model_names helpers

Per review on #4789: the malformed-response guards were inlined four times in
_probe_endpoint (two OpenAI-id comprehensions, two Ollama-name comprehensions).
Pull each into a small, directly-testable helper so the security-relevant
parsing lives in one place and a future malformed-shape fix doesn't have to be
applied in four spots (CONTRIBUTING flags repeated logic for this reason).

Behavior is unchanged. Adds direct unit tests for both helpers (non-dict body,
non-string ids, non-dict entries, name>model precedence).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cookbook): only block model launch on real port collisions (#4760)

* Fix #4507: only block model launch on real port collisions

Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so
every launch collided. Both pre-launch guards (serve panel + quick-run)
were count-based and fired regardless of port.

- quick-run now auto-assigns a free port (8080 for llama.cpp)
- both guards parse the new port and only prompt on a real overlap,
  stopping only the colliding serve
- dialog reports the actual port instead of a hardcoded 8000

* refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port

Addresses review on #4760:
- _taskPort regex now matches --port= as well as --port (space)
- _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex
- quick-run llama.cpp no longer pins 8080, so two can run concurrently

* fix(cookbook): _taskPort also parses -p; add port-parsing tests

Addresses review on #4760:
- _taskPort now matches -p <n> too, so it's the complete single reader
  (was missing the short flag that other readers already handle)
- add tests/test_cookbook_port_parsing_js.py covering the port forms,
  shared-reader reuse, and llama.cpp auto-assign

* test(cookbook): extract pure port helpers and test behavior

Addresses review on #4760: the prior tests only asserted source strings.
- extract portOf() and nextFreePort() into static/js/cookbookPorts.js
- cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate
- tests run the helpers via node and assert real behavior: all port forms
  (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the
  same-port-clash / different-port-coexist outcome

---------

Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(ui): route tasks.js + skills.js dropdowns through topPortalZ() (#4768)

Fixes #4767. #4724 routed 16 body-portaled dropdowns through the shared
topPortalZ() helper so they always render just above the currently-raised tool
modal, but two were missed and still used a hardcoded z-index, so they hit the
same #4720 bug once a modal's bring-to-front counter climbed past the literal:

  - tasks.js _showTaskDropdown(): inline z-index:100000 on .task-dropdown
  - skills.js kebab menu (.skill-kebab-menu): z-index:100002 in style.css

Both now set zIndex from topPortalZ() after they are appended to the body,
matching the other migrated sites. The dead CSS z-index on .skill-kebab-menu is
removed (the inline value always wins). test_portal_dropdown_z_js.py gains a
source guard asserting both files use topPortalZ() and that no hardcoded
100000/100002 portal literal survives in either file or style.css.

* do_list_models in ai_interaction.py dropped

---------

Co-authored-by: Max Hsu <maxmilian@users.noreply.github.com>
Co-authored-by: aubrey <kyuhex@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ahmed Dlshad <ahmed.dlshad.m@gmail.com>
Co-authored-by: Joel Alejandro Escareño Fernández <52678667+TheAlexz@users.noreply.github.com>
Co-authored-by: Kalin Stoyanov <kgs.void@gmail.com>
Co-authored-by: Pedro Barbosa <devpedrobarbosa@gmail.com>
Co-authored-by: Solanki Sumit <125974181+YAMRAJ13y@users.noreply.github.com>
Co-authored-by: Rudra Sarker <78224940+rudra496@users.noreply.github.com>
Co-authored-by: Skoh <101289702+SkohTV@users.noreply.github.com>
Co-authored-by: Jakub Grula <ramsters110@gmail.com>
Co-authored-by: Dividesbyzer0 <54127744+zoomdbz@users.noreply.github.com>
Co-authored-by: Kenny Van de Maele <kenny@kvandemaele.be>
Co-authored-by: Magiomakes <114195802+Magiomakes@users.noreply.github.com>
Co-authored-by: Samy <12219635+touzenesmy@users.noreply.github.com>
Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(upload): remove trailing whitespace

* fix(security): redact credential-bearing URLs and PII from logs (#4750)

* fix(security): redact credential-bearing URLs and PII from logs

Several log statements emitted sensitive data in clear text:

- model_routes / chat_routes / contacts_routes logged endpoint URLs raw.
  Admin-configured URLs can embed credentials in userinfo or query
  (e.g. https://user:pass@host, ?api_key=...). Route them through a
  shared core.log_safety.redact_url() that drops userinfo/query/fragment.
- note_routes / task_scheduler logged operator email addresses (smtp_user,
  recipient). Replaced with presence booleans, which keeps the diagnostic
  ("why didn't this send") without writing PII to logs.

model_routes already had a local redactor on its HTTPStatusError branch;
the generic except branch was missed, so reuse the existing helper there.

Clears CodeQL py/clear-text-logging-sensitive-data alerts 264, 317, 324,
325, 343, 344, 528.

* fix(security): re-bracket IPv6 hosts and single-source the URL redactor

Address review on #4750:
- redact_url now re-brackets IPv6 literals so host:port stays
  unambiguous (https://[2001:db8::1]:8443/v1, not the bracket-less
  ambiguous form).
- point model_routes._redact_url_for_log at the shared helper so the
  two redactors are single-sourced (also picks up the IPv6 fix).

* fix(chat): honor explicit web search denial

* fix(chat): require explicit web search enable

(cherry picked from commit 54f1d015b578832abbf608e4d367097a1b3def0e)

* fix(tasks): gate cookbook serve task execution (#5235)

* fix(security): scope owner-less email accounts to a mailbox match in route guards (#5238)

The HTTP email route guard `_assert_owns_account` and the explicit-account_id
path in `_get_email_config` gated cross-tenant access with
`if row.owner and row.owner != owner` -- which skips the check entirely when the
account row is owner-less (owner NULL or ""). `email_accounts` is the one
owner-scoped table left out of the legacy-owner migration backfill
(core/database.py), so such rows persist on multi-user deploys: an account
configured while auth was disabled, or an imported legacy row. Any authenticated
user could then pass that account's id to read/send/update-credentials/delete
another tenant's mailbox and read its decrypted IMAP/SMTP creds.

Both sibling paths already enforce the intended contract -- the same-file
`_owner_or_matching_legacy_account` fallback and the MCP `_account_visible_to_owner`
gate (whose comment says it mirrors "the HTTP email route fallback") only expose
an owner-less account when its own mailbox (imap_user / from_address) is the
caller's. Factor that row-level predicate into `_account_visible_to_owner` and
use it in both guards, so owner-less accounts are visible only on a mailbox
match. Owned accounts, the legacy-claim path, and single-user mode (owner == "")
are unchanged.

Complements #5234 (which fixes the same class on the MCP tool layer); this is
the HTTP route layer it does not touch.

* refactor(routes): move contacts domain into routes/contacts/ subpackage (#5227)

Slice 2e of the route-domain reorganization (#4082/#4071, per
specs/architecture-runtime-inventory.md §6.3). Moves contacts_routes.py into
routes/contacts/, leaving a backward-compat sys.modules shim at the old path.
Pure file reorganization, no behavior change.

The shim uses sys.modules replacement (same pattern as the merged gallery
`import routes.contacts_routes`, `from routes.contacts_routes import X`,
`importlib.import_module(...)`, the string-targeted
`monkeypatch.setattr("routes.contacts_routes.SETTINGS_FILE", ...)` used by
test_carddav_password_encryption.py, and the `import ... as cr` +
`setattr(cr, ...)` pattern in test_contacts_add_null_name.py all operate on
the same module object the application uses. This also keeps the mutable
module state `_contact_cache` identical across import paths.

The canonical module does NOT depend on the shim — routes/contacts/
contacts_routes.py imports only from core/, src/, and stdlib (zero internal
routes/ coupling). The inbound edge from routes/email_helpers.py (imports
_fetch_contacts) keeps working through the shim.

Zero source-introspection landmines — no test reads this file by path.

Adds tests/test_contacts_routes_shim.py to pin the sys.modules shim contract
(same-object + string-targeted monkeypatch reach-through).

Verified: compileall clean; full suite 4485 passed, 3 skipped.

* fix(security): pin webhook delivery to the SSRF-validated IP (DNS rebinding) (#5147)

validate_webhook_url resolves the host to accept/reject, but the delivery
connect (httpx.AsyncClient.post) re-resolved independently — a DNS record
flipping between the two lookups (rebinding) could slip an internal IP
(127.0.0.1 / 169.254.169.254 / LAN) past the check and receive the signed
payload. The module docstring already flagged this as only a "partial
defense".

Resolve + validate once via _validated_public_ips, then pin the delivery
TCP connect to that approved IP with an async _PinnedAsyncTransport built
on the public httpcore/httpx APIs (mirrors the sync search-fetch pin from
validation and vhost routing still target the original hostname; only the
socket destination is pinned.

Delivery now uses a per-request pinned client instead of one shared client,
so close() is a no-op kept for API compatibility. Adds end-to-end tests that
drive the real transport against loopback servers, proving the connect
follows the pin rather than re-resolving the URL host.

Fixes #5146

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(security): apply sensitive-file deny-list to grep tool (#5011) (#5013)

The grep tool bypassed the sensitive-file deny-list that read_file,
write_file, and edit_file all respect. Two code paths fixed:

1. ripgrep path: adds --glob exclusion patterns for each entry in
   _SENSITIVE_FILE_PATTERNS (id_rsa, known_hosts, authorized_keys, etc.)
2. Pure-Python os.walk fallback: checks _is_sensitive_path() before
   opening each file, skipping files that match the deny-list

Fixes #5011

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix(chat): sanitize web search query to strip markdown and code blocks (#4863)

Layer a defensive cleanup on top of the generated-query web-search flow so the final selected query is sanitized before reaching comprehensive_web_search.

- remove fenced code blocks from the final search query
- preserve inline code as plain text
- collapse whitespace and cap query length
- cover generated-query success plus LLM failure/empty fallback paths

Partially addresses #4547.

* fix(calendar): honor list_events date range aliases (#3283)

* fix(calendar): honor list_events date range aliases

* fix(calendar): reject partially resolved loose range queries

* fix(integrations): don't append a trailing slash when api_call path is '/'

_join_integration_url built urljoin(base + '/', '') for a bare '/'
path — the minimum execute_api_call accepts — so every request against
a POST-to-base integration went to base_url + '/'. Discord webhook
URLs 404 ('Unknown Webhook') on the trailing-slash variant, which made
the integration look broken even though the stored base URL was
correct.

Resolve a bare '/' (or empty) path to the base URL itself and keep all
other paths joining exactly as before, including deliberate trailing
slashes inside non-empty paths (linkding /api/tags/, Home Assistant
/api/). The reminder webhook sender and the discord_webhook
connectivity test already posted to the bare base URL; execute_api_call
was the remaining path that re-added the slash.

Fixes #5138

* test: split service health tests (#4972)

* test: split service health tests

* test(service-health): preserve focus selector

* fix(tools): handle non-dict JSON values in _parse_tool_args (closes #5043) (#5064)

When an LLM generates a valid JSON string that parses to a native non-dict
type (like a list, int, or string), _parse_tool_args previously returned
that object. Callers expecting a dictionary would then crash with
AttributeError or KeyError when attempting to look up action keys.

- Update _parse_tool_args in src/tool_utils.py to explicitly type-check
  the parsed JSON object and return {} for non-dict objects.
- Add test coverage in tests/test_admin_tools_registry.py for lists,
  ints, and strings.

* test(integrations): drop redundant trailing-slash assertion

The exact-equality assert on the line above (requested_url == WEBHOOK_BASE)
already implies the URL has no trailing slash, so the endswith check adds
nothing.

* Move core serve memory fields further left

* Move vllm attention farther right

* Color cookbook context fit notes

* fix(parser): parse Gemma 3/4 custom tool calling tokens (#5033)

* fix: parse Gemma 3/4 custom tool calling tokens in parser

* test: cover Gemma tool call parsing

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix(agent): preserve bare email tool parity (#5075)

* fix(agent): confine glob literal lookups to the search root (#5010)

GlobTool resolves its search root through _resolve_search_root (which
confines it to the workspace or default allowlist), but the literal
fast-path joined the model-supplied pattern onto that root without
re-confining it. os.path.join lets an absolute pattern or one containing
../ escape the root, and normpath collapsed the .. segments, so glob
returned the absolute path of arbitrary host files once they existed --
an existence/path oracle that bypasses the confinement read_file,
write_file, grep, and ls all enforce.

Keep the literal lookup inside the root via a commonpath containment
check; an escaping literal falls through to the os.walk matcher, which
only ever yields paths under the root. Wildcard matching was already
confined.

* fix(agent): skip deny-listed sensitive files in glob (#5094)

* fix(llm): normalize OpenAI-compatible chat URLs

Normalize OpenAI-compatible chat URL shapes so base /v1 endpoints route to /v1/chat/completions while already-full chat endpoints remain idempotent.

Preserve native local Ollama routing for bare localhost:11434 endpoints, keep localhost:11434/v1 as OpenAI-compatible, and add focused regression coverage for provider detection, chat target URLs, and model listing from /v1.

Part of #541.

* fix(search): use generated query for chat mode web search #4547 (#4557)

* fix(search): use generated query for chat mode web search #4547

* style(search): tidy query generation call

---------

Co-authored-by: Alexandre Teixeira <alexandremagteixeira@gmail.com>

* fix(email): validate IMAP/SMTP ports instead of crashing with 500 (#4464)

The email-account endpoints coerced user-supplied ports with a bare int(data.get("imap_port") or 993), so a non-numeric port (e.g. "imap") raised ValueError and surfaced as an HTTP 500 in the create, update, and test-config endpoints.

Add a _coerce_port(value, default) -> (port, error) helper and use it in all three endpoints, returning the endpoints standard {"ok": False, "error": ...} response (matching the existing "name required" validation) instead of crashing. A blank or missing port still falls back to the default (993/465).

* fix(security): prevent ReDoS in LLM-output tool/think parsers (#4704)

* fix(security): prevent ReDoS in LLM-output tool/think parsers

The regexes that parse untrusted model output in text_helpers.py and
tool_parsing.py are delimiter-bounded with a lazy [\s\S]*? (or an
ambiguous (\s+[^>]*)?). Applied with re.sub/re.finditer over a whole
response, they degrade to O(n^2) when the closing delimiter is absent:
the engine rescans to end-of-string from every opener. Model output is
untrusted, so a prompt-injected or malicious model can stall the agent
loop with many unclosed openers (measured ~25s on a 60KB <thought flood).

- text_helpers.py: replace ambiguous <thought(\s+[^>]*)?> with
  <thought([^>]*)> (identical capture, no \s+/[^>]* overlap); skip the
  Gemma <|channel>...<channel|> subs when no <channel|> closer is present.
- tool_parsing.py: gate _TOOL_CALL_RE, _XML_TOOL_CALL_RE and _TOOL_CODE_RE
  (in parse_tool_blocks and strip_tool_blocks) on a cheap presence check
  for their closing delimiter. With no closer the regex cannot match, so
  skipping is equivalent; only the wasted O(n^2) rescan is removed.

Resolves CodeQL py/polynomial-redos #230, #231, #232, #233, #235, #236,
(its greedy [\s\S]*\Z is linear) and left untouched.

* fix(security): close ReDoS gaps in tool/think parsers from review

Addresses two review findings on the closer-guard approach:

- Whole-string "closer exists?" checks were bypassable: a stale closer
  before an opener flood, or a closer with no reachable inner `}`, kept
  the guard true while every opener still rescanned to end-of-string
  (O(n^2)). Replace the substring guards with `_iter_delimited`, a
  forward-only scan that pairs each opener with a *later* closer and
  stops once none is reachable (O(n)). `parse_tool_blocks` and
  `strip_tool_blocks` (via `_strip_delimited`) both use it for the
  [TOOL_CALL], <tool_call>/<function_call>, and <tool_code> formats.
  Verified equivalent to the original regexes on well-formed inputs.

- `<thought([^>]*)>` dropped the tag-name boundary and corrupted
  unrelated tags (`<thoughtful>` -> `<thinkful>`). Use `<thought(\s[^>]*)?>`:
  the single fixed `\s` keeps the pattern linear (no `\s+`/`[^>]*`
  overlap) while restoring the boundary; capture is byte-for-byte
  identical for real `<thought ...>` openers.

Adds regressions for stale-closer-before-opener, closer-present-without-
inner-brace, and the <thoughtful>/<thoughts> passthrough.

* fix(security): close Gemma channel ReDoS guard flagged in review

vdmkenny noted the same bypassable whole-string guard remained in
text_helpers.py: `if "<channel|>" in out.lower()` gating the Gemma
thought/response channel subs. A stale `<channel|>` before a
`<|channel>thought` opener flood keeps the guard true while every opener
still rescans to end-of-string (measured ~7.3s at 4k openers).

Replace it with `_sub_delimited`, the same forward-only scan used for the
tool-call parsers: pair each opener with a later closer, stop when none is
reachable (O(n)). Verified output-equivalent to the original capture regexes
on well-formed multi-channel inputs; the stale-closer case now runs in <2ms.
Adds a regression for stale-closer-before-opener on the Gemma path.

* fix(security): harden strip_think() think-tag ReDoS flagged in review

The earlier fixes hardened normalize_thinking_markup and the delimiter
scanners, but the production entrypoint strip_think() still ran
_THINK_CLOSED_RE / _THINK_ATTR_RE / _THINK_OPEN_RE (and the stray-tag
_THINK_TAG_RE) over untrusted model output. Those kept the same ReDoS
shapes: the lazy `<open>[\s\S]*?</close>` rescanned to end-of-string from
every opener, and `(?:\s+[^>]*)?` / `[^>]*` attribute scans ran to
end-of-string from every opener on a "many openers, no closer" flood. On
the prior head, malformed `<think` / `<thinking` / `<thought` floods took
6-14s through strip_think(). The shipped `<thought>` normalization had the
same residual: the single-opener case was linear but an opener flood was
still O(n^2) (~4.4s).

- Replace the lazy multi-pass _THINK_CLOSED_RE loop with the existing
  forward-only _sub_delimited scan (pair each opener with the first
  reachable closer, stop when none is reachable). One pass collapses
  sequential and nested blocks as before.
- Bound every opener/stray-tag attribute scan at `<` (`[^<>]` not `[^>]`)
  so a no-`>` opener flood can't drive a single match attempt to
  end-of-string. Identical capture for well-formed think/thought tags.
- email_helpers._strip_think: compute had_think from the single linear
  _THINK_TAG_RE instead of the lazy closed/open `.search()` calls, which
  had the same O(n^2) on the email reply/summary/extraction paths.

All flood variants now finish in <10ms (were 6-14s). Output verified
byte-for-byte identical to the prior implementation over a 34-case corpus
(nested, mismatched, attr, uppercase, Gemma, prose, prompt-echo). Adds
strip_think() timing regressions for malformed openers, opener floods
(all three tag names), the closed-opener flood, and the malformed-closer
flood.

* docs: trim verbose comments in think-tag ReDoS fix

* Support mobile enter for queued agent prompts

* Keep open document context for section edits

* Add bulk email attachment downloads

* Add hover labels to mini sidebar buttons

* fix(cookbook): treat local Windows as Windows for serve commands (#3975)

* fix(cookbook): prefer native llama-server on local Windows

* fix(cookbook): harden local llama-server launch commands

* fix(cookbook): build serve commands for selected target

* fix(cookbook): only block model launch on real port collisions (#4760)

* Fix #4507: only block model launch on real port collisions

Quick-run hardcoded port 8000 and never called _nextAvailablePort(), so
every launch collided. Both pre-launch guards (serve panel + quick-run)
were count-based and fired regardless of port.

- quick-run now auto-assigns a free port (8080 for llama.cpp)
- both guards parse the new port and only prompt on a real overlap,
  stopping only the colliding serve
- dialog reports the actual port instead of a hardcoded 8000

* refactor(cookbook): share _taskPort for port parsing; auto-assign llama.cpp port

Addresses review on #4760:
- _taskPort regex now matches --port= as well as --port (space)
- _nextAvailablePort and both launch guards reuse _taskPort instead of inline regex
- quick-run llama.cpp no longer pins 8080, so two can run concurrently

* fix(cookbook): _taskPort also parses -p; add port-parsing tests

Addresses review on #4760:
- _taskPort now matches -p <n> too, so it's the complete single reader
  (was missing the short flag that other readers already handle)
- add tests/test_cookbook_port_parsing_js.py covering the port forms,
  shared-reader reuse, and llama.cpp auto-assign

* test(cookbook): extract pure port helpers and test behavior

Addresses review on #4760: the prior tests only asserted source strings.
- extract portOf() and nextFreePort() into static/js/cookbookPorts.js
- cookbookRunning.js imports them; _taskPort and _nextAvailablePort delegate
- tests run the helpers via node and assert real behavior: all port forms
  (--port, --port=, -p, -p=), next-free-port skipping taken ports, and the
  same-port-clash / different-port-coexist outcome

---------

Co-authored-by: samy <samy@odysseus.boukouro.com>

* fix(agent): execute fenced tool calls with inline args and route bare email tool names (#3681)

* fix(agent): execute fenced tool calls with inline args and bare email tool names

Two bugs made local (Ollama) models unable to use email tools, leaving
raw fences like ```list_email_accounts {}``` in the chat:

1. _TOOL_BLOCK_RE required a newline right after the fence tag, so a
   tool call with args on the same line ("```list_email_accounts {}")
…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docker Pull requests that update docker code documentation Improvements or additions to documentation needs-validation Requires focused local/manual validation before merge ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: encrypt secrets at rest via SOPS (opt-in)

3 participants