fix(agent): use native tools for workspace tasks

[uhhgoat]

Summary

Fixes Agent workspace file tasks for ChatGPT Subscription runs by preserving native Responses function-call/tool-output turns, enabling tool support for the subscription endpoint, and keeping process details behind the same disclosure-style UI used for thinking output.

Target branch

• This PR targets dev, not main. All PRs land in dev; main is curated by the maintainer at each release. If your PR is on main by accident, click "Edit" on this PR and change the base.

Linked Issue

Fixes #4968

Type of Change

• Bug fix (non-breaking - fixes a confirmed issue)
• New feature (non-breaking - adds new behaviour)
• Breaking change (changes or removes existing behaviour)
• Refactor / cleanup (behaviour unchanged)
• Documentation only
• CI / tooling / configuration

Checklist

• I searched open issues and open PRs - this is not a duplicate.
• This PR targets dev
• My changes are limited to the scope described above - no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough.

How to Test

1. Start Odysseus with the workspace override and open the app on http://127.0.0.1:7000.
2. In Agent mode with a ChatGPT Subscription model, run: Read the README in the workspace and tell me exactly what it says. Confirm the agent calls workspace tools and returns a clean final answer with process details collapsed.
3. Run: Create agent_ui_native_tool_test.txt in the workspace with one line: native UI tools worked. Confirm the file is written and the final answer reports success after tool evidence.
4. Run: List the workspace files and confirm whether agent_ui_native_tool_test.txt exists. Confirm the file appears in the listing and the visible final answer stays separate from process details.

Validation run locally after resolving conflicts:

git diff --check
python -m py_compile src/agent_loop.py src/tool_execution.py core/database.py routes/chatgpt_subscription_routes.py src/chatgpt_subscription.py src/llm_core.py
node --check static/js/chat.js
node --check static/js/chatRenderer.js
python -m pytest tests/test_agent_loop.py tests/test_chat_helpers.py tests/test_chatgpt_subscription_routes.py tests/test_llm_core_streaming.py tests/test_llm_core_temperature_reasoning.py
python -m pytest tests/test_agent_process_collapse_static.py::test_agent_process_has_collapsed_panel_styles tests/test_agent_rounds_exhausted.py::test_incomplete_no_tool_promise_gets_recovery_round tests/test_gpu_compose_standalone.py::test_nvidia_standalone_equals_base_plus_overlay tests/test_gpu_compose_standalone.py::test_amd_standalone_equals_base_plus_overlay tests/test_gpu_compose_standalone.py::test_amd_odysseus_adds_only_overlay tests/test_workspace_confine.py::test_final_answer_contract_prompt_is_injected

The focused pytest runs were executed in the Odysseus Docker image with the working tree mounted and passed: 155 passed, 1 warning and 6 passed, 1 warning.

Manual Agent smoke tests were run with both gpt-5.5 and gpt-5.3-codex-spark against the mounted workspace. The UI smoke was also run with gpt-5.3-codex-spark on the local Docker app.

Visual / UI changes - REQUIRED if you touched anything that renders

Anything that changes what the UI looks like - buttons, icons, padding, colors, fonts, spacing, layout, CSS, HTML, SVG, or any static/js/ module that draws to the DOM - needs all of the following. PRs that change rendering without these WILL be closed.

• Screenshot or short clip of the change in the running app, attached below. Mobile screenshot too if the change affects mobile.
• Style match: the change uses Odysseus's existing visual language. Specifically:
  • Reuse existing CSS variables (--red, --fg, --bg, --card, --border, etc.) - do not introduce new color values, font sizes, or spacing units.
  • Reuse existing button/input/card/border classes. Don't invent parallel styling.
  • No Unicode emoji in UI or code. Use inline SVG (matching the monochrome icon style already in static/index.html) or plain text.
  • Monospaced font (Fira Code) for primary UI text. Don't override.
  • Dark theme is the default; any light-mode work must be wired through the existing theme system, not hard-coded.
• No new component patterns. If a similar widget already exists in the app, extend it instead of writing a parallel one.
• I am not an LLM agent submitting a bulk PR. Issue Agent workspace file tasks can produce contradictory answers and expose process output #4968 was opened first and this is a focused bug fix, not a bulk submission.

Screenshots / clips

Agent README read - process collapsed

Agent workspace write - process collapsed

Agent workspace list/confirm - process collapsed

[ashvinctrl]

The workspace shell-write guard (_workspace_shell_write_block_reason) over-blocks a lot of legitimate read-only shell, which cuts against its own message ("Shell is still available for read-only diagnostics").

_WORKSPACE_SHELL_REDIRECT_RE ((?<!\d)>{1,2}(?![&|])) is a plain text match, so it can't tell a real redirect from a > that's a comparison operator, inside quotes, or a /dev/null sink. Ran the two regexes against common diagnostics with a workspace active:

== read-only / diagnostic commands ==
BLOCKED  awk '$3 > 100 {print $1}' data.csv
BLOCKED  cat data.json | jq '.items[] | select(.size > 5)'
BLOCKED  echo "use > to redirect"
BLOCKED  ls -la > /dev/null 2>&1
BLOCKED  grep -rn 'a -> b' src/
BLOCKED  python -c "print(1 > 0)"
ok       git log --oneline | head -20
ok       diff <(sort a.txt) <(sort b.txt)

awk/jq comparisons and > /dev/null are exactly the read-only diagnostics an agent leans on, so while a workspace is active the model keeps getting bounced to write_file for commands that write nothing.

Flip side: it's also trivially under-inclusive as a guard — the (?<!\d) lookbehind means a fd-prefixed redirect like cmd 1> out.txt writes a file and is NOT blocked, while the bare cmd > out.txt is. So as a hard barrier it leaks, and as a soft nudge it has too many false positives.

Reliably separating redirects from comparisons/quoted text really needs tokenization (shlex) rather than a regex over the raw string. If you want to keep it lightweight, at minimum special-case > /dev/null / >&, and consider only blocking when the redirect target resolves to a path inside the workspace — but the quoted->/awk cases will still bite a regex approach.

(Separately, this PR is bundling several independent things — run records, ChatGPT-subscription tools, the workspace guard — which will make it hard to land; might be worth splitting the guard out on its own.)

[uhhgoat]

Good catch. I pushed 2ca3c779 to replace the raw redirect regex with token-aware parsing via shlex, so quoted/comparison > no longer trips the workspace shell-write guard.

The guard now allows read-only diagnostics like awk/jq comparisons, quoted >, process substitution, /dev/null, and fd duplication, while still blocking redirects that write into the active workspace, including fd-prefixed forms like 1> out.txt.

I also added regression coverage for the examples you listed plus fd-prefixed redirects.

Validation:

docker compose build odysseus
python -m pytest tests/test_workspace_confine.py -q
python -m pytest tests/test_workspace_confine.py tests/test_tool_path_confinement.py -q

Focused Docker runs passed: 44 passed and 68 passed.

[alteixeira20]

Thanks for the detailed validation and follow-up on the workspace shell-write guard.

I started maintainer audit, but this is currently blocked by a real merge conflict against current dev:

• GitHub reports the PR as conflicting / dirty.
• Local merge probe also conflicts in src/tool_parsing.py.

The conflict appears to be from current dev now including the newer fenced-tool parser changes from #3681. This PR adds zero-arg get_workspace fenced-call support on the older parser shape, so the rebase should preserve that behavior inside the newer _fenced_tool_call / empty-fence handling model.

Expected rebase intent:

• Keep fix(agent): execute fenced tool calls with inline args and route bare email tool names #3681's parser/strip invariant: only fences that actually dispatch are stripped.
• Keep fix(agent): execute fenced tool calls with inline args and route bare email tool names #3681's metadata safety: bash / python same-line metadata remains visible and inert.
• Add this PR's intended zero-arg behavior for get_workspace, with regression coverage.
• Rerun the focused agent/workspace/native-tool validation after rebasing.

Once the branch is clean against current dev, I can continue the audit.

[uhhgoat]

Resolved in 541727e by merging current dev into this branch and resolving the parser/workspace test conflicts. I kept the #3681 fenced-parser invariant intact: same-line bash/python metadata remains inert, non-dispatching fences stay visible, and only actual tool calls are stripped. This PR's zero-arg fenced get_workspace path is preserved and now has parse/strip regression coverage. I also kept the workspace confinement/shell-guard tests alongside the incoming dev glob coverage.\n\nValidation run in Docker:\n- docker compose build odysseus\n- python -m pytest tests/test_fenced_inline_args.py tests/test_fenced_invoke_no_raw_xml.py tests/test_workspace_confine.py tests/test_tool_path_confinement.py tests/test_agent_rounds_exhausted.py -q\n\nResult: 112 passed, 2 dependency deprecation warnings.

[alteixeira20]

Thanks for rebasing this and resolving the parser conflict. The previous conflict blocker is gone, GitHub checks are green, local merge probe is clean, and the focused validation is strong.

I am requesting changes because the new durable agent-run record path appears internally inconsistent.

The new AgentRunRecord model and agent_run_records.finish() support these fields:

partial_chars
assistant_message_id

latest_for_session() also exposes partial_chars, and the unit test verifies that finish(..., partial_chars=42, assistant_message_id="assistant-msg-1") persists those values.

However, the real detached-run manager path in src/agent_runs.py currently calls:

agent_run_records.finish(
    run.record_id,
    status=run.status,
    stop_reason=run.stop_reason,
    error=run.error,
    event_count=len(run.buffer),
    last_event_type=_last_event_type(run.buffer[-1]) if run.buffer else "",
)

So real detached runs never pass partial_chars or assistant_message_id.

I verified this with a direct probe against the PR worktree. The actual finish payload was:

{'status': 'done', 'stop_reason': '', 'error': '', 'event_count': 4, 'last_event_type': 'done'}

and the probe reported:

MISSING_DURABLE_FIELDS: partial_chars, assistant_message_id

Please either:

1. populate these fields on the real detached-run path, ideally by deriving assistant_message_id from the streamed message_saved event and computing a meaningful partial_chars, with integration coverage for agent_runs.start(...); or
2. remove/de-scope those fields from the durable run contract/API/tests if they are intentionally unused.

Given this PR adds a DB table and user-visible durable run/resume behavior, the stored lifecycle record should not expose fields that are only populated in unit-level direct calls but never by the production path.

[uhhgoat]

Addressed in 1510c7f. The detached-run production path now derives durable finish fields from the actual buffered SSE stream before calling agent_run_records.finish():\n\n- assistant_message_id is captured from the streamed message_saved event.\n- partial_chars is computed from non-thinking visible delta chunks, with agent_final.text as a fallback when no deltas were emitted.\n- latest_for_session now includes assistant_message_id alongside partial_chars.\n\nI also updated the real agent_runs.start(...) regression coverage so the detached-run manager test now fails if those fields are omitted, and added coverage for the agent_final fallback path.\n\nValidation after Docker rebuild:\n\n ext\ndocker compose build odysseus\npython -m py_compile src/agent_runs.py src/agent_run_records.py tests/test_agent_run_records.py\npython -m pytest tests/test_agent_run_records.py tests/test_compare_stop_disconnect_poll.py -q\n\n\nResult: 21 passed, 1 existing SQLAlchemy deprecation warning.

[uhhgoat]

Resolved the new dev conflicts in 99d2aa41 by merging current origin/dev into this branch.

Conflict files resolved:

• routes/chat_routes.py
• src/agent_loop.py
• src/tool_parsing.py
• static/js/chat.js
• static/js/sessions.js

Resolution notes:

• Kept the zero-arg fenced get_workspace support and preserved dev's plain ui_control open_panel rescue parser.
• Combined agent save behavior so real final text still goes through prepare_agent_response_for_save(...), while dev's tool-only completion fallback still saves Done. when only tool events exist.
• Kept both workspace/file-mutation routing and dev's newer code/script intent routing.
• Preserved round on streamed tool_output events and dev's document-tool output metadata.
• Kept agent process-collapse rendering while preserving the duplicate-footer guard.
• Kept durable terminal run rendering in session polling.

Local validation:

git diff --cached --check
python -m py_compile routes/chat_routes.py src/agent_loop.py src/tool_parsing.py src/agent_runs.py src/agent_run_records.py
node --check static/js/chat.js
node --check static/js/sessions.js
node --check static/js/chatRenderer.js
python -m pytest tests/test_agent_run_records.py tests/test_compare_stop_disconnect_poll.py tests/test_agent_process_collapse_static.py tests/test_fenced_inline_args.py tests/test_fenced_invoke_no_raw_xml.py tests/test_workspace_confine.py tests/test_tool_path_confinement.py tests/test_agent_rounds_exhausted.py tests/test_plain_ui_control_open_panel.py -q
docker compose build odysseus

Focused pytest result: 140 passed, 2 existing dependency deprecation warnings.

GitHub now reports the head as 99d2aa41; checks are running, and the PR state is no longer DIRTY on my latest check.

[uhhgoat]

Conflict resolution update for #4969:

I traced the conflict source to the dev update from #5283 (c67deaa, "Merge pull request #5283 from pewdiepie-archdaemon/sync-main-into-dev-20260707"), which brought in the main-sync checkpoint 017903d after #4969's merge base (5e9b415). That overlap touched the same agent/chat streaming paths as this PR:

• routes/chat_routes.py
• src/agent_loop.py

I resolved this by merging current dev into the PR branch with a normal merge commit, not by rebasing or force-pushing:

• e87ee72 - Merge dev into native tool calling
• 3edf426 - Fix final answer contract scope after dev merge
• 6edad9c - Align final answer contract provider message

Resolution notes:

• Preserved this PR's native-tool/workspace intent and cancellation/save-path behavior.
• Kept the newer dev request-grounding/contextual-web-follow-up logic in routes/chat_routes.py.
• Kept thinking metadata when saving agent fallback responses by carrying thinking_response through copied metrics dictionaries instead of mutating or dropping the merged metrics.
• Preserved the Odysseus finetune prompt routing added on dev while still injecting the final-answer contract before the latest user message.
• Fixed the CI-exposed low-signal direct-answer path by moving the final-answer contract into a shared helper rather than referencing a prompt-builder local.
• Aligned that contract message with the existing provider-call expectation that _protected is not present in the captured provider payload.

Validation:

• Local: git diff --check
• Local: git diff --cached --check
• Local: node --check static/js/chat.js
• Local: node --check static/js/chatRenderer.js
• GitHub Actions on head 6edad9c: Python syntax, JS syntax, full Python tests, actionlint, gitleaks, hadolint, zizmor, dependency-review, pip-audit, and Trivy advisory all passed.

GitHub now reports the PR as mergeable: MERGEABLE; the remaining mergeStateStatus: BLOCKED is not a merge-conflict state after these checks.

[ashvinctrl]

The shlex rewrite sorted out the redirect false-positives from before — thanks for that. Noticed one sibling gap in the same guard though: _WORKSPACE_SHELL_MUTATION_CMD_RE is still a plain text search over the whole command, so it's got the exact token-unaware behavior the redirect path just moved away from.

Over-blocks a few read-only commands (which is the thing the guard's own message says it won't do):

grep -E 'mv|cp' log.txt      # trips on the |cp inside the pattern
awk '/mv|cp/' file
echo "a;cp b"

And it under-blocks the opposite way, since it only anchors on ^ / ;&| — anything in a subshell or command substitution slips past:

(cp secret.txt out.txt)
$(mv a.txt b.txt)
`cp a b`
{ cp a b; }

You've already got _workspace_shell_tokens right there, so running the mutation check through it too — flag cp/mv/tee/etc. only when it lands in command position after tokenizing — would close both directions the same way the redirect check now does. Nothing security-critical (shell's confined either way), mostly just guard consistency + not blocking a legit grep -E 'a|cp'.

[uhhgoat]

Thanks, good catch. Addressed in 27de2479 by moving the workspace shell mutation-command check onto the same token-aware path as redirects.

The guard now uses _workspace_shell_tokens(...) to flag cp/mv/tee/touch/rename/copy and sed -i / perl -pi / awk -i only when they appear in command position. The old raw regex is retained only as a fallback when tokenization fails.

Added regression coverage for the cases you called out:

• allowed: grep -E 'mv|cp' log.txt, awk '/mv|cp/' file, echo "a;cp b"
• blocked: (cp secret.txt out.txt), $(mv a.txt b.txt), `cp a b`, { cp a b; }

Validation:

docker compose build odysseus
docker run --rm -e PYTHONDONTWRITEBYTECODE=1 -v "${PWD}\tests:/app/tests" --entrypoint pytest odysseus-odysseus tests/test_workspace_confine.py -q
docker run --rm --entrypoint python odysseus-odysseus -m py_compile src/tool_execution.py
git diff --check

Focused pytest result: 60 passed, 2 warnings.