Containerized/wrapped provider agents (podman exec / PATH-shim) can't initialize: host-path prompt files, unhandled slow startup dialogs, herdr agent_status stuck 'unknown'

[arikon]

Summary

When a provider agent (e.g. claude_code) is executed inside a container — via a PATH-shadow shim that does podman exec <task-container> claude "$@" on the host, so CAO/herdr run on the host and the agent process lives in the container — CAO cannot complete terminal initialization. The agent binary itself runs fine and completes real turns, but three assumptions in the provider/herdr layer break for wrapped (containerized/remote) execution. I got a full turn to complete only by manually driving the startup dialogs and delivering the task; CAO's own auto-init/auto-delivery never succeeds.

This is a "run agents in isolated containers" use case (one container + worktree per task) on a shared host; the shim is what makes CAO's host-side provider detection resolve (shutil.which("claude") → shim). Filing so the wrapped-execution path can be supported (or documented).

Environment

• CAO: git main (also repro on v2.2.0), --terminal herdr
• herdr: 0.7.x, host-side, session cao
• Agent: Claude Code 2.1.204, launched inside a rootless-podman container via podman exec -it -e IS_SANDBOX=1 -w /home/.../arcadia at-<task> claude "$@"
• Host: Linux, rootless podman 3.4.4

Blockers (in order hit)

1. Host-written --append-system-prompt-file / --mcp-config are inaccessible to a containerized agent

claude_code._build_claude_command() writes the profile system prompt and MCP config to host temp files under ~/.aws/cli-agent-orchestrator/tmp/<id>.prompt / .mcp.json and passes those host paths to the claude command. When claude runs in a container, that path does not exist there:

claude --dangerously-skip-permissions --append-system-prompt-file /home/<user>/.aws/cli-agent-orchestrator/tmp/<id>.prompt --mcp-config .../<id>.mcp.json --strict-mcp-config
Error: Append system prompt file not found: /home/<user>/.aws/cli-agent-orchestrator/tmp/<id>.prompt

claude exits immediately → herdr never registers an agent → init times out.

Workaround (integration side): bind-mount ~/.aws/cli-agent-orchestrator/tmp into the container at the same path. Works, but couples the container to CAO's host layout. A CAO-side option to pass the system prompt/MCP config inline (already shlex-escaped), or to make the temp dir configurable, would remove the host-filesystem coupling for wrapped execution.

2. Startup trust / bypass-permissions dialogs are not auto-accepted for slow-starting agents

After (1) is worked around, claude starts but shows two interactive dialogs: the workspace-trust dialog ("Yes, I trust this folder") and the --dangerously-skip-permissions bypass confirmation ("Yes, I accept"). _handle_startup_prompts() is meant to answer both (its TRUST_PROMPT_PATTERN / BYPASS_PROMPT_PATTERN do match the current wording), but it runs once for startup_prompt_handler_timeout (~20 s) before wait_until_status, and a containerized claude renders those dialogs after that window (cold-ish node start + gateway connect). Result: Startup prompt handler timed out, dialogs left unanswered, then Claude Code initialization timed out after 60s.

Manually sending Enter (trust) then Down+Enter (bypass) via herdr pane send-keys cleared both and claude reached its REPL normally. Suggestion: keep answering trust/bypass prompts during the wait_until_status wait (not only in a fixed pre-window), so agents that start slower than the handler timeout are still handled.

3. get_native_status maps herdr agent_status="unknown" → ERROR, blocking init/inbox for wrapped agents

HerdrBackend.get_native_status() maps herdr agent_status="unknown" to TerminalStatus.ERROR ("pane exists but no agent registered yet"). For a containerized agent the pane's foreground process is podman exec …; herdr's native agent detection never registers the nested claude, so agent_status stays "unknown" permanently → CAO reports ERROR → wait_until_status({IDLE, COMPLETED}) never succeeds → init fails and inbox auto-delivery is blocked. Even with blockers (1) and (2) resolved and claude's REPL fully up (verified: banner + bypass permissions on + a real turn), agent_status remained "unknown".

"unknown" at startup is arguably transient rather than a terminal error; mapping it to ERROR (instead of e.g. a not-yet-ready state, or falling through) turns "herdr can't classify this pane" into a hard init failure. This is the core blocker for any agent herdr can't natively introspect (containerized / remote / wrapped).

Repro sketch

1. Put a wrapper named claude on the CAO host PATH that execs podman exec -it -e IS_SANDBOX=1 -w <container-workdir> <container> claude "$@".
2. cao launch --agents <profile-with-system-prompt> --provider claude_code --headless --yolo --working-directory <dir> "<task>".
3. Observe: Append system prompt file not found in the pane (herdr pane read), then—after bind-mounting the temp dir—Startup prompt handler timed out → Claude Code initialization timed out after 60s; herdr pane get shows agent_status: "unknown" throughout.

What already works

The agent and turn themselves are fine: driving the two dialogs and delivering the task by hand via herdr pane send-text/send-keys produced a correct, completed turn (tool use + file write) inside the container. Only CAO's automatic init/status/delivery is blocked by the three items above.

Happy to provide full logs or test a patch.

[arikon]

Update — a working (if heavyweight) workaround for the containerized case, in case it helps others or informs a fix:

Instead of host-herdr + podman exec claude (where the pane foreground is podman, so herdr never registers the agent → agent_status stuck unknown → mapped to ERROR), run herdr inside the container and bridge its socket out to the host:

1. Bind-mount the herdr session dir container↔host at the same path (~/.config/herdr/sessions/<session>), and run herdr --session <session> server inside the container (with HOME set so the socket lands on the bind path). CAO's HerdrBackend._ensure_session_running() then finds the existing socket and transparently drives the container's herdr — verified herdr --session <s> workspace list from the host returns the container server's state.
2. herdr integration install claude inside the container so the pane's native claude reports state via the hook → agent_status correctly becomes idle/working (no longer unknown). This resolves blocker #3.
3. Pre-seed skipDangerousModePermissionPrompt: true and per-project hasTrustDialogAccepted/hasCompletedProjectOnboarding in the container's claude config so the startup dialogs never render (avoids the slow-start timing race in blocker #2).
4. Pass --working-directory as the container path (the agent runs in the container), not the host path — otherwise herdr can't resolve cwd inside the container and falls back to $HOME.

With that, cao launch reaches a registered terminal, agent_status is detected, and a real turn completes and writes to the workspace. Two rough edges remain: the initial cao launch <message> didn't always land at the REPL (re-delivery via herdr pane send-text works), and this ties one herdr session to one container.

Still think the cleanest upstream improvement would be to (a) not map transient unknown straight to ERROR, and (b) keep answering trust/bypass prompts during the init wait rather than only in a fixed pre-window — both would make wrapped/slow-start agents work without the in-container herdr dance.

[arikon]

Root-cause follow-up (isolated with a clean test): the reason the herdr backend reports agent_status: unknown for a containerized agent is that herdr gates agent detection on the pane's foreground process, not on rendered screen content.

For a podman exec -it … claude pane, pane.process_info reports the foreground as podman (the real claude runs in the container's PID namespace, invisible to the host herdr). So herdr never assigns a claude agent label and its screen-manifest rules are not applied — even though the claude REPL is fully rendered and readable in the pane.

Isolated (rules out earlier confounders):

• vanilla claude at a clean REPL via podman exec → unknown
• OMC-wrapped claude at a clean REPL via podman exec → unknown (so it's not the custom statusline)
• native claude in a herdr pane (no wrapper) → detected fine (control)

Delta is exactly the podman exec wrapper / foreground process, confirmed via pane.process_info (name=podman).

Implication: for any wrapped/remote agent execution (podman/docker/ssh exec), herdr's native status detection can't work by construction, so CAO's get_native_status will always see unknown there. This reinforces the two suggested fixes: (a) don't map transient unknown → ERROR, and (b) allow status to come from explicit pane.report_agent reports (custom lifecycle hooks) as an authority — I verified pane.report_agent --state working|idle|blocked does override for such a pane.

[haofeif]

@arikon agreed this is a valid wrapped/containerized provider support gap.

The three blockers line up with CAO today: Claude profile decomposition writes host temp files, startup prompt handling only runs in a fixed pre-readiness window, and herdr agent_status=unknown is treated as ERROR.

One important implementation detail: changing unknown -> None is necessary but not sufficient. On herdr, CAO skips FIFO/pipe-pane output, so the fallback buffer is normally empty. If native status is unavailable, status detection needs to read pane history on demand, or wrapped agents need to provide explicit lifecycle state via pane.report_agent.

For Claude prompt/MCP config, inline flags are available, but MCP inline config may expose env values in process args/history. I’d treat inline prompt as straightforward, and choose between inline MCP vs configurable/mounted runtime temp files based on security/usability.

I’d keep this open as a valid P2 support gap for wrapped/containerized/remote agents, not a normal host-native provider regression.

@anilkmr-a2z would you like to review

[anilkmr-a2z]

Investigation Findings

Root Cause

Three independent blockers prevent provider agents from initializing when running inside containers or wrappers (podman exec, docker exec, nerdctl, ssh, devcontainer exec):

1. Host-path coupling — claude_code.py writes host temp files (system prompt, MCP config) and passes host paths via CLI flags. In-container agent can't access host filesystem → immediate exit.

2. Fixed startup prompt window — _handle_startup_prompts() runs only during a fixed ~20s window before readiness wait. Cold containers render trust/bypass dialogs after this window → dialogs left unanswered, agent hangs.

3. unknown → ERROR mapping (universal blocker) — herdr_backend.py maps unknown agent_status to TerminalStatus.ERROR. For wrapped commands, herdr's foreground process is the wrapper (not the agent CLI), so herdr never registers the nested agent. Status resolution returns ERROR immediately with no buffer fallthrough → init always fails.

Fix

1. Status mapping fix — Map unknown → None (not ERROR). Added staleness-aware resolution: IDLE if not dispatched, PROCESSING if fresh (<120s), ERROR only if stale (>120s).

2. Container-aware path translation — Added _translate_path() on BaseProvider with longest-prefix-match substitution. New ContainerConfig model with path_maps on AgentProfile. Users declare volume mount mappings in their profile.

3. Idle-gap prompt handling — Changed timeout semantic from "total window" to "idle gap between consecutive prompts." Timer resets each time a prompt is found. Outer cap from provider_init_timeout. Updated all 3 providers (claude_code, kimi_cli, antigravity_cli).

4. Per-profile init timeout — Added optional provider_init_timeout on AgentProfile so container profiles can declare longer init times (e.g., 180s) without changing global config.

Testing

• 6 integration tests in test_container_wrapped.py (full lifecycle, no Docker required)
• Unit tests for each status matrix row, path translation, and idle-gap behavior
• 1076 tests passing across providers + backends

Draft PR incoming.

[anilkmr-a2z]

Update: Fix implemented, PR incoming

All three blockers identified in the investigation comment have been addressed:

Blocker	Fix
unknown → ERROR mapping kills wrapped processes	Staleness-aware resolution: unknown → IDLE (not dispatched), PROCESSING (fresh <120s), or ERROR (stale >120s)
Host temp paths invisible inside containers	_translate_path() on BaseProvider with longest-prefix-match; new ContainerConfig/ContainerPathMap models on AgentProfile
Fixed 20s prompt window too short for cold containers	Idle-gap semantic (timer resets on each prompt found, capped by provider_init_timeout) across all 3 providers

Test coverage: 6 new integration tests in test_container_wrapped.py + unit tests for each fix path. 2088 tests passing across providers, backends, and services.

Draft PR coming shortly.