bug: _compose_restart_llama_server ignores subprocess return codes — silent model switch failures on native Linux/macOS

[yasinBursali]

Bug Report: _compose_restart_llama_server ignores subprocess return codes — silent model switch failures on native Linux/macOS

Severity: Medium
Category: Error Handling
Platform: Linux (systemd agent), macOS (launchd agent)
Confidence: Confirmed

Description

_compose_restart_llama_server() in dream-host-agent.py calls subprocess.run for docker compose stop, docker compose up, docker restart, and docker start without checking any return codes. If any Docker operation fails (e.g., permission issue, missing compose file, daemon error), the function returns None silently. The caller _do_model_activate then proceeds to the health-check loop, which times out after 5 minutes before rolling back — a very slow silent failure path.

Affected File(s)

• dream-server/bin/dream-host-agent.py (L1444–1478, _compose_restart_llama_server)

Root Cause

All subprocess.run calls in the function use capture_output=True but do not inspect .returncode or raise on non-zero:

# dream-host-agent.py L1467-1471 (non-AMD path, compose available):
subprocess.run(["docker", "compose"] + compose_flags + ["stop", "llama-server"],
               cwd=str(INSTALL_DIR), capture_output=True, timeout=120)
subprocess.run(["docker", "compose"] + compose_flags + ["up", "-d", "llama-server"],
               cwd=str(INSTALL_DIR), capture_output=True, timeout=300)
# No returncode check — function returns None regardless

If compose stop fails, compose up still runs. If compose up fails, the model activation flow proceeds to a 5-minute health check loop before declaring failure and rolling back.

Platform Analysis

• macOS: Affected — this is the native-host path (not _in_container). macOS Docker Desktop edge cases (daemon restart, resource pressure) can cause compose failures.
• Linux: Affected — primary path for systemd agent installs. Docker daemon errors (disk full, cgroup limits) would hit this path.
• Windows/WSL2: Not affected — Windows uses the _in_container path (_recreate_llama_server), not this function.

Reproduction

1. Run host agent natively on Linux (systemd) with a valid install.
2. Temporarily make docker compose up fail (e.g., set INSTALL_DIR to a bad path or break .compose-flags).
3. Activate a model via POST /v1/model/activate.
4. Expected: immediate error response.
5. Actual: agent waits full 5-minute health check loop, then rolls back with "Health check failed" — the actual Docker error is never surfaced in the response.

Impact

On native Linux/macOS installs, a Docker-layer failure during model switch produces a 5-minute hang instead of an immediate error. The original model may or may not be running after rollback. Operators see "health check failed" with no indication of the real cause (Docker error output is silently discarded).

Suggested Approach

Check .returncode after each subprocess.run call and return an error (or raise) immediately if non-zero. The stderr from the failed command should be included in the error detail passed back to _do_model_activate so the operator can diagnose the Docker-layer failure without reading logs.

---

Filed by automated Python auditor after full-sweep review of Python changes merged 2026-04-06 → 2026-04-11 on upstream/main @ c0600ca.

[yasinBursali]

Status check from integration test of PR stack Light-Heart-Labs#893–909

Environment: WSL2 / Ubuntu 24.04 / NVIDIA RTX 3070 Laptop / Tier 1
Branch tested: local/integration-test containing all 17 open yasinBursali PRs merged --no-ff onto Light-Heart-Labs/DreamServer@c0600ca
Install: bash install.sh --all --non-interactive --no-comfyui → phase 13, 17 healthy containers

Verdict

FIXED by PR Light-Heart-Labs#906 (structural — the wrapping is in place. Recovery path exercised at runtime; raise-on-non-zero-exit path not directly manufactured.)

Evidence — deployed binary wraps subprocess calls and raises on failure

/home/rosenrot/dream-server/bin/dream-host-agent.py after fresh install. _compose_restart_llama_server now defines an inner _run helper that captures stderr and raises RuntimeError on non-zero exit, then branches on gpu_backend:

def _run(argv, timeout):
    result = subprocess.run(
        argv, cwd=str(INSTALL_DIR),
        capture_output=True, text=True, timeout=timeout,
    )
    if result.returncode != 0:
        raise RuntimeError(
            f"{' '.join(argv[:3])} failed (exit {result.returncode}): "
            f"{(result.stderr or '').strip()[:300]}"
        )

if gpu_backend == "amd":
    # Lemonade: restart preserves cached binary, reads models.ini on boot
    if compose_flags:
        _run(["docker", "compose"] + compose_flags + ["restart", "llama-server"], 300)
    else:
        _run(["docker", "restart", "dream-llama-server"], 300)
else:
    # llama.cpp: recreate to pick up new GGUF_FILE from .env
    if compose_flags:
        _run(["docker", "compose"] + compose_flags + ["stop", "llama-server"], 120)
        _run(["docker", "compose"] + compose_flags + ["up", "-d", "llama-server"], 300)
    else:
        _run(["docker", "stop", "dream-llama-server"], 120)
        _run(["docker", "start", "dream-llama-server"], 300)

Every subprocess.run is now routed through _run, so any non-zero exit will raise RuntimeError and propagate to _do_model_activate instead of dropping the failure on the floor and letting the 5-minute health-check loop time out.

The PR also adds the AMD-specific restart path (Lemonade preserves cached binary on restart) vs the non-AMD stop + up -d path (llama.cpp needs recreate to re-read GGUF_FILE). This is a behavior change beyond the originally-filed bug — note that on AMD hosts the compose stop && compose up -d pattern is replaced with compose restart, which IS the correct call for Lemonade.

Runtime exercise

I killed dream-llama-server and triggered a model activate to drive the function:

$ docker kill dream-llama-server
dream-llama-server

$ curl -X POST -H "Authorization: Bearer $DASHBOARD_API_KEY" \
    http://127.0.0.1:3002/api/models/qwen3.5-9b-q4/load
{"status":"activated","model_id":"qwen3.5-9b-q4"}
# elapsed: 53s (most of this is llama-server boot + healthy wait, not the docker calls)

Host-agent journal during the activation:

llama-server restarted via compose (backend: nvidia)
Waiting for llama-server health at http://127.0.0.1:11434/health
Health check attempt 1: {"error":{"message":"Loading model","type":"unavailable_error","code":503}}
llama-server healthy after 6 attempts
"POST /v1/model/activate HTTP/1.1" 200 -

The _run helper executed cleanly for both compose stop (no-op since I'd already killed the container — compose stop of an already-stopped service still exits 0) and compose up -d (which recreated the container). No RuntimeError was raised because no docker call returned non-zero. The recovery path is healthy.

What was NOT tested

• The actual non-zero-exit branch — I did not manufacture a real docker failure (e.g., by breaking the compose file, stopping the docker daemon, or removing the binary from PATH). To do that safely without breaking the running stack would have required an isolated-state setup that the timebox didn't allow. The wrapping is verified by code inspection on the deployed binary; the raise path is unit-tested at the source level by PR fix(host-agent): surface docker failures in _compose_restart_llama_server Light-Heart-Labs/DreamServer#906's tests but is not exercised end-to-end in this run.
• The AMD/Lemonade restart branch (no AMD hardware in this environment).

The structural fix is correct and minimal: every subprocess.run is now routed through a returncode-checking helper. The original symptom in this issue's repro section ("agent waits full 5-minute health check loop, then rolls back with 'Health check failed'") cannot occur because the docker error now raises immediately and propagates to the caller via RuntimeError.

Pre-requisite worth flagging

Same as #313 / #315: the installer didn't restart dream-host-agent.service after rewriting the binary (see #334). Without a manual restart, the OLD host-agent process from before the install was still serving the unwrapped subprocess.run calls — making the fix appear missing at runtime. Critical context for any QA pass on host-agent PRs.

---

Filed during full-stack integration test of open PR stack Light-Heart-Labs#893–909 on Light-Heart-Labs/DreamServer@c0600ca3. Cross-referenced from #334.

[yasinBursali]

Follow-up: manufactured-failure test confirms the raise-on-non-zero path

In my earlier comment I noted that the wrapping was in place and the recovery path was healthy, but I'd not actually exercised the raise-on-non-zero branch under a real docker failure. Just ran that test now.

Test

1. Stack healthy, all 17 containers up
2. Temporarily rename docker-compose.base.yml so docker compose -f docker-compose.base.yml stop llama-server fails immediately with "file not found"
3. Trigger model activate via POST /api/models/qwen3.5-9b-q4/load (the SAME currently-loaded model, so .env mutation is a no-op and any rollback is safe)
4. Restore the compose file immediately after

Result

$ mv ~/dream-server/docker-compose.base.yml ~/dream-server/docker-compose.base.yml.testbak
$ time curl -X POST -H "Authorization: Bearer $KEY" \
    http://127.0.0.1:3002/api/models/qwen3.5-9b-q4/load
{"detail":"Model activation failed: docker compose -f failed (exit 1): open /home/rosenrot/dream-server/docker-compose.base.yml: no such file or directory"}
real    0m0.149s
$ mv ~/dream-server/docker-compose.base.yml.testbak ~/dream-server/docker-compose.base.yml

Host-agent journal:

"POST /v1/model/activate HTTP/1.1" 500 -

What this proves about PR Light-Heart-Labs#906

• Failure surfaces in <1 second, not after the 5-minute health-check timeout the original bug describes.
• Docker's stderr is captured and propagated — the response includes the literal open .../docker-compose.base.yml: no such file or directory so the operator can diagnose the root cause without grepping logs.
• The error message is formatted by _run — the prefix docker compose -f failed (exit 1): is the exact format from the _run helper added by fix(host-agent): surface docker failures in _compose_restart_llama_server Light-Heart-Labs/DreamServer#906.
• The error reaches the dashboard-api response unmodified — {"detail":"Model activation failed: docker compose -f failed (exit 1): open ..."} shows _do_model_activate caught the RuntimeError and surfaced it as a 500 with the message intact.
• The stack stayed healthy through the test — 17/17 containers still up after restore. The failed activate properly aborted without leaving partial state.

The original symptom in this issue's body — "agent waits full 5-minute health check loop, then rolls back with 'Health check failed' — the actual Docker error is never surfaced" — is decisively gone. The fix works under real docker failure, not just under code inspection.

Caveat

I tested via the non-AMD branch (NVIDIA host). The AMD/Lemonade restart path uses the same _run helper but I couldn't manufacture a failure on AMD hardware (none here). The wrapping is identical for both branches in source, so I'm confident the AMD path raises the same way.

Status

PR Light-Heart-Labs#906's fix holds, end-to-end. This issue can stay closed with high confidence.

---

Verified during full-stack integration test of open PR stack Light-Heart-Labs#893–909.

[yasinBursali]

Addressed by PR Light-Heart-Labs#906 (currently draft, pending upstream review).