bug: apply_template async endpoint makes blocking urllib calls — event loop stalls for up to 300s

[yasinBursali]

Bug Report: apply_template async endpoint makes blocking urllib calls — event loop stalls for up to 300s

Severity: High
Category: Error Handling / Blocking I/O
Platform: All (macOS, Linux, Windows/WSL2)
Confidence: Confirmed

Description

async def apply_template in routers/templates.py makes multiple synchronous urllib.request.urlopen calls directly inside the async handler body. Because these use blocking I/O with timeouts up to 300 seconds, they stall the asyncio event loop, preventing every other request in the process from being served for the duration. For a template that enables many services, the stall is cumulative (3 blocking calls × 300s timeout × N services if agent is unresponsive).

Affected File(s)

• dream-server/extensions/services/dashboard-api/routers/templates.py (L97–226, async def apply_template)

Root Cause

apply_template is declared async def, so FastAPI executes it directly in the event loop. Inside the handler, _call_agent_hook(), _call_agent(), and _call_agent_invalidate_compose_cache() — all imported from routers/extensions.py — use urllib.request.urlopen with _AGENT_TIMEOUT=300 or _AGENT_LOG_TIMEOUT=30. These are synchronous blocking calls. Blocking the event loop means no other async endpoint in the process responds until the urllib call returns.

Evidence

# routers/templates.py L97 — async handler
async def apply_template(template_id: str, api_key: str = Depends(verify_api_key)):
    ...
    for svc_id in template.get("services", []):
        ...
        # _call_agent_hook uses urllib.urlopen with _AGENT_TIMEOUT=300:
        _call_agent_hook(svc_id, "pre_start")
        start_ok = _call_agent("start", svc_id)   # also 300s timeout
        _call_agent_hook(svc_id, "post_start")
        _call_agent_invalidate_compose_cache()     # 30s timeout, inside lock

Platform Analysis

• macOS: Affected — FastAPI async event loop runs the same way on macOS. Hooks add bash-version check overhead on first call.
• Linux: Affected — same event loop model; host agent at 7710 is local but timeouts still block if agent is slow or briefly down.
• Windows/WSL2: Affected — same. host.docker.internal adds latency between Docker Desktop VM and WSL2.

Reproduction

1. Start dashboard API.
2. Call POST /api/templates/{template_id}/apply for a template with 3+ services.
3. Simultaneously call GET /health from another client.
4. Expected: health check responds immediately.
5. Actual: health check hangs for the full duration of the template apply (potentially minutes if agent is slow).

Impact

Entire dashboard API becomes unresponsive for all users during template apply. Status polling, log streaming, and GPU metric refreshes all freeze until the blocking call chain completes. The preview_template endpoint has the same class of issue with lighter calls.

Suggested Approach

Convert apply_template (and preview_template) to a regular def (non-async) so FastAPI routes them through the thread-pool executor, which isolates the blocking I/O from the event loop. This is the minimal-change fix and consistent with how the other mutation endpoints (enable_extension, disable_extension) are already defined.

---

Filed by automated Python auditor after full-sweep review of Python changes merged 2026-04-06 → 2026-04-11 on upstream/main @ c0600ca.

[yasinBursali]

Status check from integration test of PR stack Light-Heart-Labs#893–909

Environment: WSL2 / Ubuntu 24.04 / NVIDIA RTX 3070 Laptop / Tier 1
Branch tested: local/integration-test containing all 17 open yasinBursali PRs merged --no-ff onto Light-Heart-Labs/DreamServer@c0600ca
Install: bash install.sh --all --non-interactive --no-comfyui → phase 13, 17 healthy containers

Verdict

Structurally fixed by PR Light-Heart-Labs#909 — but unreachable at runtime due to a separate :ro mount conflict (see #331 filed during this test pass).

Evidence — the non-blocking guarantee is in place

PR Light-Heart-Labs#909 ports apply_template to wrap every blocking call in asyncio.to_thread. Verified two ways:

1. Unit tests pass under pytest 9 + pytest-asyncio (after installing the plugin in a user-local venv):

$ pytest tests/test_templates.py -q
...................                                                      [100%]
19 passed in 1.03s

All 19 tests in test_templates.py pass, including:

• test_apply_template_blocking_calls_run_in_to_thread — AST-walks the source of apply_template and asserts that every call to _call_agent_hook, _get_missing_deps_transitive, etc. appears inside an asyncio.to_thread(...). This is a structural regression-proof test, not a timing test — if a future dev removes wrapping, the test fails.

This is a strong correctness signal: the non-blocking guarantee is now structurally enforced rather than just hoped-for.

2. Source on the deployed binary (/home/rosenrot/dream-server/extensions/services/dashboard-api/routers/templates.py) confirms the AST proof's invariant: apply_template's helper closures _install_with_lock and _activate_with_lock are wrapped in await asyncio.to_thread(...), so the lock acquisition stays in the thread pool and never holds across an await boundary.

Cannot verify end-to-end at runtime (different bug blocks the path)

POST /api/templates/<id>/apply returns 500 Internal Server Error in <1 second for every template in the catalog, BUT the failure is not the blocking-urllib bug this issue is filed about. The failure is:

File "/app/routers/extensions.py", line 1032, in _activate_service
    os.rename(str(disabled_compose), str(enabled_compose))
OSError: [Errno 30] Read-only file system:
    '/dream-server/extensions/services/comfyui/compose.yaml.disabled'
    -> '/dream-server/extensions/services/comfyui/compose.yaml'

The async call chain executes (control reaches _activate_with_lock → _activate_service) and dies on EROFS — which means the asyncio.to_thread wrapping is correct AND the apply_template handler is no longer blocking. It just hits a separate bug introduced by the interaction between PRs Light-Heart-Labs#908 and Light-Heart-Labs#909.

Filed as #331. Once that's resolved, apply_template will both terminate correctly AND not stall the event loop. The fix this issue asks for is structurally complete; the next layer just needs to land.

Concurrent-load test (the original symptom in this issue's repro section)

Not directly run because the request now fails before it can stall the event loop. But the AST proof test makes the structural guarantee deterministic — the regression class in the repro section ("health check hangs for the full duration of the template apply") cannot occur for any code path that goes through apply_template after Light-Heart-Labs#909.

---

Filed during full-stack integration test of open PR stack Light-Heart-Labs#893–909 on Light-Heart-Labs/DreamServer@c0600ca3. Cross-referenced from #331.

[yasinBursali]

Addressed by PR Light-Heart-Labs#909 (currently draft, pending upstream review).