Host agent rejects all built-in extensions from API management — CORE_SERVICE_IDS misused as blocklist

[yasinBursali]

Problem

The host agent's validate_service_id() at bin/dream-host-agent.py:291-305 rejects any service in CORE_SERVICE_IDS with HTTP 403 "Cannot manage core service".

The problem: CORE_SERVICE_IDS (loaded from config/core-service-ids.json) contains all 20 built-in services, including optional extensions like tts, whisper, comfyui, n8n, qdrant, embeddings, searxng, privacy-shield, perplexica, litellm, token-spy, ape, dreamforge, langfuse, openclaw, opencode.

Only 4 of these are truly "always on" base-compose services: llama-server, open-webui, dashboard, dashboard-api. The other 16 are built-in extensions that users should be able to enable/disable/start/stop via the API.

Impact

The dashboard API cannot manage 16 built-in extensions that it knows about:

• Cannot start them via `POST /v1/extension/start`
• Cannot stop them via `POST /v1/extension/stop`
• Cannot get logs via `POST /v1/extension/logs`

This blocks:

• Any "Enable" or "Start" button in the dashboard UI for built-in extensions
• PR feat(extensions): service dependency visibility + lifecycle hooks Light-Heart-Labs/DreamServer#877 (extension-deps-hooks): auto_enable_deps=True can't enable built-in deps
• Any template feature (PR 2 service templates): templates that reference built-in extensions get 403
• The entire mental model of "dashboard API as complete extension manager"

Reproduction

1. Fresh DreamServer install
2. Disable a built-in extension via CLI: `dream disable tts`
3. From the dashboard, try to re-enable it (if it appears in the catalog)
4. Expected: extension starts
5. Actual: Host agent returns 403 "Cannot manage core service: tts"

Root Cause

The JSON file is misnamed. core-service-ids.json was originally a whitelist of "valid service IDs the agent knows about" but got repurposed as a blocklist of "services the API is not allowed to touch". These are different concepts:

• Whitelist for input validation: all known built-in service IDs
• Blocklist for protection: only truly-always-on base-compose services

Suggested Fix

Introduce two separate constants:
```python

Whitelist: all known built-in service IDs (for input validation)

KNOWN_BUILTIN_SERVICES = load_known_services("config/core-service-ids.json")

Blocklist: services that should never be managed via API (always on)

ALWAYS_ON_SERVICES = frozenset({"llama-server", "open-webui", "dashboard", "dashboard-api"})
```

Then in `validate_service_id()`:
```python
if sid in ALWAYS_ON_SERVICES:
json_response(handler, 403, {"error": f"Cannot manage always-on service: {sid}"})
return None

Valid service: either a known built-in OR a user extension with a manifest

if sid not in KNOWN_BUILTIN_SERVICES and not (USER_EXTENSIONS_DIR / sid).is_dir():
json_response(handler, 404, {"error": f"Extension not found: {sid}"})
return None
```

Also update the extension dir check to support built-in extensions (currently only checks `USER_EXTENSIONS_DIR`):
```python
ext_dir = USER_EXTENSIONS_DIR / sid
if not ext_dir.is_dir():
ext_dir = EXTENSIONS_DIR / sid
manifest_exists = any((ext_dir / n).exists() for n in ("manifest.yaml", "manifest.yml", "manifest.json"))
```

Files Affected

• `dream-server/bin/dream-host-agent.py` — `validate_service_id` function
• `dream-server/config/core-service-ids.json` — consider renaming to `known-service-ids.json` for clarity
• `dream-server/extensions/services/dashboard-api/routers/extensions.py` — may also need updates for consistency

Priority

High — blocks multiple in-flight features (PR Light-Heart-Labs#877 deps+hooks, PR 2 templates, any future API-driven extension management).

[yasinBursali]

Verification Result — PARTIALLY CONFIRMED

Verified against: upstream/main @ c0600ca + static analysis (live test attempted but host agent hung — known issue #305)
Platform scope: All (macOS + Linux) — same Python code on both

---

Evidence

dream-host-agent.py validation logic (lines 363–380, not 291–305 as cited):

# line 51
CORE_SERVICE_IDS: set = set()
# line 53
TOGGLABLE_CORE_SERVICES: set = {"privacy-shield"}

# line 368 — the blocklist gate
def validate_service_id(handler, body: dict) -> str | None:
    sid = body.get("service_id", "")
    ...
    if sid in CORE_SERVICE_IDS and sid not in TOGGLABLE_CORE_SERVICES:
        json_response(handler, 403, {"error": f"Cannot manage core service: {sid}"})
        return None

core-service-ids.json — all 20 built-in services:
ape, comfyui, dashboard, dashboard-api, dreamforge, embeddings, langfuse, litellm, llama-server, n8n, open-webui, openclaw, opencode, perplexica, privacy-shield, qdrant, searxng, token-spy, tts, whisper

Behavior — it IS a blocklist:

• Check is if sid in CORE_SERVICE_IDS and sid not in TOGGLABLE_CORE_SERVICES → blocklist
• TOGGLABLE_CORE_SERVICES = only {"privacy-shield"} (line 53) — the sole exception
• Result: 19 of 20 services are blocked with HTTP 403. Of those 19:
  • 4 correctly blocked (always-on base services): llama-server, open-webui, dashboard, dashboard-api
  • 15 incorrectly blocked (optional extensions): ape, comfyui, dreamforge, embeddings, langfuse, litellm, n8n, openclaw, opencode, perplexica, qdrant, searxng, token-spy, tts, whisper

Live API test: Host agent PID 6855 was running but hung in CLOSE_WAIT state (related to issue #305) — curl connections timed out. Static analysis is conclusive for this bug.

---

Analysis

What's broken: All 20 built-in service IDs are in core-service-ids.json. The validate_service_id() gate blocks any ID in that list (except privacy-shield). So 15 optional extensions return 403 on /v1/extension/start and /v1/extension/stop.

Where the description is slightly inaccurate:

1. Line numbers cited as 291–305 — actual function is at lines 363–380
2. "CORE_SERVICE_IDS was originally a whitelist" — the code comments (lines 36–38) show it was always designed as a blocklist to prevent management of core services: "Prevents fail-open: without this, a missing JSON file would allow anyone with the API key to stop core services like llama-server or dashboard-api." The real issue is that the JSON file includes all 20 services instead of just the 4 always-on ones. Note: CORE_SERVICE_IDS is also used as an allowlist in validate_core_recreate_ids() (line 250), creating dual-use ambiguity.

Impact is confirmed as described: Dashboard API cannot start/stop/manage 15 built-in optional extensions. The TOGGLABLE_CORE_SERVICES exception only covers privacy-shield.

---

Verified via static analysis on 2026-04-11. Automated verifier (dreamserver-verifier agent).

[yasinBursali]

Status check from integration test of PR stack Light-Heart-Labs#893–909

Environment: WSL2 / Ubuntu 24.04 / NVIDIA RTX 3070 Laptop / Tier 1
Branch tested: local/integration-test containing all 17 open yasinBursali PRs merged --no-ff onto Light-Heart-Labs/DreamServer@c0600ca
Install: bash install.sh --all --non-interactive --no-comfyui → phase 13 reached, 17 healthy containers, host-agent running as systemd user service

Verdict

Host-agent layer: FIXED by PR Light-Heart-Labs#907. Dashboard-API public HTTP layer: still broken — see #333 (filed during this test pass) for the residual gap.

Evidence — host-agent layer (the layer this issue is filed against)

1. Constants are split exactly as suggested in the issue body. From /home/rosenrot/dream-server/bin/dream-host-agent.py after fresh install:

# L48–54
AGENT_API_KEY: str = ""
CORE_SERVICE_IDS: set = set()
# Distinct from CORE_SERVICE_IDS (which is the allowlist of known service IDs).
ALWAYS_ON_SERVICES: frozenset = frozenset({"llama-server", "open-webui", "dashboard", "dashboard-api"})

CORE_SERVICE_IDS retains its original purpose as the input-validation whitelist (if service_id not in CORE_SERVICE_IDS: at L219). ALWAYS_ON_SERVICES is the new immutable blocklist used as the actual guard at L337.

2. Direct runtime test against the host-agent at 127.0.0.1:7710 from the host.

$ curl -X POST -H "Authorization: Bearer $DASHBOARD_API_KEY" -H "Content-Type: application/json" \
    -d '{"service_id":"llama-server"}' http://127.0.0.1:7710/v1/extension/stop
{"error": "Cannot manage always-on service: llama-server"}                  # ✅ blocked (exact wording from suggested fix)

$ curl ... -d '{"service_id":"tts"}' http://127.0.0.1:7710/v1/extension/stop
{"status": "ok", "service_id": "tts", "action": "stop"}                     # ✅ permitted (was 403 before #907)

$ curl ... -d '{"service_id":"n8n"}' http://127.0.0.1:7710/v1/extension/stop
{"status": "ok", "service_id": "n8n", "action": "stop"}                     # ✅ permitted

$ curl ... -d '{"service_id":"nonexistent"}' http://127.0.0.1:7710/v1/extension/stop
{"error": "Extension not found: nonexistent"}                               # ✅ unknown ID still rejected by CORE_SERVICE_IDS whitelist

$ curl ... -d '{"service_id":"tts"}' http://127.0.0.1:7710/v1/extension/start
{"status": "ok", "service_id": "tts", "action": "start"}                    # ✅ start works too

docker ps after the round-trip shows dream-tts and dream-n8n healthy again. The host-agent layer now allows API management of all built-in extensions while still blocking the 4 always-on services. Matches the issue's suggested fix exactly.

Caveat: dashboard-API HTTP layer is still broken

PR Light-Heart-Labs#907 also touches dashboard-api/routers/extensions.py to relax _assert_not_core, and that part works (verified POST /api/extensions/llama-server/disable → 403 with the same error wording). But the dashboard-api's public enable_extension (L1037) and disable_extension (L1147) handlers only look in USER_EXTENSIONS_DIR, so calling them for a built-in returns 404 Extension not installed: n8n even though the host-agent could handle it directly. Filed as #333.

What was NOT tested

• The macOS host-agent path (no Apple hardware in this environment).
• Rate-limiting / abuse cases.

Pre-requisite that almost masked this fix

The installer didn't restart dream-host-agent.service after rewriting the binary — see #334 (filed during this test pass). I had to manually systemctl --user restart dream-host-agent.service before the new ALWAYS_ON_SERVICES guard became reachable; the OLD process from before the install was still serving requests with the old CORE_SERVICE_IDS-as-blocklist behavior.

---

Filed during full-stack integration test of open PR stack Light-Heart-Labs#893–909 on Light-Heart-Labs/DreamServer@c0600ca3. Cross-referenced from #333, #334.

[yasinBursali]

Addressed by PR Light-Heart-Labs#907 (currently draft, pending upstream review).