feat(proxy): cover all remaining services with dream-proxy subdomain

[y-coffee-dev]

feat: Extend dream-proxy to every shipped service

dream-proxy already fronted chat / dashboard / auth / api / hermes / talk on :80. The other ~16 services still bound their own host ports and surfaced as http://localhost:<PORT> URLs in the installer summary. This PR brings them under the same subdomain-per-service pattern, driven by a new optional proxy: block in each extension manifest.

What lands

• Manifest schema. Each extension manifest can declare a proxy: block:

  proxy:
    subdomain: comfy
    client_max_body: 200MB   # optional; for upload-heavy services

  Validated by scripts/audit-extensions.py DNS-safe charset, no use of reserved names (chat, dashboard, auth, api, hermes, talk, root, www), and no two extensions claim the same subdomain.

• Resolver. scripts/resolve-proxy-config.sh walks enabled manifests at stack-bring-up time and writes one Caddy fragment per service into data/dream-proxy/sites.d/; the existing Caddyfile glob-imports them. Idempotent and sweeps stale fragments.

• mDNS auto-announce. bin/dream-mdns.py reads the same proxy.subdomain field, so new extensions auto-publish without code changes. The required-subdomains contract test was updated to track core_subdomain_routes.

• EXCLUSIVE toggle. DREAM_PROXY_EXCLUSIVE=true generates a docker-compose.proxy-exclusive.yml overlay that drops every routed service's host-port binding via ports: !reset null (Compose 2.20+ override tag — plain ports: [] silently merges with the base list, found that bug during live testing). scripts/resolve-compose-stack.sh picks the overlay up when the flag is on.

• Dashboard surface. /api/external-links adds a proxy_url field when dream-proxy is enabled; the Sidebar prefers it over the port URL. Installer phase 13 prints proxy hostnames as the primary URLs and tucks direct ports under a "developers" subheading.

Subdomain map

Service	Subdomain	Service	Subdomain
llama-server	llm	embeddings	embed
litellm	litellm	n8n	n8n
whisper	stt	token-spy	tokens
tts	tts	langfuse	langfuse
comfyui	comfy	privacy-shield	shield
ape	ape	perplexica	perplexica
openclaw	openclaw	brave-search	brave
searxng	searxng	qdrant	qdrant

Tests

• tests/test-proxy-config.sh — resolver unit test against fixture manifests (19 assertions; includes a static guard that fails CI if someone reverts !reset null to plain []).
• tests/contracts/test-proxy-routing.sh — manifest <-> resolver <-> Caddyfile <-> compose-mount invariants.
• make lint, make test, audit-extensions.py --strict, test_mdns_subdomains.py — all green.

Live LAN verification

Exercised on two real machines dream-1 and dream-2

• caddy validate against the merged Caddyfile + 6 generated fragments → "Valid configuration"
• dream-proxy boots; all 11 routes (6 mine + 5 core) reach their named upstreams
• mDNS records visible cross-host
• avahi-resolve-host-name llm.dream-2.local resolves to the correct IP, then curl http://llm.dream-2.local/ responds with 200
• DREAM_PROXY_EXCLUSIVE=true overlay: docker compose config shows every routed service's ports collapse from [{...:4000...}] to None.

Notes

• dream-proxy is category: optional, operators still run dream enable dream-proxy to activate.
• Multi-label .local on bare Linux clients: libnss-mdns ships with mdns4_minimal which only resolves single-label .local. macOS / iOS / Windows 10+ / Linux-with-systemd-resolved all resolve multi-label natively (verified). Older Linux setups need /etc/mdns.allow or mdns4 (non-minimal) in nsswitch.conf.

[Lightheartdevs]

Thanks for the thorough proxy plumbing here. The resolver/test shape is good, and dream-server/tests/test-proxy-config.sh passes locally for me. I’m requesting changes because this currently turns several previously loopback-style/raw service APIs into LAN-reachable proxy routes without an auth/exposure policy, and the new dashboard/summary URLs also break when the supported proxy port override is used.

Findings:

• [P1] dream-server/scripts/resolve-proxy-config.sh:160 generates plain reverse_proxy fragments for every manifest with proxy.subdomain, while dream-server/extensions/services/dream-proxy/compose.yaml:37 publishes the proxy on ${DREAM_PROXY_BIND:-0.0.0.0}. This PR adds proxy blocks to raw services such as llama-server (extensions/services/llama-server/manifest.yaml:24), embeddings, whisper, tts, searxng, and qdrant. Enabling the optional proxy would therefore expose endpoints like llm.<device>.local/v1 and qdrant.<device>.local to the LAN even though those backends do not all have their own auth checks. Please add per-route auth, an explicit manifest exposure classification, or a separate opt-in for unauthenticated API routes before routing these by default.

• [P2] dream-server/extensions/services/dashboard-api/main.py:1412 and dream-server/installers/phases/13-summary.sh:139-151 omit DREAM_PROXY_PORT. The schema/compose support non-80 proxy ports, but the sidebar and installer summary always render http://<sub>.<device>.local without :<port>. With DREAM_PROXY_PORT=8080, those links point at port 80 instead of the running proxy.

Validation run locally: bash dream-server/tests/test-proxy-config.sh passes (19/19). The blocker is policy/URL correctness, not the basic resolver mechanics.

[Lightheartdevs]

I did a second pass thinking about this less as “is the current patch mergeable?” and more as “what would make this the canonical answer to the LAN/headless-server access request?” I think the answer is: this PR has the right architecture and most of the useful plumbing; it just needs a small policy layer so we preserve Dream Server’s existing trust boundary.

What I would keep from this PR:

• Host-based routing via *.{$DREAM_DEVICE_NAME}.local, not path routing. That matches docs/DREAM-PROXY.md and avoids the subpath problems with Open WebUI, dashboard routes, websockets, and OAuth-style callbacks.
• Generated Caddy fragments from manifests. This is a good extension story.
• mDNS auto-announcement from the same manifest data. Also good.
• DREAM_PROXY_EXCLUSIVE=true with ports: !reset null. That is a nice “proxy is the only host entrypoint” mode.
• Dashboard/sidebar awareness of proxy URLs. That is the right user experience.

The design wrinkle is that the repo already treats direct service ports as loopback-first. resolve-compose-stack.sh even rejects user compose ports that are not bound to loopback unless they use the approved pattern, and docs/DREAM-PROXY.md says the proxy is the only LAN-facing surface. So if the proxy starts routing every backend with a proxy: block, we accidentally turn “one safe LAN entrypoint” into “all raw APIs are now LAN APIs.”

A concrete path I’d be happy with:

1. Add an exposure/auth policy to the manifest proxy: block. Naming can absolutely differ, but something like this is the shape I mean:

proxy:
  subdomain: comfy
  exposure: developer-api   # user | developer-api | internal
  auth: service             # service | dream-session | none
  client_max_body: 200MB

2. Make the resolver profile-aware. Default profile should only emit user-facing/safe LAN routes. A developer profile can opt into raw/API routes. Internal routes should require a very explicit opt-in.

Possible env shape:

DREAM_PROXY_PROFILE=user        # default: user-facing routes only
# DREAM_PROXY_PROFILE=developer # includes developer-api routes
# DREAM_PROXY_PROFILE=all       # includes internal routes, loud warning

Or split flags are fine too. The important part is that raw APIs like llama-server, embeddings, STT/TTS, and Qdrant do not become LAN-routable just because someone enabled the proxy for chat/dashboard/headless access.

3. Reuse the existing auth pattern where needed. hermes-proxy/Caddyfile is a good concrete template: forward_auth to dashboard-api /api/auth/verify-session, health paths left public, and websocket upgrade headers stripped from the auth subrequest. If a route says auth: dream-session, generated fragments can include that block. If a route says auth: service, the service’s own auth is the gate. If a route says auth: none, audit should probably reject exposure: user unless there is an explicit “yes, expose this unauthenticated thing” flag.

4. Update the tests around the policy, not just the mechanics. The current tests prove “one fragment per proxy block”; after this change they should prove “one fragment per policy-allowed proxy block.” I’d like to see fixtures for:

• default profile does not generate llama-server, qdrant, embeddings, STT/TTS raw API routes;
• developer/all profile does generate opt-in routes;
• auth: dream-session fragments contain forward_auth and leave /health public;
• DREAM_PROXY_EXCLUSIVE=true only strips ports for routes actually emitted under the selected profile;
• duplicate/reserved subdomain checks still fire regardless of profile.

5. Fix proxy-port URL rendering while you are in here. DREAM_PROXY_PORT is configurable, but /api/external-links and phase 13 currently render http://<sub>.<device>.local without :<port>. Add :<DREAM_PROXY_PORT> when it is not 80, and add a test for DREAM_PROXY_PORT=8080 so dashboard links and installer summary don’t drift.

6. Refresh docs with the product-level answer. docs/DREAM-PROXY.md, docs/MDNS.md, and probably the install summary should say something like: “Dream Proxy is the supported LAN/headless-server access path. By default it exposes user-facing surfaces; raw APIs require developer/internal opt-in. For off-LAN access, put the proxy behind Tailscale/WireGuard; do not publish it directly to the public internet.”

I want to emphasize: I think this PR is very close to the right answer for the community request. The generated-router approach is good. The change I’m asking for is not a rewrite; it’s making the route generator aware of the trust model that already exists in the repo.

[y-coffee-dev]

Thanks for the review! you're right that the bare PR turned every backend into a LAN endpoint.

Rebased onto current main and pushed seven follow-up commits:

• manifest exposure + auth fields — every proxy: block now declares exposure: user|developer-api|internal and auth: service|dream-session|none.

  • user / service (UIs with their own login): n8n, langfuse, openclaw
  • user / dream-session (UIs that ship with no built-in auth — get wrapped in dashboard-api's session check): comfyui, perplexica, token-spy
  • developer-api / service: litellm (master key), ape (API-only), privacy-shield (API-only)
  • developer-api / none (raw APIs, no auth): llama-server, embeddings, whisper, tts, qdrant, searxng, brave-search

• profile-aware resolver — DREAM_PROXY_PROFILE=user (default) | developer | all. auth: none + exposure: user exits non-zero unless DREAM_PROXY_ALLOW_UNAUTHENTICATED_USER=true. Missing exposure defaults to developer-api so third-party extensions don't auto-LAN. Duplicate/reserved checks fire before the profile filter so they still apply under any profile. audit-extensions.py gets matching static checks.

• forward_auth for auth: dream-session — fragment mirrors hermes-proxy/Caddyfile (literal-order route { }, @health public, websocket header strips, 303 to /auth/required). comfyui, perplexica, and token-spy now go through this under the default profile.

• DREAM_PROXY_PORT in URLs — dashboard-api/main.py and phase 13 append :<port> only when non-default. Same commit makes the phase-13 banner profile-aware so it doesn't advertise URLs the resolver wouldn't emit.

• tests — test-proxy-config.sh assertions covering all of the above. Two new pytest cases for the port rendering.

• docs — DREAM-PROXY.md gets the profiles/exposure section and an "Off-LAN access" subsection pointing at Tailscale/WireGuard/oauth2-proxy. MDNS.md gets a port note.

• bash tests/test-proxy-config.sh 42/42

• audit-extensions.py clean

• make lint clean. Pushed with --force-with-lease.

[Lightheartdevs]

Codex current-main audit

I re-audited this after the exposure/auth/profile follow-ups against current main (61283b71). This is much closer than the first version, and the direction is good: profile-aware proxy generation, explicit user / developer-api / internal exposure metadata, dream-session auth through dashboard-api forward_auth, and dashboard URL rendering with DREAM_PROXY_PORT all match the shape we want.

What I verified locally:

• PR merges cleanly into current main.
• dream-server/tests/test-proxy-config.sh passes: 42/42.
• python dream-server/scripts/audit-extensions.py --project-dir dream-server passes: 24 services, 0 errors, 0 warnings.
• Focused dashboard-api proxy/external-link pytest slice passes: 2/2.
• Diff hygiene is clean.

I would not merge yet because the new/changed proxy routing contract is stale and failing:

1. dream-server/tests/contracts/test-proxy-routing.sh hardcodes python3. In Git Bash on Windows that selects a Python without PyYAML, while the repo has Python-detection helpers/patterns elsewhere. Please switch this to the repo?s Python discovery pattern, e.g. tests/lib/python-cmd.sh / ds_detect_python_cmd, or otherwise use the same interpreter path style as the resolver/audit scripts.

2. The contract still expects one generated fragment for every manifest with a proxy: block. That no longer matches this PR?s policy. With the default DREAM_PROXY_PROFILE=user, only proxy.exposure: user routes should be emitted. The test should be profile-aware and assert the expected set for user, developer, and all.

3. The structural check expects exactly one reverse_proxy at one-tab indentation. That breaks for the new auth: dream-session shape, where reverse_proxy is intentionally nested inside route {} after forward_auth. The test should allow arbitrary indentation for reverse_proxy and separately validate the auth-specific structure: forward_auth, /api/auth/verify-session, public health handling, and websocket header stripping for dream-session routes.

Recommended fix:

• Update test-proxy-routing.sh around the actual policy this PR introduces:
  • default/user profile emits only exposure=user enabled routes;
  • developer profile adds developer-api routes;
  • all profile includes internal routes;
  • auth=dream-session fragments include the forward-auth pattern;
  • auth=service / auth=none bare reverse-proxy routes still validate;
  • exclusive overlays strip only routes emitted for the active profile.
• Rebase on current main and rerun CI.
• After that, this looks close. I would still treat it as operationally meaningful LAN/proxy exposure work and follow it with targeted proxy + owner-card + headless access validation.

Bottom line: good feature direction, likely valuable, but please don?t merge while its own routing contract is wrong/failing.