bug: PUT /api/settings/env round-trip rebuilds file from .env.example template, drops keys not in schema

[yasinBursali]

Bug Report: PUT /api/settings/env round-trip rebuilds file from .env.example template, drops keys not in schema

Severity: Low-Medium
Category: Data Fidelity / Settings Persistence
Platform: All (Linux, macOS, Windows/WSL2)
Confidence: Confirmed

Adjacent to: #317 (security: .env bind-mount changed from :ro to writable). The security boundary that #317 asks for is restored by PR Light-Heart-Labs#908 and works end-to-end (verified — see my comment on #317). This issue is about data fidelity in the new write path, not about the security model.

Description

A no-op round-trip through PUT /api/settings/env (sending the file back to itself unchanged) rebuilds the file using .env.example as a structural template instead of writing the input bytes verbatim. The pre-write file was 105 lines / 64 keys. The post-write file is 229 lines / 61 keys with an entirely different comment header. Six keys present in the input but absent from .env.example are silently DROPPED, including all GPU UUID assignments.

No live containers broke during the test (services don't re-read .env after start), but a future docker compose down && up -d after a "save settings" action could pin services to default values for any dropped key.

Affected File(s)

• dream-server/bin/dream-host-agent.py — _handle_env_update (the new endpoint added by PR fix(security): route .env writes through host agent, restore :ro mount Light-Heart-Labs/DreamServer#908)
• dream-server/extensions/services/dashboard-api/main.py — the PUT /api/settings/env route that delegates to host-agent

Root Cause (inferred from observed behavior)

_handle_env_update looks like it parses raw_text into key/value pairs, then re-renders the file using .env.example as a template, substituting the parsed values. This explains both observed effects:

• The doubled line count (.env.example has many more documentation comments than the runtime .env).
• The dropped keys (anything in raw_text but not in .env.example falls off the floor — there's no key in the template to substitute into).

The result is a file that LOOKS like a refreshed .env.example with the user's values dropped in, rather than a faithful save of what the user submitted.

Evidence

$ KEY=$(grep ^DASHBOARD_API_KEY= ~/dream-server/.env | cut -d= -f2)

# Capture pre-write state
$ wc -l ~/dream-server/.env
105 /home/rosenrot/dream-server/.env
$ grep -cE "^[A-Z_]+=" ~/dream-server/.env
64
$ md5sum ~/dream-server/.env > /tmp/pre.md5

# No-op round trip — read the file, write it back unchanged via the host-agent route
$ python3 -c "
import json, urllib.request
key='$KEY'
raw=open('/home/rosenrot/dream-server/.env').read()
req=urllib.request.Request(
    'http://127.0.0.1:3002/api/settings/env',
    method='PUT',
    headers={'Authorization': f'Bearer {key}', 'Content-Type': 'application/json'},
    data=json.dumps({'raw_text': raw}).encode(),
)
print(urllib.request.urlopen(req).status)
"
200

# Post-write state
$ wc -l ~/dream-server/.env
229 /home/rosenrot/dream-server/.env       ← grew by 124 lines
$ grep -cE "^[A-Z_]+=" ~/dream-server/.env
61                                          ← lost 3 keys net (dropped 6, added 3 from .env.example)
$ diff /tmp/pre.md5 <(md5sum ~/dream-server/.env)
< (different md5)

Keys dropped on this run

Comparing the pre-write file (preserved as .env.backup.20260411-211429 by host-agent's own backup mechanism) and the post-write file via:

$ diff <(grep -oE "^[A-Z_][A-Z0-9_]*=" data/config-backups/.env.backup.20260411-211429 | sort -u) \
       <(grep -oE "^[A-Z_][A-Z0-9_]*=" .env | sort -u)

Key dropped	Used by
COMFYUI_GPU_UUID	ComfyUI container GPU pinning (when ComfyUI is installed)
EMBEDDINGS_GPU_UUID	TEI embeddings container GPU pinning
WHISPER_GPU_UUID	Whisper STT container GPU pinning
JUPYTER_TOKEN	Jupyter (when installed as a user extension)
LLAMA_ARG_TENSOR_SPLIT	llama-server multi-GPU tensor split
LLAMA_SERVER_GPU_UUIDS	llama-server GPU pinning

All six are runtime values written by the installer / dream-cli for GPU pinning and per-service tokens. None of them appear in .env.example (which is the documentation template) — .env.example has slots for WEBUI_SECRET, N8N_PASS, LITELLM_KEY, OPENCLAW_TOKEN, etc., but not for the GPU-UUID assignments that are derived at install time.

Platform Analysis

• macOS: Affected — same Python code path runs in the host-agent regardless of host OS.
• Linux: Affected, verified end-to-end on WSL2.
• Windows/WSL2: Affected, verified.

Reproduction

Fresh install on a branch with PR Light-Heart-Labs#908 merged (i.e. the integration of the open PR stack Light-Heart-Labs#893–909 against Light-Heart-Labs/DreamServer@c0600ca).

1. Verify host-agent is the new version (post-#908): systemctl --user restart dream-host-agent.service
   (See #334 — the installer doesn't restart it automatically.)

2. KEY=$(grep ^DASHBOARD_API_KEY= ~/dream-server/.env | cut -d= -f2)
   wc -l ~/dream-server/.env                       # → 105
   grep -cE "^[A-Z_]+=" ~/dream-server/.env        # → 64

3. Send the file back to itself unchanged via PUT /api/settings/env (use the python snippet
   in the Evidence section above).

4. wc -l ~/dream-server/.env                       # → 229   (grew)
   grep -cE "^[A-Z_]+=" ~/dream-server/.env        # → 61    (shrank)

5. Compare against the host-agent's own backup at ~/dream-server/data/config-backups/.env.backup.<timestamp>:
   the dropped keys are visible in the diff.

Impact

Low-medium. No immediate failure observed — the running stack survived the test cleanly because services don't re-read .env after they start. But on the next docker compose down && up -d (e.g. an upgrade or reboot), services that depended on the dropped GPU UUID assignments will fall back to defaults. On WSL2 / multi-GPU hosts that can mean wrong GPU attachment, and on Tier 1 single-GPU systems the practical effect is usually invisible (one GPU, default selection works).

The risk is that any user who saves settings via the new dashboard editor — even unchanged — degrades their .env. The settings editor's most common path is "open, change one field, save", which would silently drop everything not in the schema.

The principle-of-least-surprise expectation when sending raw_text through a PUT is "the bytes I sent are the bytes that get written" (or at least: "the keys I sent are preserved").

Suggested Approach

Option A — preserve all keys (recommended): render the file using a CANONICAL list of keys = (existing .env keys ∪ schema keys). Substitute schema keys into the template positions; append non-schema keys to the bottom (or in their original order). Preserve all values.

Option B — true byte-for-byte round-trip: write raw_text verbatim. Validate only that no schema constraints are violated (no unknown control characters, no keys with reserved names, etc.), then write the bytes as-is. This is the principle-of-least-surprise fix and matches what PUT raw_text should naturally mean.

I'd recommend B because it has the simplest mental model and makes the security check (host-agent as single writer) the only nontrivial thing the route does. The current behavior (template-based render) is doing two jobs at once and silently doing the second one wrong.

Note

This is secondary to the higher-severity finding about Light-Heart-Labs#908 + Light-Heart-Labs#909 mount conflict (filed as #331). The main Light-Heart-Labs#908 write functionality works correctly:

• .env is mounted :ro in dashboard-api (verified via docker inspect)
• Direct write attempts inside the container are rejected
• PUT via dashboard-api delegates to host-agent (verified via docker logs + host-agent journal)
• Host-agent journal: .env updated via host agent from 172.18.0.9 (backup=...)
• Backup file is created on disk

The single-writer security boundary that #317 asked for works. Only the round-trip fidelity is off.

Cross-references

• security: .env bind-mount changed from :ro to writable — container compromise enables full-stack credential overwrite #317 (open) — adjacent parent issue. The security boundary that security: .env bind-mount changed from :ro to writable — container compromise enables full-stack credential overwrite #317 asks for is restored by PR fix(security): route .env writes through host agent, restore :ro mount Light-Heart-Labs/DreamServer#908 and works correctly. This issue is about data fidelity in the new write path, NOT about the security model.
• bug: read-only extensions mount blocks built-in template activation — every catalog template apply fails 500 #331 (filed in this same test pass) — separate, higher-severity issue about the same :ro mount blocking PR fix(dashboard-api): non-blocking apply_template + resolve built-in extensions Light-Heart-Labs/DreamServer#909's template activation. Both findings live in the dashboard-api container's relationship with the host-agent.
• bug: installer doesn't restart dream-host-agent.service after rewriting bin/dream-host-agent.py — leaves zombie reading deleted inode #334 (filed in this same test pass) — must restart host-agent after fresh install to reach the new write path.

---

Filed during full-stack integration test of open PR stack Light-Heart-Labs#893–909 on Light-Heart-Labs/DreamServer@c0600ca3. Environment: WSL2 / Ubuntu 24.04 / NVIDIA RTX 3070 Laptop / Tier 1.

[yasinBursali]

Partially applicable on macOS — the same .env.example-as-template code path exists and is reachable, but Mac installs hit an earlier 503 that masks it.

Attempted no-op round-trip on my integration-test install:

$ python3 -c "
import json, urllib.request
raw = open('/Volumes/X/dream-server-test/.env').read()
req = urllib.request.Request(
    'http://127.0.0.1:3002/api/settings/env',
    method='PUT',
    headers={'Authorization': f'Bearer \$KEY', 'Content-Type': 'application/json'},
    data=json.dumps({'raw_text': raw}).encode(),
)
urllib.request.urlopen(req)
"
ERROR: 503 {"detail":{"message":"Unknown key: HOST_RAM_GB"}}

File on disk is unchanged because the PUT is rejected before the rebuild logic runs. The cause is a separate, pre-existing macOS-only schema gap — env-generator.sh:170 writes HOST_RAM_GB=${SYSTEM_RAM_GB} to .env on Apple Silicon installs, but .env.schema.json has never had a HOST_RAM_GB entry (confirmed upstream/main; none of the 17 open PRs touches this). The schema-aware validator in _handle_env_update (routed through by api_settings_env_save) rejects the round-trip because the input contains a key that isn't in the schema.

Filing that as a separate macOS-specific issue in a moment so it doesn't pollute this thread. But it means:

On Mac, 100% of PUT /api/settings/env calls against a fresh install fail with 503 BEFORE the issue described here can manifest.

The .env.example rebuild bug itself — verified in the code

Confirmed the root cause you identified by reading dashboard-api/main.py:675 — _render_env_from_values:

def _render_env_from_values(values: dict[str, str]) -> str:
    example_path = _resolve_template_path(".env.example")
    seen: set[str] = set()
    output_lines: list[str] = []

    if example_path.exists():
        example_lines = example_path.read_text(...).splitlines()
    ...
    for line in example_lines:
        assignment = _ENV_ASSIGNMENT_RE.match(line)
        ...
        if assignment:
            key = assignment.group(1)
            output_lines.append(f"{key}={values.get(key, '')}")
            seen.add(key)
            continue
        ...
        output_lines.append(line)

    extras = [(key, value) for key, value in values.items() if key not in seen and value != ""]
    if extras:
        ...
        output_lines.extend([
            "# Additional Local Overrides",
            "# Values below were preserved because they are not part of .env.example.",
        ])
        for key, value in extras:
            output_lines.append(f"{key}={value}")

Key behaviors visible here:

• The file is rebuilt from .env.example structure (lines 688–708 walk example_lines and emit values for each assignment).
• Line counts will grow because .env.example has more comment lines than the original generated .env.
• Extras are preserved IFF value != "" (line 710). This filters out intentionally-empty keys in the input.
• The 6 *_GPU_UUID keys you saw dropped probably had empty values in your input dict — either because they were schema-stripped before reaching _render_env_from_values, or because the user had them as bare KEY= with no value. Either way, the value != "" filter kills them at line 710. Not "dropped because not in schema" — "dropped because empty after filter".
• LLAMA_ARG_TENSOR_SPLIT and LLAMA_SERVER_GPU_UUIDS are multi-GPU pinning vars that the installer may have written as commented-out lines. _ENV_COMMENTED_ASSIGNMENT_RE (line 690) handles commented-out entries but only preserves the comment form if the new value is empty (line 704). So a commented-out entry with a runtime-set value would lose its comment and the schema-filter would strip the value — net: dropped.

So Option B in your suggested fix ("round-trip raw_text BYTE-FOR-BYTE without reformatting") is the robust fix. Option A (canonical key union) still has the empty-value drop-through at line 710, and won't stop the schema validator from rejecting non-schema keys entirely (which is how it fails on Mac today).

Scope

• WSL2: this bug manifests as silent key drop after 200 OK (what you reported).
• macOS Apple Silicon: the round-trip fails 503 before this bug runs, because of the separate HOST_RAM_GB schema gap. Once that schema gap is fixed, macOS will start hitting this bug too.

Fixes should probably land together, or the Mac schema gap will be the hiding place for this regression.

(Runtime testing on macOS Apple Silicon, integration branch = upstream/main + all 17 open yasinBursali PRs.)

[yasinBursali]

Test Results — 12APRdevelopments branch (commit 2e238b3), WSL2 Ubuntu 24.04.4

Status: CONFIRMED — duplicate of the root cause in #331

Reproduction

1. Add non-schema key: echo 'MY_EXTENSION_HOOK_KEY=some_value' >> ~/dream-server/.env
2. GET /api/settings/env — MY_EXTENSION_HOOK_KEY is not visible in values (API filters to schema keys only)
3. PUT /api/settings/env with schema-only values — returns 200 OK
4. grep MY_EXTENSION_HOOK_KEY ~/dream-server/.env — key is gone

Mechanism

_prepare_env_save (main.py line 822):

current_values, _ = _read_env_map_from_path(env_path)  # reads ALL keys from .env
...
merged_values = {**current_values, **normalized_values}
return _render_env_from_values(merged_values), issues, apply_plan

_render_env_from_values only emits schema-backed keys to the rendered text. Non-schema keys in current_values are silently excluded. The host agent writes the schema-only text, overwriting the original .env.

Since GET never exposes non-schema keys, any UI-triggered save silently purges them from the file. This is the same root cause as #331.

[yasinBursali]

Fixed in Light-Heart-Labs#908 — relaxed unknown-key validation to warn-and-accept, and fixed empty-value drop in _render_env_from_values.