Skip to content

fix(dream-cli): schema-driven secret masking + macOS Bash 4 validation#994

Open
yasinBursali wants to merge 5 commits intoLight-Heart-Labs:mainfrom
yasinBursali:fix/dream-cli-config-security-macos
Open

fix(dream-cli): schema-driven secret masking + macOS Bash 4 validation#994
yasinBursali wants to merge 5 commits intoLight-Heart-Labs:mainfrom
yasinBursali:fix/dream-cli-config-security-macos

Conversation

@yasinBursali
Copy link
Copy Markdown
Contributor

@yasinBursali yasinBursali commented Apr 23, 2026

What

Five related hardenings to dream config and its helpers:

  1. Schema-driven masking in dream config show — replace the narrow keyword regex (which missed _PASSWORD, _SALT, _PASS suffixed fields) with .env.schema.json-driven detection.
  2. Bash 4+ invocationdream config validate now invokes validate-env.sh through "$BASH", and the target script adds its own Bash-4 guard. Fixes a macOS-only crash under /bin/bash 3.2.
  3. Belt-and-suspenders fallback — after a schema-miss under _schema_loaded=1, fall through to the keyword match instead of returning "not secret." Covers schema gaps and malformed schemas.
  4. Helper promoted to file-scope + reused by dream preset diff_cmd_config_is_secret was nested inside cmd_config show, so dream preset diff still used the original narrow regex and leaked N8N_PASS, LANGFUSE_SALT, and admin email fields in plaintext. Helper is now file-scope; both commands use it. N8N_USER and LANGFUSE_INIT_USER_EMAIL are also marked secret: true in .env.schema.json.
  5. Audit follow-up — close the jq-less Git Bash leak. python3 fallback in _cmd_config_load_secret_schema when jq is absent (Git Bash on Windows is the canonical case). Plus *user*|*email* keyword extension as belt-and-suspenders for the no-jq-no-python3 case.

Why

The narrow regex (SECRET|PASS|KEY|TOKEN)= missed suffix-form names like LANGFUSE_DB_PASSWORD, LANGFUSE_SALT, OPENCODE_SERVER_PASSWORD, etc. dream config validate crashed on every fresh macOS install because validate-env.sh uses declare -A (Bash 4+ only) but was invoked through /bin/bash 3.2. And dream preset diff was a forgotten sibling code path.

Maintainer audit (2026-04-28) flagged that the schema-driven path only fires when jq is on PATH. In Git Bash on Windows (no jq by default), the helper silently set _cmd_config_schema_loaded=0 and dream config show fell through to the keyword regex, which didn't cover *user*|*email*-suffixed names — so N8N_USER and LANGFUSE_INIT_USER_EMAIL printed in clear.

How

Five commits, each self-contained:

  • d4d754ae — schema-driven masking in cmd_config show.
  • cc6c40ac — Bash 4+ gate + "$BASH" invocation path.
  • ce37c119 — fallback-masking hardening (*bearer* added to keyword fallback) + test-harness fix for macOS.
  • 9cee345d — schema-miss falls through to keywords + helper promoted to file-scope + dream preset diff now uses the helper + N8N_USER / LANGFUSE_INIT_USER_EMAIL marked secret: true.
  • 2b49c8ffaudit follow-up: python3 fallback in _cmd_config_load_secret_schema, *user*|*email* keyword extension, regression test exercising all three PATH conditions.

Audit follow-up details (commit 2b49c8ff)

_cmd_config_load_secret_schema now follows this order:

  1. If jq available → use jq (existing fast path).
  2. Else if python3 available → parse .env.schema.json via heredoc'd python3 and extract keys where spec.get("secret") is True. Sets _cmd_config_schema_loaded=1.
  3. Else → leave _cmd_config_schema_loaded=0. _cmd_config_is_secret falls through to the keyword regex which now also includes *user*|*email* to cover the new schema secrets.

Verified: jq and python3 produce identical 36-key outputs on the live .env.schema.json (CG diff'd both, no delta).

Testing

  • make lint, make test, shellcheck all green.
  • New tests/test-dream-config-secret-mask.sh (7 cases, wired into make test):
    • jq+python3 both present → schema-driven jq path
    • jq absent, python3 present → python fallback
    • both absent → keyword regex fallback
    • sanity case: non-secret DREAM_VERSION=2.0.0-test shown in clear (no over-mask)
  • Each PATH case asserts both positive (KEY=*** appears in stdout) AND negative (literal sensitive value does NOT leak). Sentinel values: actual-admin-username (N8N_USER), admin@example.test (LANGFUSE_INIT_USER_EMAIL).
  • CG sabotage check: with both fallbacks reverted, 4 cases fail (jq-less and tools-less paths both leak). With only the keyword extension reverted, 2 cases fail (only the no-tools path leaks). Confirms each layer is independently load-bearing.
  • Original PR's dream config show / dream config validate / dream preset diff testing still passes.

Review

Local CG APPROVED (no warnings). Notes from review (informational only):

  • A literal-null .env.schema.json would trigger AttributeError in the python heredoc. Suppressed by 2>/dev/null and degrades to keyword regex fallback (functionally safe, just an ungraceful traceback). Could widen the except clause but project style discourages broad lists.
  • The test harness's symlink-PATH approach is sensitive to the outer PATH's Bash version (only matters for manual PATH=... debugging; standard make test is unaffected).
  • Over-mask risk from *user*|*email* extension is zero in current .env.schema.json — all 4 user/email-bearing keys are already secret: true. Hypothetical future operational keys (e.g., USER_HOME) would be over-masked but cat .env remains the unmasked escape hatch.

Known Considerations

A small number of operational keys (TOKEN_SPY_PORT, TOKEN_SPY_URL, LANGFUSE_PROJECT_PUBLIC_KEY) remain over-masked by the keyword fallback. This is intentional: dream config show should default to over-masking; raw values remain accessible via cat .env.

Platform Impact

  • macOS: primary fix target for the validate-env crash. All five hardenings apply.
  • Linux / WSL2: fixes apply identically; no platform branching.
  • Windows (Git Bash without jq): the audit follow-up is the primary win — schema-driven masking now works without jq via the python3 fallback, and the keyword extension closes the residual gap.

@Lightheartdevs
Copy link
Copy Markdown
Collaborator

Audit follow-up: needs revision for no-jq environments.

Schema-driven secret masking is useful, but the CLI only learns the schema secret flags through jq. In Git Bash without jq, newly marked user/email fields such as N8N_USER and LANGFUSE_INIT_USER_EMAIL can still print in clear. Please either make schema parsing available without jq for this command or extend the fallback mask to cover the new schema secrets.

yasinBursali and others added 5 commits April 29, 2026 02:51
The previous implementation used a narrow regex anchored to `=`:

    grep -qiE "(secret|pass|token|key)="

which required the keyword to appear immediately before `=`. It caught
`_SECRET=`, `_KEY=`, `_TOKEN=`, but missed `_PASSWORD=` (because `pass`
is followed by `word`, not `=`), `_SALT=` (not in the alternation), and
failed-open for any future secret name not ending in a matching suffix.

Real env keys that leaked in plaintext from `dream config show`:
  - LANGFUSE_DB_PASSWORD
  - LANGFUSE_SALT
  - LANGFUSE_CLICKHOUSE_PASSWORD
  - LANGFUSE_REDIS_PASSWORD
  - LANGFUSE_INIT_USER_PASSWORD
  - OPENCODE_SERVER_PASSWORD

Fix: drive masking from `.env.schema.json` — the canonical source of
truth that already marks the above keys `"secret": true`. Keys listed
there are masked; everything else is shown raw. When the schema or jq
is unavailable (older installs, minimal environments), fall back to a
case-insensitive substring match on secret/password/pass/token/key/salt
so the default behaviour errs on the side of masking rather than
leaking.

The schema already marks every known-leaking key above as
`"secret": true`; no schema change is required. A small number of
operational keys (TOKEN_SPY_PORT, TOKEN_SPY_URL, TARGET_API_KEY,
ANTHROPIC_API_KEY, OPENAI_API_KEY, TOGETHER_API_KEY, LIVEKIT_API_KEY,
LANGFUSE_PROJECT_PUBLIC_KEY) are not marked secret in the schema —
most are legitimately public (ports, URLs, publishable keys) or are
user-supplied. Left for a follow-up review; the regex fallback covers
them defensively if the schema ever goes missing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`dream config validate` crashed on every fresh macOS install. The
cause was a shebang mismatch: `scripts/validate-env.sh` is
`#!/bin/bash`, and macOS ships /bin/bash 3.2 (licensing-frozen) which
has no `declare -A`. The script uses associative arrays at lines 75,
76, 77, and 176, so it aborted immediately with
`declare: -A: invalid option` before doing any work.

Two-part fix:

- dream-cli invokes `validate-env.sh` through "$BASH" (the currently-
  running shell, guaranteed Bash 4+ by the version check at the top of
  dream-cli itself). This works regardless of the target script's
  shebang. Linux and WSL2 are unaffected (/bin/bash is already 4+
  there, so the invocation is a no-op change).

- `validate-env.sh` now starts with a Bash 4+ guard that prints a
  helpful "install a modern Bash" message and exits 1 before any
  `declare -A` is reached. This protects users running the script
  standalone (e.g. from the installer or CI), not just the dream-cli
  path. Wording matches the existing guard at the top of dream-cli.

`scripts/validate-manifests.sh` does not use associative arrays, so it
is invoked unchanged. Other `declare -A` users audited but not touched
in this PR (fix is the same pattern — either call through "$BASH" or
add a top-of-script Bash 4+ guard, depending on how each is invoked):

  scripts/pre-download.sh
  scripts/dream-test-functional.sh
  lib/service-registry.sh
  lib/progress.sh
  installers/phases/03-features.sh

Follow-up audit can cover each of these in scope.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…asking

Two small follow-up fixes flagged by the tester for the config subcommand
hardening:

1. tests/test-validate-env.sh invoked scripts/validate-env.sh by raw path,
   so shebang resolution picked /bin/bash = Bash 3.2 on macOS. The new
   Bash-4+ guard in validate-env.sh correctly blocks this invocation,
   which regressed the previously-passing macOS test cases (missing-.env
   and missing-schema, both expected exit 3 via early file-check) along
   with the Bash-4-dependent ones. Invoke the script through "$BASH" in
   the test harness, mirroring the dream-cli invocation path. One helper
   variable (VALIDATE_ENV_BASH), five invocation sites updated. Linux CI
   was unaffected; this fixes the macOS-only harness break.

2. Added "*bearer*" to the fallback substring match in cmd_config show's
   _cmd_config_is_secret, so FOO_BEARER=... does not leak when the
   schema is unavailable. Schema-driven path already covers the real
   DreamServer secrets; this closes a theoretical gap in the defensive
   fallback. Did not add *auth* (would false-positive on AUTHOR_NAME-
   style keys) or other substrings — bearer is the only real gap the
   tester surfaced.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The schema-authoritative check in the previous commit leaks keys that are
present in .env.schema.json with secret: false, which today miscovers five
real upstream-provider credentials (TARGET_API_KEY, ANTHROPIC_API_KEY,
OPENAI_API_KEY, TOGETHER_API_KEY, LIVEKIT_API_KEY). A malformed schema
(valid JSON but empty jq output) has the same silent-leak failure mode.

After a schema-miss under _schema_loaded=1, fall through to the keyword
substring match instead of returning "not secret" immediately. Schema
still defines what IS definitely a secret; the keyword pass adds defense
in depth for schema gaps. Over-masking of LANGFUSE_PROJECT_PUBLIC_KEY,
TOKEN_SPY_PORT, and TOKEN_SPY_URL is acceptable — 'show' should default
to over-masking, and raw values remain available via cat .env.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…NIT_USER_EMAIL

Maintainer audit on PR Light-Heart-Labs#994 (Lightheartdevs, 2026-04-28) flagged that
schema-driven secret detection in `_cmd_config_load_secret_schema`
ONLY fires when `jq` is on PATH. In Git Bash on Windows, `jq` is not
installed by default, so the helper silently leaves
`_cmd_config_schema_loaded=0` and `dream config show` falls through
to the keyword regex `*secret*|*password*|*pass*|*token*|*key*|*salt*|*bearer*`
— which doesn't cover the user/email-suffixed names recently flagged
as secret in `.env.schema.json`. `N8N_USER` and `LANGFUSE_INIT_USER_EMAIL`
print in clear.

Two-layer fix:

1. Add a Python3 fallback to `_cmd_config_load_secret_schema`. If `jq`
   is absent, parse `.env.schema.json` via a heredoc'd `python3` and
   extract `secret == True` keys. The installer guarantees Python 3,
   and Git Bash on Windows usually has it via `winget install Python`.
   Schema-driven masking now works without jq.

2. Extend the keyword fallback regex to include `*user*|*email*` so
   the truly-no-jq-no-python case still masks the new schema secrets.
   Light over-mask risk on operational keys like USER_HOME but
   `cat .env` remains the unmasked escape hatch — `dream config show`
   defaults to over-mask on purpose.

New regression `tests/test-dream-config-secret-mask.sh` (7 cases,
wired into `make test`) exercises all three PATH conditions:

- jq+python3 both present → schema-driven jq path
- jq absent, python3 present → python fallback
- neither present → keyword regex fallback

Each case asserts `N8N_USER=***` and `LANGFUSE_INIT_USER_EMAIL=***`
appear in stdout AND that the actual sensitive values do NOT leak.
A sanity case verifies non-secret `DREAM_VERSION` is shown in clear.

The PATH conditions are simulated by symlinking selected binaries into
a tempdir, then running dream-cli with `PATH="$tempdir"`. shellcheck
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yasinBursali yasinBursali force-pushed the fix/dream-cli-config-security-macos branch from 9236994 to 2b49c8f Compare April 29, 2026 00:03
@yasinBursali
Copy link
Copy Markdown
Contributor Author

Pushed audit follow-up (2b49c8ff).

Closes the jq-less Git Bash leak with two independent layers:

  1. Python3 fallback in _cmd_config_load_secret_schema. When jq is absent but python3 is on PATH (the typical Git Bash on Windows configuration), parse .env.schema.json via a heredoc'd python3 -c and extract keys where spec.get("secret") is True. Sets _cmd_config_schema_loaded=1 so the schema-driven path activates. CG verified jq and python3 produce identical 36-key outputs on the live schema (diff is empty).

  2. *user*|*email* keyword extension in _cmd_config_is_secret. Belt-and-suspenders for the truly-no-jq-no-python3 case. Light over-mask risk on hypothetical future operational keys; cat .env remains the unmasked escape hatch by design.

New tests/test-dream-config-secret-mask.sh (7 cases, wired into make test):

  • Builds three PATH conditions via symlink farm: jq+python3, jq-excluded, both-excluded.
  • Each scenario asserts N8N_USER=*** AND LANGFUSE_INIT_USER_EMAIL=*** appear in stdout AND that real sensitive values (actual-admin-username, admin@example.test) do NOT leak.
  • Sanity case: non-secret DREAM_VERSION=2.0.0-test shown in clear → no over-mask regression.

Local CG APPROVED (no warnings). Sabotage-checked: with both fallbacks reverted, 4 cases fail; with only the keyword extension reverted, 2 cases fail. Each layer is independently load-bearing.

Lightheartdevs pushed a commit that referenced this pull request May 1, 2026
The `_docker_cmd_arr: returns sudo docker when DOCKER_CMD is sudo docker`
test in `tests/bats-tests/docker-phase.bats` was failing in every CI
run on `upstream/main`, blocking the integration-smoke job across all
12 currently-open PRs.

Root cause: the test stub at line 94 (mirroring the real function at
`installers/phases/05-docker.sh:29`) does `echo "sudo" "docker"`, which
emits a single space-joined line `"sudo docker\n"`. The assertion at
line 100 expected `$'sudo\ndocker'` — two newline-separated lines.
That output never matched; the test had been silently red on main.

The real function's consumer at `installers/phases/05-docker.sh:36`
correctly word-splits the single-line output via
`local -a cmd=($(_docker_cmd_arr))`, so the function works in
production. The bug was assertion-only.

Fix: change `assert_output $'sudo\ndocker'` to
`assert_output 'sudo docker'`. Inline comment captures the
single-line + word-splitting contract so a future contributor doesn't
"fix" it back to a two-line assertion.

Verified: `tests/bats/bats-core/bin/bats tests/bats-tests/docker-phase.bats`
— case 2 now passes (was the only one failing in CI; local case 1
fail is environmental — no Docker daemon on dev machine — and unrelated).

Unblocks `integration-smoke` for the 12 open PRs (#974, #994, #998,
#1000, #1015-1057 etc.) that all base on upstream/main.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Collaborator

@Lightheartdevs Lightheartdevs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes the original no-jq secret leak finding: the branch now has a Python fallback for schema secret keys, and if neither jq nor python3 is available it over-masks user/email keys so N8N_USER and LANGFUSE_INIT_USER_EMAIL do not print in the clear.

Local proof:

  • tests/test-dream-config-secret-mask.sh passed 7/7 locally.
  • bash -n passed for the touched shell scripts.

Still not merge-ready because the branch is stale/dirty against current main:

  • A local merge simulation conflicts in dream-server/Makefile and dream-server/dream-cli.
  • The test emits some Git Bash symlink cleanup noise locally after reporting success; worth cleaning if it also shows up in CI logs, but the blocking item is the rebase/conflict resolution.

Please rebase onto current main. After that, this looks like a useful security hardening PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants