feat(dream-cli): --json flag on list/status and document doctor --json

[yasinBursali]

What

dream list --json and dream status --json now work as advertised. Previously the --json flag was silently dropped because the main case dispatch forwarded no arguments to the subcommands. Also documents the previously-undocumented dream doctor --json, keeps dream status-json as a back-compat hyphenated alias.

Plus an audit follow-up: a regression test proving dream list --json stays valid JSON on stdout even when registry/schema diagnostics fire on stderr.

Why

Scripts piping dream list / dream status output to jq got parse errors because the subcommands emitted human-readable output regardless of any flag passed. The human-readable hyphenated alias dream status-json was the only JSON-emitting path; --json appeared to work (no error) but emitted the wrong format.

Once the --json plumbing is fixed, the next concern (raised by maintainer audit) is that the JSON contract must hold even when registry loading emits diagnostics. After PR #1006 moved log()/warn() to stderr, the producer/consumer separation is in place — the regression locks it in.

How

Functional fix (commit af9acf0f)

• Main case dispatch now forwards "$@" to both cmd_list and cmd_status.
• cmd_list --json emits a JSON array of {id, category, status}.
• cmd_status --json delegates to the existing cmd_status_json. The delegation is wrapped in a subshell (( cmd_status_json )) to isolate the RETURN trap set inside cmd_status_json — without the subshell, the trap would leak into cmd_status's frame and crash with an unbound-variable error when the sibling nounset PR lands on top.
• Unknown flags on these subcommands now error cleanly (exit 1).
• cmd_help advertises the new flags and the existing doctor --json.

Audit follow-up (commits 6203c819 + 8d2b5493)

New dream-server/tests/test-dream-list-json-clean-stdout.sh (5 cases, wired into make test):

1. Hermetic scaffold — copies dream-cli + minimal lib/ (service-registry.sh required, plus safe-env.sh/python-cmd.sh if present) into a tempdir. Because dream-cli derives SCRIPT_DIR from BASH_SOURCE[0], and lib/service-registry.sh derives EXTENSIONS_DIR="${SCRIPT_DIR:-$(pwd)}/extensions/services", the test is fully isolated from the live repo's extensions tree.
2. Plants two manifests inside the scaffold:
   • extensions/services/valid-svc/manifest.yaml (well-formed).
   • extensions/services/broken-svc/manifest.yaml with schema_version: dream.services.v1 but missing service.id — the canonical trigger for lib/service-registry.sh:127 to emit # SKIP: <path>: missing required "id" field to stderr.
3. Runs NO_COLOR=1 DREAM_HOME=$tempdir $tempdir/dream-cli list --json with stdout and stderr captured separately.
4. Five assertions:
   • exit code 0.
   • stdout parses as JSON via python3 json.loads.
   • JSON contains valid-svc and excludes broken-svc.
   • stderr contains the literal # SKIP: diagnostic — proves the warning fired during the same invocation that produced the JSON.
   • stdout has no # SKIP: / ⚠ / [dream] / ANSI escape (literal 0x1B) bytes — the jq-pipeline contract.
5. Skips gracefully if python3 or PyYAML are unavailable on the host.

CG-follow-up tightenings in 8d2b5493: # SKIP: literal prefix (was bare SKIP), real $'\x1b[' ANSI ESC byte (was \\033\[ literal-string match), and [[ -f X ]] && cp X (was cp X 2>/dev/null || true — violated project CLAUDE.md || true ban).

Testing

• make lint PASS.
• make test PASS — new test runs at the end of the suite.
• shellcheck clean on the new test file.
• 5/5 cases pass; CG sabotage probe (replacing the SKIP token with a guaranteed-absent string) produces 4/1 fail — confirming the test is non-tautological.

Review

Local CG APPROVED WITH WARNINGS on the regression. All three labelling-strength warnings were folded into commit 8d2b5493 before push:

• # SKIP: literal prefix instead of bare SKIP substring.
• Real ANSI ESC byte test instead of literal four-char \033[ string match.
• [[ -f ]] && cp instead of cp ... 2>/dev/null || true.

CG flagged two coverage gaps as non-blocking follow-ups (not addressed here):

• cmd_status --json regression — same shape but exercises the subshell-isolated cmd_status_json path. Worth a sibling test in a follow-up.
• PyYAML parse error coverage — current scaffold triggers a registry-validation warning (# SKIP: missing id); a malformed-YAML manifest would trigger a different warn shape (warn "Service registry: manifest parser failed"). Assertion (e) would still catch leakage from either.

Known Considerations

cmd_status_json's internal RETURN trap (single-quoted, $tmp resolved at fire-time) is the latent root cause and exists pre-PR on upstream main too — dormant there because upstream currently has only set -e. The subshell wrap in this PR defensively hardens the new caller path without modifying the pre-existing function (the wider trap fix belongs with the nounset PR that adds set -u).

dream status-json hyphenated alias is kept for one release for back-compat.

Platform Impact

• macOS / Linux / Windows (WSL2): identical. No platform branching.
• The hermetic test scaffold uses mktemp -d, cp, python3, bash — portable across the supported platforms.

[Lightheartdevs]

Audit follow-up: needs a clean-JSON regression before merge.

The --json feature is useful, but machine-readable stdout must stay parseable even when registry/schema loading emits diagnostics. Now that #1006 has moved log()/warn() to stderr on main, please rebase and add a regression proving dream list --json remains valid JSON when PyYAML/registry loading warnings occur.

[yasinBursali]

Pushed audit follow-up (6203c819 + 8d2b5493).

Rebased onto current upstream/main. The post-#1006 stderr split is now in scope.

New tests/test-dream-list-json-clean-stdout.sh (5 cases, wired into make test):

1. Hermetic tempdir with copied dream-cli + minimal lib/ so EXTENSIONS_DIR resolves inside the scaffold (no live-repo pollution).
2. Plants extensions/services/valid-svc/manifest.yaml (well-formed) and extensions/services/broken-svc/manifest.yaml (missing service.id — triggers # SKIP: <path>: missing required "id" field from lib/service-registry.sh:127).
3. Runs NO_COLOR=1 DREAM_HOME=$tempdir $tempdir/dream-cli list --json with stdout/stderr captured separately.
4. Asserts:
   • exit 0
   • stdout parses as JSON via python3 json.loads
   • JSON contains valid-svc, excludes broken-svc
   • stderr contains the literal # SKIP: diagnostic — proves the warning fired in the same invocation that produced the JSON
   • stdout has no # SKIP: / ⚠ / [dream] / ANSI ESC byte (0x1B) — the jq-pipeline contract.
5. Skips gracefully if python3 or PyYAML are unavailable.

Local CG returned APPROVED WITH WARNINGS. All three labelling-strength warnings folded into 8d2b5493 before push: # SKIP: literal prefix, real $'\x1b[' ANSI byte test, [[ -f ]] && cp instead of cp ... || true (project CLAUDE.md style).

CG flagged two coverage gaps as non-blocking follow-ups: a cmd_status --json regression (same shape but exercises the subshell-isolated cmd_status_json path) and a PyYAML-parse-error case (current scaffold triggers a registry-validation warning; a malformed-YAML manifest would trigger warn "Service registry: manifest parser failed" — assertion (e) would still catch leakage from either).

[Lightheartdevs]

Audit pass (2026-05-02): --json on dream list and dream status is the right shape. Manual run confirms clean stdout ([]) with PyYAML warnings on stderr, exit code 0. Cannot review while CONFLICTING — please rebase. Codex's note about adding the focused test to CI with PyYAML installed (so it exercises the broken-manifest path instead of skipping) stands.

[Lightheartdevs]

This is closer than the first audit: the JSON flag is useful, warn() now writes to stderr via current main, and the branch adds a regression for dream list --json keeping stdout parseable while registry diagnostics go to stderr.

I can't merge it yet because the branch is stale/dirty against current main:

• GitHub marks it DIRTY.
• A local merge simulation conflicts in dream-server/Makefile.
• The added regression script syntax-checks, but in this Windows/Git Bash environment it skipped because the python3 command available to the script did not have PyYAML. Please make sure the CI path actually runs the new test rather than skipping it silently.

Please rebase/resolve the Makefile conflict and verify the JSON stdout regression runs in CI. After that, this should be a straightforward re-review.