review remediation: deploy CLI orchestration correctness — multi-host lock bypass, fresh-host bootstrap, discarded stdout/audit log, flat-alias ssh config, roll-up defects

[bpamiri]

Problem

The wheels deploy orchestration layer has correctness defects spanning the lock, fresh-host bootstrap, output handling, ssh-config resolution, and several roll-up bugs. They share the command-construction layer with the deploy-security issue and should be fixed after that architecture decision so the same files are not churned twice. Paths relative to cli/lucli/services/deploy/; verified against origin/develop @ 8971094.

1. Deploy lock is silently bypassed on any multi-host fleet (DEP-1, High)
cli/DeployMainCli.cfc:82-86, 107, 452-455 acquires the lock via $dispatchAny → SshPool.onAny (lib/SshPool.cfc:101-109), first-success-wins, catching every per-host exception. The lock works by ln -s failing when it exists (commands/LockCommands.cfc:29-39), so on a 2+ host fleet, contention on host 1 raises, onAny swallows it, acquires a fresh lock on host 2, and the concurrent deploy proceeds — mutual exclusion holds only for single-host configs. Release is also $dispatchAny, so the released host may differ from the acquired host, stranding stale locks. Fix: acquire/release on a deterministic primary host via onEach/sequential dispatch with raise=true, mirroring Kamal's primary-host semantics.

2. Fresh-host deploy/setup cannot work (DEP-5, High)
(a) deploy() boots the proxy with proxy.details() & " || " & proxy.boot() (cli/DeployMainCli.cfc:90, 437-461); details() is docker ps --filter name=kamal-proxy, which exits 0 whether or not anything matches — so boot() is unreachable and kamal-proxy is never started. (b) The check is dispatched via $dispatchAny to ONE host, but every traffic-serving host needs its own proxy. (c) docker network create kamal exists (commands/DockerCommands.cfc:14-20) but has zero call sites, while --network kamal is required by AppCommands.run, AccessoryCommands.run, and ProxyCommands.boot — first run on a bootstrapped host fails with "network kamal not found". Fix: use docker start kamal-proxy || docker run ..., dispatch proxy boot to every traffic-serving host (onEach), and wire create_network into bootstrap/setup.

3. Every read verb discards remote stdout; the audit log is never written (DEP-6, Medium)
All dispatch closures call ssh.run(c, {raise: doRaise}) and drop the result; the only writeOutput is the hook-output line (cli/DeployMainCli.cfc:500). So audit (:171-187), details (:217-244), app logs (cli/DeployAppCli.cfc:67-77), lock status (cli/DeployLockCli.cfc:44-54) return only a summary, never the actual content in live mode. Compounding it, AuditorCommands.record() (commands/AuditorCommands.cfc:15-19) has zero call sites, so /tmp/kamal-audit.log is never written — audit tails a file this tool never creates. Fix: surface remote stdout from read verbs; wire AuditorCommands.record() into the deploy flows.

4. Flat aliases wheels deploy bootstrap / wheels deploy exec ignore ssh: config (DEP-7, Medium)
cli/lucli/Module.cfc:2150 and :2164 construct a bare new SshPool() instead of $deployBuildSshPool(opts.configPath) as the nested server branch does (:2183-2185). The flat aliases exist BECAUSE picocli absorbs the nested server form before module dispatch (#2677), so for real CLI users the broken form is the only reachable one. Any deploy.yml with non-root ssh.user, custom port, or a keys: path gets root@host:22/ssh-agent instead. Fix: build the pool via $deployBuildSshPool(opts.configPath) in the flat aliases.

5. Lock metadata never expands (DEP-10, Low)
commands/LockCommands.cfc:33-37: acquire wraps the entire symlink target in single quotes while the comment claims $(hostname)/$(date) are "resolved by the remote shell". Single quotes suppress command substitution, so the lock metadata is the literal user@$(hostname)/$(date ...)/message. Contrast AuditorCommands.record (:18), which correctly uses double quotes. Fix: use double quotes for the symlink target.

6. Roll-up: same-version redeploy, extract key match, proxy volume path (DEP-11, Low)
(a) deploy() issues docker run --name <service>-<role>-<version> (commands/AppCommands.cfc:18-30) without removing an existing same-name container; redeploy with an unchanged git SHA hits a guaranteed name conflict, and old-version containers are never stopped after cutover. (b) CFML == at cli/DeploySecretsCli.cfc:68 makes extract path match PATH — diverges from Kamal's exact match. (c) ProxyCommands.boot mounts /home/#ssh.user()#/.config/kamal-proxy (commands/ProxyCommands.cfc:30); for the default root user the home is /root, so the mount path is wrong on the default config. Fix: remove/replace same-name containers and stop old-version containers after cutover; exact (case-sensitive) key matching in extract; resolve the remote home directory.

Impact

Concurrent deploys on multi-host fleets are not mutually excluded; fresh-host deploy/setup fails outright; operators see only summaries from read verbs while the audit log is never written; the only CLI-reachable bootstrap/exec aliases ignore ssh config; lock metadata is unusable; and same-version redeploys always fail.

Suggested approach

Fix after the deploy-security architecture decision (DEP-2/3/4) lands, since these touch the same DeployMainCli/AppCommands/LockCommands/Module dispatch and command-construction layer. Apply the per-finding fixes above, mirroring Kamal's primary-host and network-bootstrap semantics.

Acceptance criteria

• Deploy lock is mutually exclusive across a 2+ host fleet; release targets the acquired host.
• Fresh-host deploy/setup creates the kamal network and boots the proxy on every traffic-serving host.
• Read verbs (audit, details, app logs, lock status) surface remote stdout; the audit log is actually written.
• Flat bootstrap/exec aliases honor ssh: config.
• Lock metadata expands $(hostname)/$(date).
• Same-version redeploy succeeds; extract key match is case-sensitive; proxy volume resolves the real remote home.

Source

Internal multi-agent framework review 2026-06-09, wave 2 (issues phase). Findings: followups DEP-1, DEP-5, DEP-6, DEP-7, DEP-10, DEP-11.

---

Re-scope 2026-06-12 (verified against develop @ 271931f)

Rationale: zero DEP items have shipped, but the epic's stated blocker is gone —
the body says "fix after the deploy-security architecture decision (DEP-2/3/4) lands";
#3008 landed it (stdin secrets, Base.shellEscape, validate-not-quote). The items
cluster naturally into four small waves that each touch a disjoint-enough file set,
matching the proven point-fix campaign mechanics from #3142–#3146 (TDD red-first on
FakeSshPool/dry-run, full CLI suite in the lucee7 docker harness).

Draft replacement body section (paste into #2957)

Status 2026-06-12 — re-scoped to campaign waves (scoping pass against develop @ 271931f)

The sequencing blocker is resolved: the #2956 command-builder architecture landed in #3008 (Base.shellEscape, SSH-stdin secret delivery, $rejectEnvSecrets fail-fast, name format validation). Every DEP item below was re-verified still reproducing in source at develop HEAD. The 2026-06-12 audit-confirmation comment adds live-harness evidence per item. None of this week's point fixes (#3142–#3146) touched these items.

Wave 1 — lock correctness (DeployMainCli.cfc, LockCommands.cfc, SshPool.cfc)

• DEP-1a Acquire the deploy lock on a deterministic primary host (Kamal semantics: first host of the primary role) via onEach/sequential with raise=true — not $dispatchAny (DeployMainCli.cfc:88-92; SshPool.onAny swallows per-host failures, SshPool.cfc:101-108).
• DEP-1b Release on the same host that acquired (DeployMainCli.cfc:113).
• DEP-10 Double-quote the lock symlink target so $(hostname)/$(date) expand (LockCommands.cfc:39; keep user/message escaping from fix(cli): deploy remote-exec security — stdin registry login, shell escaping, env.secret fail-fast #3008, fix the stale comment at :32-36).

Wave 2 — fresh-host bootstrap + setup (DeployMainCli.cfc, ProxyCommands.cfc, DockerCommands.cfc, AppCommands.cfc)

• DEP-5a Replace proxy.details() || proxy.boot() (DeployMainCli.cfc:96) with docker start kamal-proxy || docker run ... — details() exits 0 regardless.
• DEP-5b Boot the proxy on every traffic-serving (web-role) host via onEach, not $dispatchAny.
• DEP-5c Wire DockerCommands.create_network("kamal") (zero call sites today) into setup/bootstrap before any --network kamal consumer.
• setup() real implementation — setup() is return deploy(opts) (DeployMainCli.cfc:182-185): add accessory boot + network create + proxy boot (audit claims fd-07/acc-07).
• Proxy gating — only web/proxy-fronted roles get proxy.deploy (DeployMainCli.cfc:98-106 currently loops all roles; audit claim cfg-04).
• DEP-11c Resolve the real remote home for the proxy config volume (ProxyCommands.cfc:30 hardcodes /home/<user>; root's home is /root).
• env.secret env-file delivery (review remediation: deploy CLI remote-execution security — secret delivery, shell escaping, registry password redaction #2956 residue, pointed here by Base.cfc:63 and the EnvSecretUnsupported message): build env file from env.secret + SecretResolver, SshClient.uploadString (SshClient.cfc:235), --env-file on docker run; then lift the fail-fast and restore the init template's secret block. Optionally fold per-role env merge (Role.cfc:25-30 parsed, AppCommands.$envArgs ignores it) into the same design.

Wave 3 — observability + ssh-config (DeployMainCli.cfc, Deploy*Cli.cfc, Module.cfc, AuditorCommands.cfc)

• DEP-6a Surface remote stdout from read verbs (audit, details, app logs/details, lock status) — dispatch closures currently drop ssh.run() results (DeployMainCli.cfc:447-449); live mode returns only host-count summaries.
• DEP-6b Wire AuditorCommands.record() (zero call sites) into deploy/rollback/setup flows so /tmp/kamal-audit.log is actually written before audit tails it.
• DEP-7 Flat bootstrap/exec aliases must build the pool via $deployBuildSshPool(opts.configPath) (Module.cfc:2419-2421, :2433-2435 use bare new SshPool()), matching the nested server branch — these aliases are the only CLI-reachable form (wheels deploy server bootstrap is interpreted as LuCLI server command instead of deploy subcommand #2677).

Wave 4 — roll-up point fixes (independent, any order)

• DEP-11a Same-version redeploy: remove/replace an existing same-name container before docker run (AppCommands.cfc:18-30) and stop old-version containers after cutover.
• DEP-11b Case-sensitive extract key match (DeploySecretsCli.cfc:70 uses CFML ==; use compare() == 0).
• $runBash timeout (from fix(cli): SecretResolver Left(str,0) crash and secret parsing corruption #2951 review comment): bounded waitFor(timeout) + destroyForcibly() + SecretResolver.ResolutionFailed (SecretResolver.cfc:185).
• rollback() drops --destination (new, found in 2026-06-12 scoping — same class as closed deploy: wheels deploy config ignores --destination, so destination overlays can't be inspected #3085): DeployMainCli.cfc:156 is the last loader.load() call site not forwarding {destination: opts.destination ?: ""}.

Acceptance criteria unchanged from the original body, plus: setup on a pristine docker-sshd fixture host succeeds end-to-end (E2EDeploySpec extension), and wheels deploy rollback --destination=X --dry-run targets the overlay hosts.

---

[wheels-bot]

Wheels Bot — Triage

Classified as bug.

Suspected layer

cli (deploy subsystem) — see .ai/wheels/deploy.md for the canonical deploy CLI reference; the affected code lives under cli/lucli/services/deploy/ and cli/lucli/Module.cfc.

Fix sketch

Six distinct correctness defects with file:line evidence in the issue body. Likely touchpoints: cli/lucli/services/deploy/cli/DeployMainCli.cfc (lock acquire/release via deterministic primary host instead of $dispatchAny; proxy boot dispatched per traffic-serving host with proper docker start || docker run and kamal network creation; read-verb closures surface remote stdout; wire AuditorCommands.record() into deploy flows), cli/lucli/services/deploy/commands/LockCommands.cfc (switch single-quoted symlink target to double quotes so $(hostname)/$(date) expand), cli/lucli/Module.cfc lines ~2150 and ~2164 (replace bare new SshPool() in the flat bootstrap/exec aliases with $deployBuildSshPool(opts.configPath) matching the nested server branch at ~2183-2185), cli/lucli/services/deploy/commands/AppCommands.cfc (handle same-name container conflict and stop old-version containers post-cutover), cli/lucli/services/deploy/cli/DeploySecretsCli.cfc:68 (replace CFML == with case-sensitive comparison for extract key match), and cli/lucli/services/deploy/commands/ProxyCommands.cfc:30 (resolve the real remote home directory for the proxy volume mount instead of hardcoding /home/<user>).

Confidence: medium

Auto-downgrade applies — the fix touches the deploy subsystem (cli/lucli/services/deploy/**), and the issue itself flags a sequencing dependency on the deploy-security architecture decision (DEP-2/3/4) landing first; propose-fix's step-4 safety net will catch the deploy-subsystem scope.

<!-- wheels-bot:triage:2957 -->
<!-- wheels-bot:triage-class:bug -->
<!-- wheels-bot:triage-confidence:medium -->

[wheels-bot]

Wheels Bot — Fix on hold for human review

The proposed fix touches a sensitive area (cli/lucli/services/deploy/**) and the bot's safety net requires a human in the loop before any code is written.

[bpamiri]

Follow-up from #2951's review (Reviewer B, non-blocking): SecretResolver.$runBash calls proc.waitFor() with no deadline — an interactively-blocking secrets command (e.g. a credential manager prompting for unlock) hangs the deploy thread indefinitely. Suggest a bounded waitFor(timeout) + process kill + ResolutionFailed with a clear message. Belongs with this issue's orchestration-correctness batch.

[bpamiri]

Deploy-guide audit: empirical scope confirmation for #2956 / #2957 (and the picocli-shadowing issues)

NOT a new issue. The P1 batch-2 guide-behavioral audit (work item p1-16-deploy, 93 claims across all 10 deployment guides, wheels CLI 4.0.3 + develop @ ddeec99/840274b2d, Lucee 7 live harness) independently re-confirmed the following defects that are already inside the scope of existing issues. The coordinator should post this as a COMMENT on #2956 and #2957 (and optionally the picocli issues) — do not refile.

Confirmed inside #2956 scope (env.secret / secrets delivery)

• env.secret never delivered, four documented examples affected (claims fd-05, acc-02, cfg-07, sec-08): on released 4.0.3, env: {clear: {...}, secret: [FOO]} dry-runs emit only the clear -e flags — FOO silently never delivered. On develop (post-fix(cli): deploy remote-exec security — stdin registry login, shell escaping, env.secret fail-fast #3008), Base.cfc::$rejectEnvSecrets throws Wheels.Deploy.EnvSecretUnsupported (AppCommands.cfc:98 / AccessoryCommands.cfc:92). Affected doc examples: first-deploy.mdx step 4, secrets.mdx flow diagram, config-reference.mdx #env, accessories.mdx Postgres (which on 4.0.3 silently boots Postgres WITHOUT a password).
• ⚠ Aggravator worth highlighting on review remediation: deploy CLI remote-execution security — secret delivery, shell escaping, registry password redaction #2956: the init template templates/deploy/init/deploy.yml.mustache still scaffolds an env.secret: [WHEELS_RELOAD_PASSWORD] block — a freshly wheels deploy init-scaffolded config will hard-fail with EnvSecretUnsupported on develop-built CLIs. The template should ship without env.secret until delivery lands.
• Registry login password-in-argv on 4.0.3 (sec-08): confirmed git show v4.0.3 RegistryCommands.cfc:18 inlined -p <password>; develop's --password-stdin via SSH exec-channel stdin (RegistryCommands.cfc:20-28) confirmed — i.e. fix(cli): deploy remote-exec security — stdin registry login, shell escaping, env.secret fail-fast #3008's fix verified present on develop.
• Per-role env merging doesn't exist (cfg-04, cfg-07): role env: parsed (Role.cfc:25-30) but AppCommands.$envArgs uses top-level env only — config-reference.mdx's 4-row precedence table is machinery that isn't implemented. (Also listed in the new accepted-but-ignored-keys issue since the dead role-key half is a distinct root cause.)

Confirmed inside #2957 scope (orchestration / remote-exec parity, by DEP item)

• DEP-4/5 — setup() == deploy() (fd-07, acc-07): DeployMainCli.cfc:162-165 literally return deploy(opts); observed setup --dry-run with a declared redis accessory emits zero accessory commands. Proxy boot chain proxy.details() || proxy.boot() (line 90) unreachable (docker ps exits 0 regardless).
• DEP-5c — docker network create kamal never issued (arch-02): confirmed absent from all dry-run output.
• DEP-6 — remote stdout discarded / audit log never written (fd-08, fd-14, acc-04, arch-02): deploy app details live mode returns "Collected app details on N host(s)" discarding inspect output (DeployAppCli.cfc:46-50); AuditorCommands.record() has zero call sites — /tmp/kamal-audit.log is never written; "output prefixed with [host]" is dry-run-only.
• DEP-7 — flat deploy bootstrap builds a bare SshPool ignoring ssh: config (fd-06): confirmed.
• DEP-11b — extract-secrets key match is case-insensitive (CFML ==), diverging from Kamal (sec-06): confirmed live (extract-secrets b returns the B value).
• DEP-11c — proxy config dir resolves to /home/root for the default root user (arch-02): confirmed (root's home is /root).
• Proxy deploy fired for EVERY role, not just web (cfg-04): observed kamal-proxy deploy myapp --target myapp-job-... for a job role — deploy() loops proxy.deploy per role (DeployMainCli.cfc:92-101). config-reference.mdx says only web gets it.
• Cumulative effect on migrating-from-kamal.mdx (mig-05): "alternate freely between Ruby Kamal and wheels deploy / nothing on the server notices" is aspirational until review remediation: deploy CLI orchestration correctness — multi-host lock bypass, fresh-host bootstrap, discarded stdout/audit log, flat-alias ssh config, roll-up defects #2957 lands (audit log, network, proxy boot, proxy config dir, per-role proxy calls, and the 4.0.3 password-in-argv all diverge on-server).

Confirmed inside the picocli-shadowing issues (#2674 / #2677 / #2697)

• wheels deploy rollback --version=abc1234 → picocli "not a boolean" (LuCLI intercepts --version flag causing failure in wheels deploy command #2674); positional rollback abc1234 works (fd-10, mig-01).
• wheels deploy server bootstrap → LuCLI's picocli server help, exit 0, never reaches the module (wheels deploy server bootstrap is interpreted as LuCLI server command instead of deploy subcommand #2677); flat wheels deploy bootstrap works (fd-06).
• All three nested wheels deploy secrets fetch|extract|print forms picocli-intercepted (wheels deploy secrets fetch invokes LuCLI global secrets command instead of Wheels deploy subcommand #2697); flat fetch-secrets/extract-secrets <key> --from=.../print-secrets work but are documented NOWHERE in secrets.mdx/first-deploy.mdx/hooks.mdx (sec-04, sec-06, mig-01).

Good news the audit also confirmed (no action)

• Dry-run/no-SSH invariant holds everywhere probed; [host] cmd prefix in dry-run (DeployMainCli.cfc:417).
• /health and /ready examples work verbatim; RequestId middleware emits the header once enabled; sentry package installs with the documented mixin surface.
• Container/label/lock/audit-path CONSTANTS all match Kamal byte-for-byte in the command strings (arch-02) — the parity gaps are in execution, not naming.

— guide-behavioral-audit P1 batch 2, work item p1-16-deploy (44 works / 30 docs-wrong / 9 both / 3 code-broken / 7 unverifiable of 93 claims). Full per-claim evidence in the batch-2 raw JSON (catalog header has the path).