Skip to content

SSH gateway loses remote exit status (channel-close vs exit-status race) #5027

Description

@mu-hashmi

apps/ssh-gateway (ssh.app.daytona.io). The command output is correct but the exit code is wrong, so anything that checks exit codes over gateway SSH misbehaves: CI scripts, set -e pipelines, deploy gates.

Symptom

ssh <token>@ssh.app.daytona.io <command> usually exits 255 even when the remote command ran and delivered its output. Occasionally the real exit code comes through. That on-and-off behavior is the signature of a race.

Evidence (2026-06-11, fresh default Linux sandbox)

round remote command stdout local exit
1 echo gateway-exit-probe correct 0
1 exit 5 (none) 255, want 5
2 echo gateway-exit-probe correct 255, want 0
2 exit 5 (none) 255, want 5
3 echo gateway-exit-probe correct 255, want 0
3 exit 5 (none) 255, want 5

A Windows sandbox shows the same behavior (found while verifying #4788), so the bug is platform-independent. The sandbox daemon itself propagates exit codes correctly. Loopback ssh -p 22220 inside the sandbox returns RC=5 for a remote exit 5. The loss happens in the gateway hop.

Suspected root cause

In handleChannel (apps/ssh-gateway/main.go), goroutines forward channel requests, including exit-status, between client and runner. The main handler does the bidirectional data copy, returns on EOF, and its deferred clientChannel.Close() and runnerChannel.Close() run. Those closes race the request forwarder: when the client channel closes before the forwarder relays the runner's exit-status, the OpenSSH client reports 255.

OpenSSH sends exit-status before close. The bridge needs to drain pending channel requests after data EOF before it closes the client channel, either by waiting for the request forwarder to observe the channel close or by forwarding exit-status explicitly before the deferred closes run.

Repro

  1. Create any sandbox, call createSshAccess, and take the printed ssh <token>@ssh.app.daytona.io command.
  2. ssh -o BatchMode=yes <token>@ssh.app.daytona.io 'echo hi'; echo $? prints the output and usually exits 255.
  3. ssh -o BatchMode=yes <token>@ssh.app.daytona.io 'exit 5'; echo $? usually prints 255 instead of 5.
  4. Repeat a few times. An occasional correct code confirms the race.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions