srun step output is withheld until exit; blocking steps produce no output at all

[sunxxuns]

Summary

A blocking step produces no output at all, ever. Output is withheld until the process exits, so ray start --block and every other long-running service step is silent for its entire lifetime.

The same command run over SSH or under docker run on that node streams immediately. Only Spur's step path withholds it.

On bare-metal compute nodes, srun step execution buffers each node's entire stdout/stderr in memory on that node and returns it only after the process exits. Long-running service steps such as ray start --block are therefore silent for their whole lifetime, and step output is bounded by the node daemon's memory and the unary gRPC response limit.

Spur's own standalone srun streams output live, so this is a divergence between two paths of the same command rather than a capability Spur lacks. The fix is not client-side, though: spurd drains the pipe only at process exit, and RunStepResponse is a unary message with no way to carry incremental output, so both have to change before the client can print anything early.

Code evidence (b5c6ece)

The per-node daemon spurd (Spur's slurmd) launches the process with piped output and waits for completion, in crates/spurd/src/agent_server.rs::run_command:

let mut child = cmd
    .stdout(Stdio::piped())
    .stderr(Stdio::piped())
    .spawn()?;

let output = child.wait_with_output().await?;

The full strings are returned in RunCommandResponse. crates/spurctld/src/server.rs::run_step waits for every node and concatenates all returned strings into a single RunStepResponse; crates/spur-cli/src/srun.rs::dispatch_step prints them only after that response arrives.

By contrast, the standalone srun path calls try_stream_output (crates/spur-cli/src/srun.rs:930) and streams live. Step mode — the path taken whenever srun runs inside an existing allocation via inherited SPUR_JOB_ID/SLURM_JOB_ID — never calls it.

Because RunStepResponse is a unary message subject to the configured max gRPC message size, a sufficiently verbose step can also fail at completion even though the process itself succeeded.

This is not application-side stdio buffering

Slurm's srun forwards task stdout/stderr from slurmstepd over its I/O connection while the step runs, so output appears as the application flushes it. The familiar delay there is libc block buffering inside the application when stdout is not a tty, which srun -u/--unbuffered, stdbuf, or python -u resolves.

Here the pipe is drained only by wait_with_output() after the process exits, so no amount of application-side flushing produces earlier output, and a step that never exits produces none. Spur also has no --unbuffered equivalent.

Reproduction

Within an allocation:

srun bash -c 'for i in $(seq 1 1000); do echo tick-$i; sleep 1; done'

Expected: lines appear as they are produced, as with Slurm and as with standalone srun.

Actual: nothing until the step exits.

For a service process:

srun --overlap -w node2 ray start --address=$HEAD:6379 --block

there is no output for the lifetime of the Ray worker.

Development impact

Miles' multi-node Slurm adapter starts the Ray head and all workers as backgrounded srun --overlap steps and depends on their live logs:

https://github.com/radixark/miles/blob/main/examples/experimental/openenv/glm52_tbench2/launch_16node_slurm.sh

Ray join failures, GPU resource registration, worker death, RCCL init stalls, and NIXL/Mooncake transfer errors are all diagnosed from the last lines a process emitted before it stopped making progress. These failures characteristically present as a hang, not as an exit — so with output withheld until exit, the step that hangs is exactly the step that tells you nothing. Blocking service steps never exit at all, so their logs never arrive.

Redirecting each command to a file works, but it drops normal Slurm step I/O semantics and does not fix the buffering on the compute node when a caller omits the redirect.

Expected behavior

Provide streaming stdout/stderr for RunStep, multiplexed with node/task labels and with backpressure. Options include a server-streaming RunStep, incremental writes to job/step output files plus a StreamStepOutput RPC, or direct client-to-node streams as InteractiveSession already does.

Client disconnect behavior should be explicit: a disconnected reader must not silently orphan or kill a running step unless requested.

Suggested coverage

• first output arrives before process exit;
• node-labelled multi-node streaming;
• bounded memory on the compute node under high-volume output;
• output larger than the unary gRPC response limit;
• reconnect/tail against a still-running step;
• cancellation of a blocking service step.

Decision requested

Confirm whether streaming step output is planned. It requires a proto change, so it is not something a downstream user can work around in the client.

If it is not planned, srun should at minimum warn at step start that output will be withheld until exit. Silently swallowing the logs of a hung distributed job is the most expensive failure mode in this set to debug, because the hang and the silence are indistinguishable from a slow step.