`srun --container-image` is silently ignored on bare-metal job steps

[sunxxuns]

Summary

This is a silent no-op, not a missing feature. srun accepts --container-image, runs the command somewhere else entirely, and exits 0. Nothing warns the user, so the failure surfaces later as a missing binary or as the host's unpatched Python packages being imported instead of the image's.

Running that same image by hand on the node with docker run gives you the image's environment. Asking Spur for it gives you the host's.

On bare-metal compute nodes (the non-Kubernetes mode Spur calls "native host"), srun --container-image=... accepts the container options but runs the step directly on the host. The image, mounts, working directory, named-container state, and container environment are lost before they ever reach the compute node: the RPC that the controller (spurctld) sends to the per-node daemon (spurd, Spur's slurmd) to run a step has no fields to carry them.

This blocks the standard Slurm + Pyxis/Enroot pattern used by distributed AI launchers, including Miles' checked-in multi-node launcher, which starts one containerized Ray head/worker step per allocated node with srun --overlap -w <node> --container-image=....

Code path (b5c6ece)

crates/spur-cli/src/srun.rs builds a JobSpec containing the container fields and sets srun_job: true.

For native nodes, crates/spurctld/src/scheduler_loop.rs sets srun_step_dispatch = true, calls register_allocation_on_nodes, and deliberately sets dispatch_spec to None. The command is subsequently sent through CreateJobStep / RunStep.

RunStepRequest (proto/slurm.proto:881) and RunCommandRequest (proto/slurm.proto:1099) have no container fields at all — no image, mounts, workdir, name, or save/remap options. In spurd's step handler, crates/spurd/src/agent_server.rs::run_command, the command is launched as a plain host process:

let mut cmd = tokio::process::Command::new(&program);
cmd.args(&program_args)
    .current_dir(&work_dir)
    .process_group(0);

This path does not call setup_rootfs, container_init, pivot_root, or build_launch_plan, and it does not enter an existing job container.

Reproduction

Use an image containing a marker that does not exist on the host:

srun --container-image=marker-image.sqsh \
     sh -c 'test -e /inside-image-marker'

Expected: success inside the requested image.

Actual on a bare-metal node: the command is executed by the host sh; the marker is absent. If the executable exists only in the image, the step fails with command not found.

The same applies inside an allocation:

salloc -N2 --gpus-per-node=8
srun --overlap -N1 -w node1 --container-image=miles.sqsh ray start --head ...

ray is resolved on the host, not in miles.sqsh.

Workaround, and why it does not close the gap

Multi-node containerized execution does work through the batch path: sbatch dispatches LaunchJob to every allocated node, and each node's script runs inside the image with SPUR_NODE_RANK, SPUR_NNODES, RANK, WORLD_SIZE, and MASTER_ADDR set. This is covered by tests/native_host/e2e/test_container.py::TestContainerMultiNode. A Ray cluster can therefore be brought up by branching on rank inside one script:

#!/bin/bash
#SBATCH -N2 --gpus-per-node=8 --container-image=miles.sqsh
if [ "$SPUR_NODE_RANK" = 0 ]; then
  ray start --head --block ...
else
  ray start --address="$MASTER_ADDR:6379" --block ...
fi

That workaround requires replacing the launcher of every framework being brought up on Spur, and it does not restore what steps provide:

• no way to restart one node's rollout engine or trainer inside an existing allocation — the unit of restart becomes the whole job;
• no containerized interactive iteration under salloc;
• upstream launchers (Miles, and any Pyxis-based launcher) cannot run unmodified;
• per-step images, mounts, and named containers are unavailable, so rollout and training steps cannot use different images in one allocation.

Submitting the parent script with sbatch --container-image does not help either: nested srun steps still go through RunCommand and execute on the host rather than entering the parent container.

Development impact

Miles' Slurm adapter uses the step pattern directly:

https://github.com/radixark/miles/blob/main/examples/experimental/openenv/glm52_tbench2/launch_16node_slurm.sh

It starts the Ray head and every Ray worker as concurrent containerized srun --overlap steps. Under Spur, those steps lose Miles' patched SGLang/Megatron/Ray stack and all requested mounts. The steps do not fail loudly at submission — the container flags are accepted and silently dropped, so the failure surfaces later as a missing binary or an import of the host's unpatched packages.

Expected behavior

For bare-metal allocations, container options on srun should either:

1. create/reuse the requested step container and execute the step inside it, or
2. enter the parent allocation's container when appropriate.

The step must retain normal job-step semantics: selected node/GPU subset, cgroup membership, signal propagation, output handling, and cleanup on allocation termination.

If containerized steps are out of scope for bare-metal nodes, srun should reject the container flags there instead of accepting and ignoring them.

Suggested coverage

Bare-metal E2E tests for: standalone srun --container-image; salloc followed by a containerized srun; sbatch --container-image followed by a nested containerized srun; simultaneous srun --overlap container steps on different nodes. Each should assert an image-only executable runs, requested mounts are present, GPUs are visible, and cancellation cleans up the step container.

Decision requested

Pick one and say which:

1. implement containerized steps on bare-metal nodes, which requires container fields on RunStepRequest/RunCommandRequest plus the container path in the step handler; or
2. reject --container-image and the related container flags at submission on that path, with an error naming the batch workaround.

Continuing to accept the flags and drop them is the only outcome that leaves users unable to tell whether their job ran in the requested image. Frameworks targeting Spur need to know which of the two to plan against.