Commit 4ddffa3
authored
## Summary
CPU-only jobs (`device_type: CPU`) submitted through the SLURM launcher
fail on CPU partitions:
- `common/distutils.py::setup` hardcodes `backend="nccl"` on the SLURM
branch, so a CPU job raises `ValueError: ProcessGroupNCCL is only
supported with GPUs, no GPUs found!`.
- `launchers/slurm_launch.py` sets `gpus_per_node = ranks_per_node`
unconditionally, so a CPU job still requests a GPU per node and SLURM
rejects it with `Requested node configuration is not available` on a CPU
partition.
The local-launch branch already honors `config["distributed_backend"]`
(which `map_job_config_to_dist_config` sets to `gloo` for
`DeviceType.CPU`). This brings the SLURM path in line and stops
requesting GPUs for CPU jobs.
## Changes
- `common/distutils.py`: SLURM branch uses
`config["distributed_backend"]` instead of a hardcoded `"nccl"` (matches
the local-launch branch).
- `launchers/slurm_launch.py`: request `gpus_per_node = 0` when
`device_type == DeviceType.CPU`.
## Test plan
Submitted a single-node CPU job (`device_type=CPU`, `qos=cpu_lowest`,
`cpus_per_task=192`) running a `Runner` that only writes files (no GPU
work).
- Before: rejected at submission (`Requested node configuration is not
available`), and when a GPU was not requested it failed at runtime with
the NCCL error above.
- After: the job initializes the Gloo backend (`Torch distributed
initialized with: file://...`), runs to `COMPLETED`, and produces all
expected outputs on a CPU node.
No change to GPU behavior: for non-CPU `device_type`, the backend
remains `nccl` and `gpus_per_node` is unchanged.
1 parent 3681524 commit 4ddffa3
2 files changed
Lines changed: 4 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
| 124 | + | |
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
278 | 278 | | |
279 | 279 | | |
280 | 280 | | |
281 | | - | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
282 | 284 | | |
283 | 285 | | |
284 | 286 | | |
| |||
0 commit comments