Skip to content

Commit 4ddffa3

Browse files
authored
fix(slurm): support CPU-only jobs on SLURM- #2137 (#2138)
## Summary CPU-only jobs (`device_type: CPU`) submitted through the SLURM launcher fail on CPU partitions: - `common/distutils.py::setup` hardcodes `backend="nccl"` on the SLURM branch, so a CPU job raises `ValueError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!`. - `launchers/slurm_launch.py` sets `gpus_per_node = ranks_per_node` unconditionally, so a CPU job still requests a GPU per node and SLURM rejects it with `Requested node configuration is not available` on a CPU partition. The local-launch branch already honors `config["distributed_backend"]` (which `map_job_config_to_dist_config` sets to `gloo` for `DeviceType.CPU`). This brings the SLURM path in line and stops requesting GPUs for CPU jobs. ## Changes - `common/distutils.py`: SLURM branch uses `config["distributed_backend"]` instead of a hardcoded `"nccl"` (matches the local-launch branch). - `launchers/slurm_launch.py`: request `gpus_per_node = 0` when `device_type == DeviceType.CPU`. ## Test plan Submitted a single-node CPU job (`device_type=CPU`, `qos=cpu_lowest`, `cpus_per_task=192`) running a `Runner` that only writes files (no GPU work). - Before: rejected at submission (`Requested node configuration is not available`), and when a GPU was not requested it failed at runtime with the NCCL error above. - After: the job initializes the Gloo backend (`Torch distributed initialized with: file://...`), runs to `COMPLETED`, and produces all expected outputs on a CPU node. No change to GPU behavior: for non-CPU `device_type`, the backend remains `nccl` and `gpus_per_node` is unchanged.
1 parent 3681524 commit 4ddffa3

2 files changed

Lines changed: 4 additions & 2 deletions

File tree

src/fairchem/core/common/distutils.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ def setup(config) -> None:
121121
assign_device_for_local_rank(config["cpu"], local_rank)
122122

123123
dist.init_process_group(
124-
backend="nccl",
124+
backend=config["distributed_backend"],
125125
init_method=init_method,
126126
timeout=timeout,
127127
)

src/fairchem/core/launchers/slurm_launch.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,9 @@ def slurm_launch(cfg: DictConfig, log_dir: str) -> list:
278278
mem_gb=scheduler_cfg.slurm.mem_gb,
279279
timeout_min=scheduler_cfg.slurm.timeout_hr * 60,
280280
slurm_partition=scheduler_cfg.slurm.partition,
281-
gpus_per_node=scheduler_cfg.ranks_per_node,
281+
gpus_per_node=(
282+
0 if cfg.job.device_type == DeviceType.CPU else scheduler_cfg.ranks_per_node
283+
),
282284
cpus_per_task=scheduler_cfg.slurm.cpus_per_task,
283285
tasks_per_node=scheduler_cfg.ranks_per_node,
284286
nodes=scheduler_cfg.num_nodes,

0 commit comments

Comments
 (0)