Skip to content

Derive the autocast test backend and scope the bf16 gate to NCCL - #8559

Open
delock wants to merge 1 commit into
deepspeedai:masterfrom
delock:pr-f-autocast-cpu
Open

delock wants to merge 1 commit into
deepspeedai:masterfrom
delock:pr-f-autocast-cpu

Conversation

@delock

@delock delock commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Description

bf16_required_version_check() (tests/unit/util.py) requires torch >= 1.10, CUDA >= 11.0 and NCCL >= 2.10.3. On the cpu accelerator, bf16 collectives run over gloo/ccl and none of those transport dependencies exist, so the check always returns False and every bf16 test is skipped — about 40 call sites across 15 files. test_zero_autocast.py is worse off: it raises instead of skipping, so each of its cases counts as a failure (24 on the multi-rank CPU run in #8381).

This PR scopes the version floors inside the check itself:

if (cpu_accelerator and accelerator_pass) or (torch_version_available and cuda_version_available
                                              and nccl_version_available and accelerator_pass):
    return True
  • cpu: only the accelerator's own bf16 support (is_bf16_supported()) is required — the torch/CUDA/NCCL floors are transport dependencies that gloo/ccl does not have
  • every other accelerator (cuda, npu, hpu, xpu, mlu, ...): evaluates the exact original floors; cpu_accelerator is False so the expression is bit-identical to before, and the npu/hpu/xpu exemption branches are untouched
  • test_zero_autocast.py: the bf16 gate goes back to the bare call every other caller uses, and skips instead of raising

The hardcoded init_distributed(dist_backend='nccl') in the same test is deliberately left alone: it is a no-op (the harness already initialized the process group, comm.py:838-839), and deriving the backend per accelerator would change behavior on non-cuda accelerators (npu/hpu resolve to hccl etc.). The baseline DDP(device_ids=[i]) pinning is left for a follow-up (#8399 fixed the same pattern elsewhere).

Validation (executed on real hardware)

  • 20-core x86_64 CPU, torch 2.13.0+cpu, gloo backend
  • Direct call: bf16_required_version_check() on cpu returns False before, True after. Note: CPU_Accelerator.is_bf16_supported() is currently a stub that always returns True, so the cpu path does not gate on the hardware's bf16 instructions — giving it a real capability probe is left as a follow-up
  • Non-cpu equivalence: with cpu_accelerator == False the new expression reduces exactly to the original A and C and N and P
  • pre-commit (yapf / flake8 / check-torchdist / codespell) passes on both changed files

Sibling PRs from the same series: #8397, #8398, #8399, #8407, #8409. Exposed by the LOCAL_SIZE=4 multi-rank CPU run in #8381.

bf16_required_version_check() requires torch >= 1.10, CUDA >= 11 and
NCCL >= 2.10.3, so on the cpu accelerator, where bf16 collectives run
over gloo/ccl and none of those dependencies exist, the check always
fails and every bf16 test is skipped, about 40 call sites across 15
files. test_zero_autocast is worse off: it raises instead of skipping,
so each case counts as a failure.

Exempt the cpu accelerator inside the check itself so all call sites
benefit: on cpu only the accelerator's own bf16 support is required.
Every other accelerator evaluates the exact original floors, and
test_zero_autocast now skips like every other caller instead of
raising.

Signed-off-by: Guokai Ma <guokai.ma@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant