Skip to content

Use local rank for distributed Torch CUDA binding - #267

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/torch-local-rank-device
Open

Use local rank for distributed Torch CUDA binding#267
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/torch-local-rank-device

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Bind each distributed Torch worker to its node-local CUDA device instead of using the worker's global distributed rank as a device index.

init_distributed() currently uses RANK both for process-group identity and torch.cuda.set_device(). That works on a single node, but in a standard multi-node torchrun launch global ranks continue across nodes while CUDA device indices restart at zero on each host.

Fixes #266.

Fix

  • keep global RANK for dist.init_process_group();
  • read LOCAL_RANK for CUDA device selection;
  • fall back to global RANK when LOCAL_RANK is absent, preserving existing single-node/custom-launch behavior;
  • keep output suppression keyed to global rank so only global rank zero prints normally.

Regression coverage

Adds a CPU-only mocked regression with RANK=5, LOCAL_RANK=1, and WORLD_SIZE=8 verifying that:

  • process-group initialization receives global rank 5;
  • CUDA binding and warmup use cuda:1;
  • output suppression still receives global rank 5.

No tensor-parallel math, NCCL warmup behavior, or process-group configuration is otherwise changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Distributed Torch inference binds CUDA devices using global rank

1 participant