Skip to content

fix(zero): warn when process group silently falls back to single-rank #8117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
NovusEdge wants to merge 1 commit into from
Closed

fix(zero): warn when process group silently falls back to single-rank #8117

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions deepspeed/runtime/zero/partition_parameters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1035,6 +1035,17 @@ def __init__(self,
self.rank = dist.get_rank(group=self.ds_process_group)
self.dp_world_size = dist.get_world_size(group=self.ds_process_group)

# Detect silent single-rank fallback when multi-GPU launcher env is present
env_world_size = int(os.environ.get('WORLD_SIZE', '1'))
if env_world_size > 1 and self.dp_world_size == 1:
logger.warning(
f"[!] zero.Init detected WORLD_SIZE={env_world_size} in env but process group has "
f"world_size={self.dp_world_size}. This usually means torch.distributed was not "
f"initialized before model construction. Each rank will load the FULL model, "
f"causing OOM on large models. Fix: call deepspeed.init_distributed() BEFORE "
f"model construction (e.g., before from_pretrained())."
)

self.zero_param_process_group = zero_param_parallel_group
if _ds_config is not None and _ds_config.zero_config.zero_hpz_partition_size > 1 and self.zero_param_process_group is None:
groups._create_zero_param_parallel_group(_ds_config.zero_config.zero_hpz_partition_size)
Expand Down
Loading