Skip to content

Commit eeb059b

Browse files
committed
fix: add use_partial_reduce_for_shared_embedding to DistributedDataParallelConfig
The plugin finalize_model_grads.py references this field but it was missing from the upstream config dataclass after the sync.
1 parent 46d066e commit eeb059b

1 file changed

Lines changed: 5 additions & 0 deletions

File tree

megatron/core/distributed/distributed_data_parallel_config.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,11 @@ class DistributedDataParallelConfig:
7676
"""If true, keep the compute param in fp4 (do not use any other intermediate dtype) and
7777
perform the param all-gather in fp4."""
7878

79+
# FlagScale Begin
80+
use_partial_reduce_for_shared_embedding: bool = False
81+
"""If true, use partial reduce for shared embedding gradient allreduce in hetero training."""
82+
# FlagScale End
83+
7984
reuse_grad_buf_for_mxfp8_param_ag: bool = False
8085
"""If true, reuse the grad buffer for param AG when using mxfp8 recipe. Should be
8186
set to True only when fp8_recipe is mxfp8 and fp8_param_gather is True."""

0 commit comments

Comments
 (0)