Skip to content

Commit 2ced5cf

Browse files
authored
Bug Fix - Fix bugs in optional tokens per expert logging (#54)
This PR fixes two bugs in #41 - `--moe-tokens-logging` doesn't control whether the related tracked names are added. - `sinkhorn` tokens per expert metrics are never tracked in original if statement.
1 parent b55340e commit 2ced5cf

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

megatron/training/training.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1527,6 +1527,7 @@ def training_log(loss_dict, total_loss_dict, learning_rate, decoupled_learning_r
15271527
track_names = []
15281528
if args.moe_router_load_balancing_type in ["aux_loss", "seq_aux_loss", "global_batch_loss"]:
15291529
track_names.append("load_balancing_loss")
1530+
if args.moe_tokens_logging:
15301531
if args.moe_router_load_balancing_type == "sinkhorn":
15311532
track_names.append("sinkhorn_tokens_per_expert")
15321533
elif args.moe_router_load_balancing_type == "aux_loss":

0 commit comments

Comments
 (0)