Skip to content

Commit 6f84ea6

Browse files
authored
fix(config): use 1.0e-2 for moe_aux_loss_coeff to ensure correct float parsing (#234)
1 parent c817d08 commit 6f84ea6

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

primus/configs/models/megatron/grok_base.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,6 @@ num_query_groups: 8
1212
num_experts: 8
1313
moe_router_topk: 2
1414
moe_router_load_balancing_type: none
15-
moe_aux_loss_coeff: 1e-2
15+
moe_aux_loss_coeff: 1.0e-2
1616
moe_grouped_gemm: true
1717
moe_token_dispatcher_type: alltoall

primus/configs/models/megatron/mixtral_base.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,6 @@ num_query_groups: 8
1212
num_experts: 8
1313
moe_router_topk: 2
1414
moe_router_load_balancing_type: aux_loss
15-
moe_aux_loss_coeff: 1e-2
15+
moe_aux_loss_coeff: 1.0e-2
1616
moe_grouped_gemm: true
1717
moe_token_dispatcher_type: alltoall

0 commit comments

Comments
 (0)