[moe] Good 10T: sweep capacity factor

[dlwh]

Summary

This issue asked whether the current MoE capacity factor was causing avoidable slowdown on the good-enough 10T path. The answer was yes: follow-up results in #4016 showed that the default 1.25 padding cost about 8% throughput at 1e20 scale and about 11% at 1e21, while lower settings caused only negligible loss impact in the tested regimes. The practical outcome was to move the default capacity factor to 1.0, with 1.1 noted as a more conservative option if future higher-EP or smaller-batch runs show overflow risk.

Helpful links

• Capacity-factor sweep table at 3e18 scale
• Detailed overflow analysis and 1.1 vs 1.0 recommendation
• Decision note to update the default to 1.0

Description

TL;DR: Sweep capacity factor on the current good-enough 10T candidate and verify that it is not a hidden source of problems.

Hypothesis or Goal

We want to know whether the chosen capacity factor is already safe or whether it is masking avoidable overflow or throughput loss.

Links

• Parent sweep: MoE Sweep: Hyperparameters, Routing, Architecture, and Optimizer #3469
• Gate: [moe] Good 10T gate for #3469 #4013
• Overflow + capacity-factor results: [moe] Good 10T: measure capacity overflow #4016

Results

Capacity-factor sweep results were reported in #4016 (not in this issue) by @ClassicLarry:

• EP=2, 3e18 runs: capacity-factor sweep table with BPB/macro-loss/tokens/s: [moe] Good 10T: measure capacity overflow #4016 (comment)
• Detailed overflow analysis and recommendation (1.1 conservative, 1.0 aggressive): [moe] Good 10T: measure capacity overflow #4016 (comment)
• 1e20 EP=4 direct compare: cap=1.0 vs 1.25 was 8.3% faster with +0.001 BPB: [moe] Good 10T: measure capacity overflow #4016 (comment)
• Final decision note: move default capacity factor to 1.0: [moe] Good 10T: measure capacity overflow #4016 (comment)

[github-actions]

🤖 I'm looking into this issue. I'll post a task list shortly with my plan and progress.

[claude]

🤖 Plan:

• Add capacity_factor field to GrugModelConfig (default 1.25)
• Pass config value through MoEMLP.__call__() instead of hardcoded constant
• Create sweep experiment script experiments/grug/moe/sweep_capacity_factor.py
• Add test for configurable capacity_factor (4 tests, all pass)
• Run pre-commit and tests (all green)
• Open PR: [moe] Make capacity_factor configurable and add sweep script #4056
• Verify CI passes (checks queued, waiting for runners)

[dlwh-golem]

@dlwh Added a managed TL;DR block summarizing the capacity-factor conclusion from #4016 and labeled this issue tldr so newcomers can see the default-1.0 decision and supporting links quickly.