[moe] Good 10T: sweep E in {128,256,512}

[dlwh]

Description

TL;DR: Sweep E over {128,256,512} within the good-enough 10T gate instead of assuming the current expert count is settled.

Hypothesis or Goal

We want to know whether expert count is one of the main remaining levers for the baseline recipe.

Links

• Parent sweep: MoE Sweep: Hyperparameters, Routing, Architecture, and Optimizer #3469\n* Gate: [moe] Good 10T gate for #3469 #4013

Results

[github-actions]

🤖 I'm looking into this issue. I'll post a task list shortly with my plan and progress.

[claude]

🤖 Plan:

• Research codebase for existing MoE sweep patterns and 10T gate config
• Write sweep experiment script for E in {128, 256, 512}
• Run pre-commit and tests
• Open PR: [moe] Add expert count sweep for E in {128,256,512} #4049
• Verify CI passes (16/26 completed, 0 failures, rest queued for runners)

Script at experiments/grug/moe/sweep_expert_count.py sweeps num_experts across {128, 256, 512} using the grug MoE template. Base config mirrors the trial template; update it once #4013 locks the 10T gate config.