[moe] Good 10T: compare AdamH vs Adam

[dlwh]

Description

TL;DR: Run a single roughly 1e19-FLOP comparison between AdamH and Adam in the good-enough 10T gate.

Hypothesis or Goal

We want to know whether AdamH is already a justified default for this path or still only an interesting alternative.

Links

• Parent sweep: MoE Sweep: Hyperparameters, Routing, Architecture, and Optimizer #3469\n* Gate: [moe] Good 10T gate for #3469 #4013

Results

[github-actions]

🤖 I'm looking into this issue. I'll post a task list shortly with my plan and progress.

[claude]

🤖 Plan:

• Research codebase for MoE + AdamH infrastructure
• Port GrugAttentionMlpLmHeadAdamHConfig (grug-compatible AdamH mask) to main
• Write experiment script: Adam vs AdamH at ~1e19 FLOPs on grug MoE
• Add test for the new optimizer config
• Run pre-commit and tests
• Open PR

[claude]

🤖 Plan:

• Research codebase for MoE + AdamH infrastructure
• Port GrugAdamHConfig (grug-compatible AdamH mask) to main
• Write experiment script: Adam vs AdamH at ~1e19 FLOPs on grug MoE
• Add test for the new optimizer config
• Run pre-commit and tests (all passing locally)
• Open PR: [moe] Add AdamH vs Adam comparison experiment at 1e19 FLOPs #4059
• Verify CI checks pass (runners still queued, local pre-commit and tests pass)