Agent MoE Experiment: Leading 2 dense layers (no MoE on first 2)

[ClassicLarry]

TL;DR

Make the first 2 layers dense (3x hidden_dim MLP, no router, no shared expert). MoE routing starts at layer 2. Tests whether early layers benefit more from full-width dense computation.

User prompt

follow agent.md and try leading 2 dense layers. That is, make the first two layers have a shared expert of width 3x hidden_dim, and no routed experts. make sure that all the QB weights and logging stuff still works.

Scope

• Parent: MoE Scaling up to April goal #4281
• Branch: moe_leading_dense
• Sweep file: experiments/grug/moe/leading_dense_sweep.py
• Config: GrugModelConfig.num_leading_dense_layers=2

Layer layout (d512, L=6)

• Layer 0-1: Dense MLP (3x width)
• Layer 2-5: MoE (E=64, K=4 + shared)

Gate 1 runs (2 total)

Dim	Budget	Status
512	2.19e17	pending
768	1.70e18	pending

Decision log

empty

Conclusion

pending

[ClassicLarry]

🤖 Gate 1 Results: Leading 2 Dense Layers (macro_loss)

Both runs finished. Quality worse but throughput significantly better.

Dim	Run	macro_loss	tps	tps ratio	Eff Speedup
512	baseline	3.8104	405,630	—	—
512	leading-dense-2	3.8242 (+0.014)	465,905	1.149	1.075
768	baseline	3.4339	273,532	—	—
768	leading-dense-2	3.4407 (+0.007)	311,741	1.140	1.096

Dense first 2 layers are ~14% faster (no MoE dispatch overhead) but lose quality (+0.007 to +0.014). The throughput gain just barely compensates, giving marginal effective speedup (1.08–1.10×).

Verdict: Marginal PASS gate 1 on effective speedup, but quality regresses. The model is trading MoE capacity in early layers for speed. Not recommended if quality is the priority.