Agent MoE Experiment: Fixed bias increment (replace QB load balancing)

[ClassicLarry]

TL;DR

Replace QB load balancing with a simpler fixed bias increment. Each step, increase bias for underloaded experts and decrease for overloaded ones by a fixed delta.

User prompt

follow agent.md and implement fixed bias increment instead of QB load balancing. That is, on each step increase or decrease each router bias term by 0.01, depending on if the expert is underloaded or overloaded compared to its fair share. Test fixed increment scales of 0.01, 0.05, 0.005, 0.001.

Scope

• Parent: MoE Scaling up to April goal #4281
• Branch: moe_fixed_bias_increment
• Sweep file: experiments/grug/moe/fixed_bias_increment_sweep.py
• Config: GrugTrainerConfig.fixed_bias_increment

Mechanism

fair_share = total_tokens * K / E
bias[e] += delta * sign(fair_share - count[e])

QB computes an optimal threshold from top-k statistics. This tests whether a simpler sign-based update achieves similar balancing.

Gate 1 runs (8 total)

Delta	d512	d768
0.001	pending	pending
0.005	pending	pending
0.01	pending	pending
0.05	pending	pending

Decision log

empty

Conclusion

pending

[ClassicLarry]

🤖 Gate 1 Results: Fixed Bias Increment (macro_loss)

All 8 runs finished.

Delta	Dim	Run	macro_loss	Eff Speedup
—	512	baseline (QB)	3.8104	—
—	768	baseline (QB)	3.4339	—
0.001	512	fixed-bias	3.8964 (+0.086)	0.665
0.001	768	fixed-bias	3.4827 (+0.049)	0.761
0.005	512	fixed-bias	3.8356 (+0.025)	0.887
0.005	768	fixed-bias	3.4459 (+0.012)	0.932
0.01	512	fixed-bias	3.8116 (+0.001)	0.990
0.01	768	fixed-bias	3.4378 (+0.004)	0.980
0.05	512	fixed-bias	3.8138 (+0.003)	0.984
0.05	768	fixed-bias	3.4297 (−0.004)	1.028

Key findings:

• Too small (0.001, 0.005): bias adjusts too slowly, experts stay imbalanced → significant quality loss
• 0.01: closest to QB but still slightly worse at both scales
• 0.05: slightly helps d768 but hurts d512
• No delta consistently passes at both scales

Verdict: FAIL gate 1. QB load balancing outperforms all fixed increment values. The adaptive threshold in QB is important — a fixed step size can't match it.

[ClassicLarry]

🤖 Closing — fixed bias increment fails gate 1. QB is better.