Feature Request: μP (Maximal Update Parameterization)

[sbhavani]

Summary

Request to add support for μP (Maximal Update Parameterization) to enable hyperparameter transfer across model scales.

Motivation

μP allows tuning hyperparameters on small models and transferring them to larger models, reducing the cost of large-scale pretraining. It has been adopted in production models like Falcon-H1.

Requested Features

1. Per-layer Learning Rate Scaling - Different LR multipliers for specific layers
2. Width-dependent Initialization - μP-style init scaling based on model width

References

• MuP Paper
• Falcon-H1

[janEbert]

Other references to consider for automatic hyperparameter transfer:

• Modular norm
• Modular norm 2
• Scion optimizer (Muon generalization)

[BoxiangW]

Link to the older issue #1263

[Skylion007]

Ditto: would be great to add. We have an old fork here that tries to add it, but would be good to expand and properly integrate/test it: https://github.com/LLM360/Megatron-LM-muP

[Skylion007]

Other references to consider for automatic hyperparameter transfer:

* [Modular norm](https://arxiv.org/abs/2405.14813)

* [Modular norm 2](https://arxiv.org/abs/2409.20325)

* [Scion optimizer (Muon generalization)](https://arxiv.org/abs/2502.07529)

Also two other more recent references for parameter transfer:
https://arxiv.org/abs/2505.01618
https://arxiv.org/abs/2512.22382