Will talos support Maximal Update Parametrization later?

[starlitsky2010]

1) I think Talos should add

Support Maximal Update Parametrization as it could help get optimal HPs from small proxy models and apply them on large LLMs

More details, please refer to : https://github.com/microsoft/mup

2) Once implemented, I can see how this feature will

It will acclerate the LLMs development as optimal HPs could be found by small proxy models and apply them on large LLMS. And this will save a lot of training cost. So there are quite a lot benefits for the LLM community.

3) I believe this feature is

• [YES] critically important
• [YES] must have
• [YES] nice to have

Is there any plan for this?

[mikkokotila]

Thank you for brining this up. It for sure looks interesting. Would you be interested in contributing towards implementation?