recipe for hyperparameter sweep

[felipemello1]

Torchtune could provide a recipe to do HPO, where the user provides a config, the recipe, eval dataset, params to sweep and budget.

I just played with optimizer. Our default in lr 3e-4. I tried 3e-3 and the ScheduleFree optimizer. Big difference.

I don't know how well it works for finetuning, but an extension could be to allow HPO with smaller models, and transfer the best settings to the large models using muP (https://github.com/microsoft/mup).

[RdoubleA]

We would need some form of validation during training before we can do HPO, no?

On a related note, can we even run eval on a custom benchmark dataset that's not in EleutherAI?

[felipemello1]

I don’t know if necessarily during training. Maybe it’s ok to train for X steps and then do eval. I also think that as long as you are training for a single epoch, the train loss may be a good enough proxy, since the data doesn’t get repeated, and therefore there is no memorization