Saving multiple checkpoints per epoch

[EugenHotaj]

Does torchtune support saving multiple checkpoints per epoch? AFAICT we only save checkpoints at epoch boundaries but please correct me if I'm wrong.

If not, would it be feasible to extend torchtune to save multiple checkpoints per epoch? One potential approach could be to pass a step variable to save_checkpoint here and either save subdirectories (e.g. epoch_i/step_j/ or update the current checkpoint naming to include the step number (e.g. epoch_i_step_j/).

[felipemello1]

hey @EugenHotaj , thats a great request! @joecummings has an RFC on it, but it is not implemented yet: #2105

@joecummings , do we have an estimate about when this would come out?

PS: @EugenHotaj , any thoughts on the current checkpoint folder format? Is it working ok or are you experiencing any gaps. We changed it recently.

[EugenHotaj]

PS: @EugenHotaj , any thoughts on the current checkpoint folder format? Is it working ok or are you experiencing any gaps. We changed it recently.

It's been working quite well! One thing worth considering is copying the model / tokenizer config files into the epoch_i subdirs. Right now we have a manual step to merge the configs with the weights before we can eval / serve which add some slight inconvenience.