Huge and sustained loss after resume wan 2.2 i2v training lora

[bazanga912]

Hey everyone.
I'm training for the first time and I had to cancel the job. When I started it again I saw that the loss had a spike. So I read about it and it seemed that it would normalize in about 200-300 steps. But that's not what's happening.

As you can see in blue I run it until the step 6645 and then I had to cancelled it with about 0.0091 loss. When I resumed it I did it with a save at the start of an epoch on the 6000's step. And how you can see after 1299 steps it only dropped to 0.0095 and this came after a huge spike that reached about 0.0104. Something is wrong, right? Don't know what. I have the --save_state in the command.

Can someone help a newbie?
Thanks!

[dorpxam]

Hi. I'm not an expert, but for me there's no problem. The training process do not restart strictly at the same previous (average) loss value. Reading your chart, you'll reach the same loss approximately at 2K steps.

[bazanga912]

Hi. I'm not an expert, but for me there's no problem. The training process do not restart strictly at the same previous (average) loss value. Reading your chart, you'll reach the same loss approximately at 2K steps.

Thanks for you response!
But then, is always a bad thing to do to stop and resume, right? Comparing with a non stop training that is way worst, right?

[kohya-ss]

Once training is resumed, it should generally follow the same progress as if it had not been stopped. However, since the state of some random number generators is not recorded in the state, there is no guarantee that it will be exactly the same.

Also, since resuming resets the average loss, loss values will fluctuate greatly for a while after resuming.

The peak in the graph after resuming is probably a temporary increase and may become decreasing slowly with 1-2k additional training steps.

[dorpxam]

But then, is always a bad thing to do to stop and resume, right? Comparing with a non stop training that is way worst, right?

As @kohya-ss said, the resume needs some additional training steps to return to its previous state. I think it's always better to not break the training, you're right.

[bazanga912]

Hi! So nice to read a responde from you. Thank you for all your hard work, first of all.
So, It would help the training if I did a warmup each time I resume training? Or just add 1k extra steps to compensate it?

Once training is resumed, it should generally follow the same progress as if it had not been stopped. However, since the state of some random number generators is not recorded in the state, there is no guarantee that it will be exactly the same.

Also, since resuming resets the average loss, loss values will fluctuate greatly for a while after resuming.

The peak in the graph after resuming is probably a temporary increase and may become decreasing slowly with 1-2k additional training steps.

[kohya-ss]

When resuming, the optimizer state is restored, so at the very least, optimizer warmup is disabled.

Resuming should basically be the same as continuing training, but Musubi Tuner does not reproduce the dataset order when resuming, so continuing training without interrupting it is better in terms of data bias. This problem can be significantly alleviated by saving the state when the epoch changes.