Loss sudden increase

[SmaugPool]

What happened?
Does this mean that the model has denegerated and would the best model be the one just before the increase in this case?

Inference output produces empty sounds with the last models.

Note that I used the default learning rate of 0.000075 and 100000 epochs. Would other values have allowed to train longer?

[sngraam]

may your training data error!

[SmaugPool]

Not sure to understand what you mean. The training went fine during about a day. The loss increase came after.

[omid77922]

Not sure to understand what you mean. The training went fine during about a day. The loss increase came after.

[omid77922]

Not sure to understand what you mean. The training went fine during about a day. The loss increase came after.

[isolveit-aps]

Can I ask how you get the graph? Which folder do you point tensorboard to?

[SmaugPool]

Can I ask how you get the graph? Which folder do you point tensorboard to?

Choose tensorboard logger, install it with pip install tensorboard, then start it with:

tensorboard --logdir runs/F5TTS_Base/

Then connect to the web interface by clicking the displayed URL.

[isolveit-aps]

Thank you @SmaugPool. Could it be that the sudden loss is because all the projects are using the same runs folder, so the sudden loss is for a different project than the previous one?

[SmaugPool]

I don't think so. I was training only one model, and after the loss increase, the model doesn't work anymore (no sound output), but it works before. I wonder if I could have been able to train longer with a lower learning rate (1e-5 for example instead of 7.5e-5).

[sarpba]

I also had the problem when I taught from scratch, but the quality of the training data was lower. It probably also depends on the data quality.