loss oscillate

[ilovecv]

Hi,

Thanks for sharing the awesome work!

I am training the model on cc3m. However, after training for 50K steps, I found the loss started to oscillate. Did you observe similar phenomenon during training? I tried to reduce the learning rate by 10 times, but it didn't help. Do you have any suggestions?

Thanks!

[ogkalu2]

@ilovecv

Hey. Did this change ?

Also May I ask how much it cost, how long it took and what systems you used to train ?

[liqiangniu]

How many gpus are you used? A100 or V100?

[liqiangniu]

@ilovecv