Training loss doesn't decrease

[Uirseita]

Hi, I tried to run your notebooks. For notebook 2, my training loss is still around 0.19 after 200 epochs so the resulting model is still pretty bad. The only thing I changed is "use_multiprocessing" to False because it keeps getting an error when it's True.

Do you know how to fix this issue?

[liuxuankai]

me too ,tho loss doesn't decrease!!

[Golbstein]

Use notebook 2.1
The reason it doesn't work for notebook 1 and 2 is because we train it on a very small model compared to what we really want to train on.
It's just an example of how to fine-tune your pre-trained model on your internal dataset

[Golbstein]

I've made few changes now. Check it out

[Uirseita]

I've made few changes now. Check it out

Thanks for the response. I was able to run notebook 2.1 on my own pc with GTX1080Ti, but it always crashes because out of graphics memory on the computation center of my university which has Tesla V100 32GB. Do you know the reason for this by any chance?

[Golbstein]

How can I know if you didn't provide any details, not even the error message...
Usually it's due to the fact that your GPU is already "full" or you didn't reset it. Try to reboot the server..