Yolov5 training problem

[dilsadunsal]

Hello, I have trained Yolov5 with my own dataset on Colab. Hyperparameters; image size=640, batch size=16 with 50 epoch.

Now I would like to train same dataset with different hyperparameter; image size should be 576.

But it is failed. I received memory fault. Whats the reason of this problem?

[glenn-jocher]

@siangfei 👋 Hello! Thanks for asking about CUDA memory issues. YOLOv5 🚀 can be trained on CPU, single-GPU, or multi-GPU. When training on GPU it is important to keep your batch-size small enough that you do not use all of your GPU memory, otherwise you will see a CUDA Out Of Memory (OOM) Error and your training will crash. You can observe your CUDA memory utilization using either the nvidia-smi command or by viewing your console output:

If you encounter a CUDA OOM error, the steps you can take to reduce your memory usage are:

• Reduce --batch-size
• Reduce --img-size
• Reduce model size, i.e. from YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s
• Train with multi-GPU at the same --batch-size
• Upgrade your hardware to a larger GPU
• Train on free GPU backends with up to 16GB of CUDA memory:

[dilsadunsal]

Thanks for your reply.

I surprised that I could have trained batch size=16 in my previous training.

You mentioned about multi gpu training. When I use --nproc_per_node... Script, where do I need to determine the epoch number in the command?

[dilsadunsal]

Thanks for your reply.

I surprised that I could have trained batch size=16 in my previous training.

You mentioned about multi gpu training. When I use --nproc_per_node... Script, where do I need to determine the epoch number in the command?

I also received attached error while trying multi gpu command that you shared the link.