torch.multiprocessing.spawn.ProcessRaisedException: -- Process 0 terminated with the following error:

[khawar-islam]

Dear @naga-karthik,

Thank you for your tutorials! I have followed your all steps but at the end i am getting two corrupt models. I have no idea what is the main reason? I implemented your DDP code into below code, also if training got successful i got two corrupt models.

https://github.com/mindee/doctr/blob/main/references/recognition/train_pytorch.py

[Error] https://stackoverflow.com/questions/75318245/torch-multiprocessing-spawn-processraisedexception-process-0-terminated-with

[naga-karthik]

Hey @khawar-islam, thanks for opening the issue! Honestly, it has been a while since I have used DDP directly from torch. I just checked the requirements and noticed that the torch version is 1.3.1, which is quite old. A lot of things must have changed since then.

While I unable to find any reasons for models getting corrupted, I highly recommend you to take a look at PyTorch Lightning instead. It makes DDP training a lot easier compared to directly using PyTorch. Here's a Colab file from their docs to get you started. If you're really interested in understanding DDP, this article seems like a good place to start. Hope these help!

[khawar-islam]

@naga-karthik thanks i am using someone library and its very hard to align code with PyTorch lightning.