[Feature] python script based ddp?

[hiyyg]

In pytorch-lightning, the default ddp is based on python script rather than spawn, which can overcome some limitations of spawn based ddp.
https://pytorch-lightning.readthedocs.io/en/stable/advanced/multi_gpu.html

I wonder is it possible to add this ddp feature into mmcv?

[nbei]

Hi @hiyyg , I have checked the implementation of DDP and DDP2 from PL. In fact, DDP in PL still adopts spawn, but it use scripted based launcher instead of smp.SimpleQueue().

Importantly, when using official PyTorch and MMCV, you will launch your project with torch.distributed.launch, e.g. in MMDet. After having a close look at the source code of PyTorch launcher, we find that the DDP in PL is similar to the official launcher in PyTorch. Thus, I just think it is no need to implement the DDP in PL. Besides, we have not observed related issues caused by this implementation.

[hiyyg]

@nbei In fact, I was almost always seeing programs using the spawn-based ddp consuming more gpu memory than those using the same batch size per gpu. And sometimes I got strange error of something can not be forked. However, I recently changed to the PL's script based ddp for training those programs, then those headaches never happened again.

So, although I don't know much about the differences between them, the PL's script-based ddp does have some advantages over the spawn-based ddp.