Too much GPU memory consumption and seems wrong

[lxj1999]

When I use resnet101 with deeplabv3+ in this lib 'https://github.com/VainF/DeepLabV3Plus-Pytorch', I can train for 40 batch size on 2 GPUs, 20 batch size for each GPU, and it consumes about 36 GB. However, when I use your lib, for 2 batch size on 1 GPUs, it consumes about 15 GB

[lxj1999]

The batch size 2 is too big for my two 24G GPUs, it crush down when doing the 2400 iteration and needs to restart the computer.

[wujiang0156]

The batch size 4 is too big for my two 24G GPUs, my input img is 512x512. when the batch size 8 , my server out of memory.
How many pixels is the picture? @lxj1999

[lxj1999]

My image is 840 by 480

[wujiang0156]

My image is 840 by 480
so your batch size 2 , the same as 512. out of memory! why is the bath size so small? @everyone

[wujiang0156]

Batch size increased but speed decreased? why?

[lxj1999]

My image is 840 by 480
so your batch size 2 , the same as 512. out of memory! why is the bath size so small? @everyone

The library is over memory consumption, maybe some optimization issue and memory leakage.

[lxj1999]

Batch size increased but speed decreased? why?

The same issue, when I use batch size 2, although only takes up 15G out of 24G memory, the GPU is full-power and usage is 100%, not usual in other librarys.

[wujiang0156]

8xb2 means batch size is 2 on 1gpu, need 8 gpus for training, so we need 8 gpus for multigpus training, oh ? memory consumption?

[wujiang0156]

when I use 2 gpus, the speed is slower than 1gpu. Have you ever had this kind of problem?

[wujiang0156]

@lxj1999

Training is performed by mmengine. Maybe it's where the memory is consumed. do you run the mmengine?

[davidhuangal]

I am having this issue as well, specifically with DeepLabV3+ models.

[tlian96]

Hi, guys. Does anyone solve this kind of problems. When I use detectron2 to train Deeplabv3+ with res101 backbone, I set batch size 8 with crop size (512,1024) for my 24GB gpu. But for mmsegmentation, it will run out of gpu memory if I use batch size 8. It seems mmsegmentation consum too much memory than other library.

[tlian96]

Hi, did you solve this problem. I think I stuck in the same situation with you.

[wujiang0156]

so try to use multi-GPU to solve based on linux