Excessive Memory Usage

[digantamisra98]

ResNet-26 on CIFAR-10 (8.3M params) using your implementation without any changes fails to fit in memory in HAL and even on Colab - P100, T4 and P4. What was your system configuration for doing the benchmarks?

[acthelen]

I have the same problem ResNet-26 on CIFAR-10. Running an RTX 2070 with 8gb graphics card RAM, I get the message:

Tried to allocate 308.00 MiB (GPU 0; 8.00 GiB total capacity; 5.08 GiB already allocated; 266.22 MiB free; 24.83 MiB cached)

This seems really odd. I would also like to know what hardware this was created on.

[Lishey]

I have met the same issue, the memory seems never enough.

[sammens]

It uses about 13.00 GiB so if your GPU memory is less than that, you may get excessive memory usage problem. Maybe you should reduce the batch size to resolve this.

[digantamisra98]

13 GB for a ResNet-26?

[sammens]

I don't know why it is so but this is how it looks like at the moment. Check the GPU Process 1

[digantamisra98]

Holy cow. How did this pass muster? I don't have access to my cluster now but 13.6 gigs for a ResNet-26 is absolutely ridiculous. What's the use of such costly models?

[sammens]

I agree with you. This is very costly.

[ksouvik52]

yes, this is too costly and slow to replace Convs.

[CCYChongyanChen]

This is ridiculous...I am using p2.16xlarge with 16 GPU and the CUDA memory is still not enough?

[iyaja]

Have you all tried running the experiments with different batch sizes? I was able to run the default CIFAR10 experiments with a batch size of 8 using an RTX 2060. Total memory consumption was under 5 GB. On a more powerful machine with 32 GB V100 cards, I was able to run the default experiment configuration for CIFAR10 while using ~15 GB.

[CCYChongyanChen]

Have you all tried running the experiments with different batch sizes? I was able to run the default CIFAR10 experiments with a batch size of 8 using an RTX 2060. Total memory consumption was under 5 GB. On a more powerful machine with 32 GB V100 cards, I was able to run the default experiment configuration for CIFAR10 while using ~15 GB.

@iyaja
Hi Ajay, thank you a lot for your reply!
I am running it on the ChestXray dataset. It doesn't work with even a batch size of 1.
Could I ask how did you change the batch size? I do it via --batch-size=1.
What is weird is that the RuntimeError always stays the same no matter how small the batch size I set.
Setting batch size to 32, it says "CUDA out of memory. tried to allocate 25 MB"
Setting batch size to 1, it still says "CUDA out of memory. tried to allocate 25 MB".

[CCYChongyanChen]

Have you all tried running the experiments with different batch sizes? I was able to run the default CIFAR10 experiments with a batch size of 8 using an RTX 2060. Total memory consumption was under 5 GB. On a more powerful machine with 32 GB V100 cards, I was able to run the default experiment configuration for CIFAR10 while using ~15 GB.

@iyaja
Hi Ajay, thank you a lot for your reply!
I am running it on the ChestXray dataset. It doesn't work with even a batch size of 1.
Could I ask how did you change the batch size? I do it via --batch-size=1.
What is weird is that the RuntimeError always stays the same no matter how small the batch size I set.
Setting batch size to 32, it says "CUDA out of memory. tried to allocate 25 MB"
Setting batch size to 1, it still says "CUDA out of memory. tried to allocate 25 MB".

I also have no idea why it shows gpu_devices: None and gpu: None....Shown in my last image, it seems that all 16 GPUs are occupied.

Another weird thing is... When I was using 8 GPUs, it says CUDA tried to allocate 47MB. When I changed it to 16 GPUs, it says CUDA tried to allocate 25MB....Why increasing GPU just help that little...

[SangHoon95]

I am trying to use AttentionConv on another model and I got a small Params size but got a super large Forward/backward pass size at the same time.
Why is Forward/backward pass size so large?

[chenslcool]

I am trying to use AttentionConv on another model and I got a small Params size but got a super large Forward/backward pass size at the same time. Why is Forward/backward pass size so large?

because it use unfold, it will bring huge gpu memory cost. The param is small... but intermediate featuremaps is very big.