Memory using Redistribute on GPUs

[martinapr]

Hello!
We are using AMReX for our Plasma Physics particle Code GEMPICX (still not public yet, I am afraid). While we were trying to simulate a big spatial problem on our current GPU cluster, we ran into memory problems using AMReX' Redistribute function: after a couple of time steps, the code aborts with the following message:

amrex::Abort::0::Out of gpu memory. Free: 4435902464 Asked: 8294402048 !!!

This is a prohibitively (and unnecessarily) large number for memory to ask for in our simulation.

The cluster that we simulate this problem on is the viper GPU cluster from MPCDF:

GPU nodes:
- 228 GPU nodes, 456 APUs
- Processor type: AMD Instinct MI300A APU
- Main memory (HBM3) per APU: 128 GB
- 24 CPU cores per APU
- 228 GPU compute units per APU

https://docs.mpcdf.mpg.de/doc/computing/viper-gpu-user-guide.html

The following file is our main file reduced towards a particle push including our GEMPICX particle container and a Redistribute in each step:

Reduced.cpp

Running this file on viper GPU, we get the following output and error files:

job_viper.out.txt

Backtrace_viper.0.txt

Using the input and submission file

astro.input.txt

run_viper.sh

Due to the overhead of the GEMPICX particle container, the following files are necessary to compile the code:

GEMPIC_ComputationalDomain.cpp

GEMPIC_ComputationlDomain.H.txt

GEMPIC_Config.H.txt

GEMPIC_Parameters.H.txt

GEMPIC_Parameters.cpp

GEMPIC_ParticleGroups.H.txt

GEMPIC_Verbosity.H.txt

GEMPIC_Verbosity.cpp

Sidenote: since viper GPU is providing AMD GPUs, we also did some testing on the classical Nvidia GPU cluster raven from MPCDF:

192 GPU-accelerated compute nodes 768 GPUs, 30 TB HBM2, 14.6 PFlop/s theoretical peak performance (FP64).

https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html

Note, that the following information was not obtained, using the Reduced.cpp file, but our original main file:

raven.err.txt

run_raven.sh

Backtrace_raven.0.txt

Thank you, for taking a look into this!

[AlexanderSinn]

In the error message, it is trying to allocate 8GB however, only 4GB are available. If I understand the input script correctly, there is 2000 * 2160 with 320 particles per cell. Assuming AMREX_SPACEDIM is 2 and ParticleReal is double, this would equal 72 GiB total (2000 * 2160 *320 * ((1 + 2 + 4) * 8) / 2^30). AMReX places the individual allocations of the particle vectors inside a large memory arena (CArena). When particles move between grids, the vectors need to be resized to accommodate more particles, which can result in a full reallocation of the vector. This creates memory overhead, as the old place of the vector is too small to fit something else and can't be extended if the memory directly next to it is in use. It is not unheard of that this can result in 1.5x - 2x memory overhead.

It would be interesting to see the output from a simulation that only does 6 time steps and runs to completion. There is a CArena printout that shows the memory used vs. allocated.

Additionally, the AMReX parameters can be tweaked to try different allocation strategies, eg. having no preallocated memory or preallocating as much as possible.

amrex.the_arena_init_size = 0
amrex.the_arena_release_threshold = 0
amrex.the_arena_defragmentation = 0

[atmyers]

In addition, you could try setting amrex.vector_growth_factor = 1.1 instead of the default 1.5.

[martinapr]

thank you for all your feedback. I tried out all the suggested options -

amrex.the_arena_release_threshold = 0

would actually do the trick, and the code runs through all 5k steps. The other options did not work though.

[martinapr]

I add here the results for all the options, in case it is of interest:

amrex.the_arena_init_size = 0

init_size.txt

amrex.the_arena_defragmentation = 0

defrag.txt

amrex.vector_growth_factor = 1.1

vector_grow_factor.txt

[AlexanderSinn]

There was a similar problem a few months ago with WarpX and there the setting amrex.the_arena_init_size = 0 also reduced memory usage; however, at a heavy performance cost. I did some testing and found that #4532 could work in certain scenarios without the performance penalty; however, we didn't end up using that. Maybe you could try to test with https://github.com/AlexanderSinn/amrex/tree/Use_geometric_grow_for_resize_in_RedistributeGPU and the default the_arena_init_size or even a larger one.