Host memory allocation grows constantly with MPI GPU awareness on

[mszpindler]

Memory allocation on host side of WarpX job grows constantly with MPI GPU awareness ON.

This is observed on LUMI (AMD GPUs amd-gfx90a) using user's input. Reproducer run is on two one GPU node (8 devices).

WarpX was built using instruction for LUMI from the documentation.

Steps taken so far:

• Originaly observed with WarpX release 24.01 (and embedded AMReX 24.01)
• Recently tested with releases 25.03 and standalone AMReX 25.03
• Tested with AMD ROCm (6.0.3 and 6.2.2) and Cray compilers
• Tested with manually installed libfabric 2.0.0 and recent MPICH (no GTL library)

The only mitigation found so far is to disable MPI support for GPU entirely at runtime:

• MPICH_GPU_SUPPORT_ENABLED=0 for Cray MPICH
• MPICH_ENABLE_GPU=0 for vanilla MPICH
• Turning off GPU IPC only does not prevent memory grow

Observed with Slurm memory reports, pmap and valgrind.

[kngott]

Hi Maciej,

Thanks for bringing this to our attention!

Could you provide your reproducer and precisely how you're measuring the memory, so we can take a look? Also, have you tried this on a CUDA system to compare to CUDA-aware MPI, or have you only tried on LUMI with AMD-aware MPI?

[mszpindler]

How to reproduce:

1. Environment:

module load CrayEnv PrgEnv-cray rocm/6.0.3 craype-accel-amd-gfx90a buildtools

2. Configure AMReX:

cmake \
-DAMReX_SPACEDIM="1;2;3" \
-DAMReX_MPI=ON \
-DAMReX_OMP=OFF \
-DAMReX_GPU_RDC=OFF \
-DAMReX_GPU_BACKEND=HIP \
-DAMReX_AMD_ARCH=gfx90a \
-DAMReX_AMRLEVEL=OFF \
-DAMReX_EB=ON \
-DAMReX_PARTICLES=ON \
-DAMReX_FFT=ON \
-DAMReX_PIC=ON \
-DAMReX_HDF5=OFF \
-DAMReX_MEM_PROFILE=ON \
-DCMAKE_C_COMPILER=amdclang \
-DCMAKE_CXX_COMPILER=amdclang++ \
-DCMAKE_INSTALL_PREFIX=$AMREX_HOME \
..

3. Configure WarpX:

cmake \
-DWarpX_COMPUTE=HIP \
-DWarpX_FFT=ON \
-DWarpX_QED_TABLE_GEN=ON \
-DWarpX_QED_TABLES_GEN_OMP=OFF \
-DWarpX_DIMS="1;2;RZ;3" \
-DWarpX_amrex_internal=OFF \
-DAMReX_DIR=$AMREX_HOME \
-DBoost_INCLUDE_DIR=/appl/lumi/SW/LUMI-24.03/G/EB/Boost/1.83.0-cpeCray-24.03/include \
-Dblaspp_DIR=$HOME/EasyBuild/SW/LUMI-24.03/G/blaspp/2024.05.31-cpeCray-24.03-rocm/lib/cmake/blaspp \
-Dlapackpp_DIR=$HOME/EasyBuild/SW/LUMI-24.03/G/lapackpp/2024.05.31-cpeCray-24.03-rocm/lib/cmake/lapackpp \
-DAMDGPU_TARGETS=gfx90a \
-DAMReX_GPU_BACKEND=HIP \
-DCMAKE_C_COMPILER=amdclang \
-DCMAKE_CXX_COMPILER=amdclang++ \
-DCMAKE_EXE_LINKER_FLAGS="$PE_MPICH_GTL_DIR_amd_gfx90a $PE_MPICH_GTL_LIBS_amd_gfx90a" \
..

[mszpindler]

4. Input file https://gist.github.com/mszpindler/e5e74ef32f6c4802b9b52d4b3ec464c7

[mszpindler]

5. SLURM job script (including job's memory occupation summary):

#!/bin/bash
#SBATCH --job-name=multipole-mirror-testcase 
#SBATCH --partition=dev-g 
#SBATCH --time=20
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH -c 6
#SBATCH --gpus-per-node=2


export FI_MR_CACHE_MAX_COUNT=memhooks
export ROCFFT_RTC_CACHE_PATH=/dev/null
export MPICH_SMP_SINGLE_COPY_MODE=NONE
export FI_CXI_RX_MATCH_MODE=software

export OMP_NUM_THREADS=1

#export MPICH_MEMORY_REPORT=1

module load PrgEnv-cray craype-accel-amd-gfx90a rocm

export MPICH_GPU_SUPPORT_ENABLED=1   

srun ./select_gpu ./warpx.3d.MPI.HIP.DP.PDP.OPMD.FFT.EB.QED.GENQEDTABLES multipole-mirror.warpx &

sleep 60
cat /sys/fs/cgroup/system.slice/slurmstepd.scope/job_${SLURM_JOBID}/memory.stat > memory_stat_${SLURM_JOBID}.in
sleep 1080
cat /sys/fs/cgroup/system.slice/slurmstepd.scope/job_${SLURM_JOBID}/memory.stat > memory_stat_${SLURM_JOBID}.out

6. select_gpu wrapper

#!/bin/bash
export ROCR_VISIBLE_DEVICES=$SLURM_LOCALID
exec $*

[mszpindler]

7. The other method used to observe host memory allocations is:

• srun to a node executing the job
• identify PID of a process for rank 0
• watch -d pmap on PID