running on multiGPU machine

[denisbertini]

I am trying to use more than one GPU for warpx on our AMD machine without success.
The setting is the following

• AMD GPU MI 100 with ROCM 5.4.x
• warpx using the std [laser_ion](example https://github.com/ECP-WarpX/WarpX/blob/development/Examples/Physics_applications/laser_ion/inputs)
• submit slurm command:sbatch --reservation gpu_tests --nodes 1 --ntasks-per-node 4 --cpus-per-task 1 --gres=gpu:4 --mem-per-gpu 48000 --no-requeue --job-name warpx --mail-type ALL --mail-user d.bertini@gsi.de --partition gpu --time 0-8:00:00 -D ./ -o %j.out.log -e %j.err.log --nodelist=lxbk1099 ./run-file.sh

From the ressource selection used (--gres=gpu:4) i would expect the usage of 4 gpus. Instead only one is used .
Is there anything else one should be aware when running on multi gpus machine?
If yes could you deliver me an example i can test on our system ?

[RemiLehe]

Thanks for your interest in the code.
Could you share the content of your run-file.sh script? In particular, are you calling an MPI runner, such as mpirun, srun or jsrun from inside run-file.sh?

[denisbertini]

for example submitting with the above slurm command gives at initialisation:

MPI initialized with 4 MPI processes
MPI initialized with thread support level 3
Initializing HIP...
HIP initialized with 1 device.
AMReX (23.05) initialized
PICSAR (1903ecfff51a)
WarpX (23.05)

    __        __             __  __
    \ \      / /_ _ _ __ _ __\ \/ /
     \ \ /\ / / _` | '__| '_ \\  /
      \ V  V / (_| | |  | |_) /  \
       \_/\_/ \__,_|_|  | .__/_/\_\
                        |_|

Level 0: dt = 1.530214125e-17 ; dx = 5.580357143e-09 ; dz = 8.081896552e-09

Grids Summary:
  Level 0   29 grids  9977856 cells  100 % of domain
            smallest grid: 2688 x 128  biggest grid: 2688 x 128

Should it be HIP initialised with 4 devices and 1 GPU device per MPI rank?

[denisbertini]

run-file.sh :

#!/bin/bash

#export CONT=/lustre/rz/dbertini/containers/prod/gpu/rlx8_rocm-5.4.3_warpx.sif
export CONT=/cvmfs/phelix.gsi.de/sifs/gpu/rlx8_rocm-5.4.3_warpx.sif
export WDIR=/lustre/rz/dbertini/gpu/warpx

export OMPI_MCA_io=romio321
export APPTAINER_BINDPATH=/lustre/rz/dbertini/,/cvmfs

srun --export=ALL -- $CONT warpx_2d  $WDIR/scripts/inputs/warpx_opmd_deck

[RemiLehe]

OK, thanks.
Based on the output that you sent, I think that WarpX is actually using 4 GPUs. I think that the message
HIP initialized with 1 device. is to be understood per MPI rank.
@WeiqunZhang Could you confirm that this is the case?

[denisbertini]

unfortunately on the node where the job is running i can notice the use of only one GPU, as it is showed by the utility
program rocm-smi

[denisbertini]

this for example a snapshot of the rocm-smi output when the job is running:

GPU  Temp (DieEdge)  AvgPwr  SCLK     MCLK     Fan  Perf  PwrCap  VRAM%  GPU%
0    71.0c           198.0W  1502Mhz  1200Mhz  0%   auto  290.0W   84%   99%
1    38.0c           34.0W   300Mhz   1200Mhz  0%   auto  290.0W    2%   0%
2    38.0c           34.0W   300Mhz   1200Mhz  0%   auto  290.0W    2%   0%
3    39.0c           34.0W   300Mz   1200Mhz  0%   auto  290.0W    2%   0%
4    59.0c           40.0W   300Mhz  1200Mhz  0%   auto  290.0W    0%   0%
5    38.0c           34.0W   300Mhz  1200Mhz  0%   auto  290.0W    0%   0%
6    40.0c           34.0W   300Mhz  1200Mhz  0%   auto  290.0W    0%   0%
7    40.0c           35.0W   300Mhz  1200Mhz  0%   auto  290.0W    0%   0%
=====40===============9========================================================
=====40===============4====== End o ROCm SMI Log ==============================

one can see that only GPU at index 0 is used the other 4 requested are idle... any idea?

[WeiqunZhang]

HIP initialized with 1 device. means only one device is being used by all the processes. This is most likely a job script issue.

[denisbertini]

yes the question for me it still not clear what could be possibly wring in my job script ...

[WeiqunZhang]

It depends on how slurm is configured on that system. Maybe try to change --cpus-per-task 1. Figure out how many CPUs you have on a node and divide that by 4. The issue might be all your processes were using cpus that were close one GPU and that GPU was mapped to all 4 processes. There is nothing we can do in a C++ code, if the GPUs are not visible to us.

[denisbertini]

i have 96 processors on one machine, so if i change to --cpus-per-task 24 i still use only one GPU, does not help

[WeiqunZhang]

Maybe instead of --gres=gpu:4, you can try --gpus-per-taks=1 and --gpu-bind=verbose,single:1.

[denisbertini]

doing your change i got the following error:

gpu-bind: usable_gres=0x8; bit_alloc=0xF; local_inx=4; global_list=3; local_list=3
gpu-bind: usable_gres=0x1; bit_alloc=0xF; local_inx=4; global_list=0; local_list=0
gpu-bind: usable_gres=0x4; bit_alloc=0xF; local_inx=4; global_list=2; local_list=2
gpu-bind: usable_gres=0x2; bit_alloc=0xF; local_inx=4; global_list=1; local_list=1
amrex::Abort::1::HIP error in file /tmp/warp/WarpX/build_2d/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 163 no ROCm-capable device is detected !!!
SIGABRT
amrex::Abort::2::HIP error in file /tmp/warp/WarpX/build_2d/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 163 no ROCm-capable device is detected !!!
SIGABRT
amrex::Abort::3::HIP error in file /tmp/warp/WarpX/build_2d/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 163 no ROCm-capable device is detected !!!

[denisbertini]

so it seems that from warpx or amrex perspective, there is only one GPU device on the node ?

[denisbertini]

resubmitting with --ntasks-per-node 1 works fine, only one GPU is visible to warpx

[WeiqunZhang]

The error message means processes 1, 2 and 3 see zero GPUs as reported by hipGetDeviceCount. Only process 0 sees a GPU.

You can also run rocm-smi instead of warpx. I suspect you will see the same behavior. That is only one GPU in total is available.