Rank to GPU mapping warning

[denisbertini]

Using the latest ( 24.01) WarpX version i get these warning on the 8 AMD-GPU node using exactly 8 MPI ranks:

Multiple GPUs are visible to each MPI rank, This may lead to incorrect or suboptimal rank-to-GPU mapping.!

SLURM submit script is the following:

sbatch  --reservation=dbertini --nodes 1 --ntasks-per-node 8 --cpus-per-task 12 --gres=gpu:8 --gpu-bind verbose,closest --mem-per-gpu 48000 --no-requeue --job-name warpx  --mail-type ALL --mail-user d.bertini@gsi.de --partition gpu --time 7-0:00:00 -D ./ -o %j.out.log -e %j.err.log   ./run-file.sh

Any idea what could be wrong here ?

[WeiqunZhang]

It's a warning. The issue is each MPI process can see 8 GPUs. In that case, we try our best to assign GPUs to MPI processes. However, the mapping may not be optimal. For example, a GPU might be assigned to a CPU far away.

You could try --gpus-per-task=1 instead of --gres=gpu:8 to see if it helps.

[denisbertini]

using --gpus-per-task=1 instead of --gres=gpu:8 did not help.

[denisbertini]

Can i just ignore this warning or my job will have unbalanced workload on GPU device ?
How to change that mapping ?

[WeiqunZhang]

You could ignore it. It's not a correctness issue, but the performance may not be optimal.

You might be able to use ROCR_VISIBLE_DEVICES to control which GPU is visible. For example, on perlmutter (which has 4 gpus per node), one could do something like, https://warpx.readthedocs.io/en/latest/install/hpc/perlmutter.html#id1

# CUDA visible devices are ordered inverse to local task IDs
#   Reference: nvidia-smi topo -m
srun --cpu-bind=cores bash -c "
    export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
    ${EXE} ${INPUTS} ${GPU_AWARE_MPI}" \
  > output.txt

The problem is the mappings are different on different machines. You may not want to use the perlmutter approach on other machines. For example, on Frontier, GPU 0 and 1 are closest to NUMA3, GPU 2 and 3 are closet to NUMA1, GPU 4 and 5 are closest to NUMA0 and GPU 6 and 7 are closest to NUMA2. So you should try to read the user guide for your system or ask the system's administrators for help.

[denisbertini]

I run now my simulation on 8 nodes each having 8 GPUs devices.
From the WarpX output i see the following

Initializing AMReX (24.01)...
MPI initialized with 64 MPI processes
MPI initialized with thread support level 3
Initializing HIP...
HIP initialized with 64 devices.
AMReX (24.01) initialized
PICSAR (23.09)
WarpX (24.01)

    __        __             __  __
    \ \      / /_ _ _ __ _ __\ \/ /
     \ \ /\ / / _` | '__| '_ \\  /
      \ V  V / (_| | |  | |_) /  \
       \_/\_/ \__,_|_|  | .__/_/\_\
                        |_|

Level 0: dt = 5.471961054e-15 ; dx = 1.937984496e-06 ; dy = 7.352941176e-05 ; dz = 3.680147059e-05

Grids Summary:
  Level 0   64 grids  2576862720 cells  100 % of domain
            smallest grid: 967 x 204 x 204  biggest grid: 968 x 204 x 204

So 64 MPI rank for 8*8 GPUs...
But how to be sure that only one MPI rank matches exaclty 1 GPU devices?
From the warning it is unclear ...

[WeiqunZhang]

The warning is not about multiple processes use the same GPU. It's about a process sees multiple GPU. Suppose there are two CPUs, one in north pole and the other in south pole, and there are also two GPUs, one in north pole and the other in south pole. If each process only see one GPU, then there is no choice to make and WarpX will not issue a warning even if the north pole CPU sees only the south pole GPU. If each process sees two GPUs, we will make a choice. But it might be the wrong choice that uses the north pole GPU on the south pole CPU.

[denisbertini]

Ah it is about affinity optimisation then ... not one MPI rank using 2 GPUs for example

[WeiqunZhang]

Right. It's about affinity between CPUs and GPUs within a node. Unless there are more processes per node than the number of GPUs per node, we will make sure each process uses a unique GPU.

[denisbertini]

but what if i use the following slurm option:

--gpu-bind verbose,closest

?

[denisbertini]

Other question, i do not see in the warpx example usage of mem-per-gpu slurm option, is there any reason for that ?

[denisbertini]

Indeed when using ROCR_VISIBLE_DEVICES the warning is gone ...

[WeiqunZhang]

Make sure the right GPU is visible.

[denisbertini]

i am using the same script as proposed in the documentation:

# CUDA visible devices are ordered inverse to local task IDs
#   Reference: nvidia-smi topo -m
srun --cpu-bind=cores bash -c "
    export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
    ${EXE} ${INPUTS} ${GPU_AWARE_MPI}" \
  > output.tx

and no warning anymore, so i suppose the mapping is now correct...

[WeiqunZhang]

For perlmutter, the mapping is gpu 0 is mapped to slurm cpu id 3, and so on.

If each process only see one GPU, then there is no choice to make and WarpX will not issue a warning even if the north pole CPU sees only the south pole GPU.

No warning does not mean it's correct. It's just that it's not WarpX's fault anymore even if it's incorrect, because warpx cannot do anything else.

[WeiqunZhang]

In fact, ignoring the warning might have a better chance of being right than using ROCR_VISIBLE_DEVICES without knowing how the hardware in the system is mapping.