GACODE GPU parallelization on Princeton Traverse Cluster

[jfparisi]

I would like to run gacode with multi-node GPU support on Princeton's Traverse cluster. I encounter an MPI_ABORT issue when running on any number of nodes using srun on Traverse (previously, mpirun ran successfully on a single node). Traverse IT support told me that to run on multiple nodes requires srun. I have attached the exec and wrap files and an example slurm submission script that reproduce this issue. I have tried various iterations of wrap.TRAVERSE files inspired by the existing wrap.* files for other machines.

Possibly important details: -C gpu for srun is not supported. OMPI variables are also not supported.

Thanks!
job.slurm_2node.txt
exec.TRAVERSE.txt
wrap.TRAVERSE.txt

[jcandy]

@jfparisi Sorry for the delay. Maybe to start you could push the current TRAVERSE platform files to a development branch and we can work from there.

Can we try a case where there is 1 MPI task per GPU (so, 8 MPI tasks spread over 2 nodes)? I think TRAVERSE has 128 threads/node, so -nomp 32 would work.

[jfparisi]

@jcandy Thanks for the reply! I attempted pushing a new branch: git push -u origin development_pppl_traverse but got an error remote: Write access to repository not granted. Could you advise on how I should proceed?

I tried a case with -n 8 -nomp 32 -numa 1 -mpinuma 0 over 2 nodes, but got similar error as before

MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 SPLIT FROM 0
with errorcode 1.

[jcandy]

I just gave you developer access. If you push I will have a look, then probably ping Igor.

[jfparisi]

Thanks, I pushed the dev branch.