How to run test in parallel and GPU acceleration?

[tianninglyu]

Hi all,

I am new in using QUOKKA code and I feel sorry for my naive question. I follow the steps in "installation" part in the documentation, but it seems running "ninja test" should lead to only use one CPU core to do all the tests, because in the output file, run time is almost equal to CPU time. I compile the code in 1D and CPU only, my script is as follows:

#!/bin/bash

#BSUB -J quokka_test         
#BSUB -o quokka_test_%J.out  
#BSUB -e quokka_test_%J.err  
#BSUB -q snode    
#BSUB -n 8                  
#BSUB -R "span[hosts=1]"    
#BSUB -W 00:30              

echo "--- Task Information ---"
echo "Task ID: $LSB_JOBID"
echo "Task starts on node $(hostname)"
echo "Initial work dir: $(pwd)"
echo "----------------------"
echo ""

echo "Setting environment..."

source "/public/home/users/ssctmf038t/anaconda3/etc/profile.d/conda.sh"

conda activate quokka

echo "Go to build dir..."

cd /public/home/users/ssctmf038t/Software/quokka/build

echo "start 'ninja test'..."
ninja test

echo "--- Task completed ---"

Now I have compiled the code in 3D and with GPU enabled. I want to use one node on our supercomputer (including 24 cpu cores and 4 V100 cards) to run the test. Could you please tell me how to do?

Yours Sincerely,
Tianning

[dosubot[bot]]

Hi @tianninglyu! I'm Dosu and I’m helping the quokka team.

To run Quokka tests in parallel, use ctest -j N (for example, ctest -j8 to use 8 CPU cores). This is more effective than ninja test, which runs tests serially by default. For GPU acceleration, make sure you have built Quokka with GPU support using the CMake option -DAMReX_GPU_BACKEND=CUDA and that your environment has CUDA ≥ 11.7 and CUDA-aware MPI. On a node with multiple GPUs, all GPUs should be visible to each MPI rank for efficient communication. You can also exclude slow tests on GPUs (like matter-energy exchange tests) with ctest -E "MatterEnergyExchange*" if needed. Example cluster scripts are available in the scripts/ directory of the repository for reference. See installation and GPU notes.

If you want to specify which GPUs to use, you can set the CUDA_VISIBLE_DEVICES environment variable in your job script before running tests. For example:

export CUDA_VISIBLE_DEVICES=0,1,2,3
ctest -j24

This will use all 24 CPU cores and all 4 V100 GPUs on your node.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[markkrumholz]

Hi @tianninglyu, just to add a bit to what the bot said: instructions for how to run on GPUs on a supercomputer tend to be a bit machine-specific, but you can see example scripts to run on GPU for a large number of systems in the scripts/pbs and scripts/slurm directories. The best way to proceed is probably to use one of these scripts as a starting point (depending on whether your system uses PBS or slurm as its queue manager), then modify the environment variables and command options following the instructions that should be available as part of the documentation for whichever HPC system you're running on.

[tianninglyu]

So the actual mode is completely opposite to what is described in A: each test is performed serially using a single core, with different cores responsible for different tasks.

[markkrumholz]

Yes, I believe that is correct, though I'd need to dive into the ctest documentation to be sure: when you run ctest in parallel, it will run each test serially on a single core, and just run multiple tests at once. This is exactly analogous to the way make, cmake, and similar things handle parallelism.

[tianninglyu]

That might not be consistent with my purpose. I want to use parallel + GPU acceleration to run a simulation to test the speed of QUOKKA on different clusters, rather than running many single-core simulations in parallel.

[markkrumholz]

OK, in that case you don't want to use ctest. Instead, the best way to accomplish this is to just to run a single problem on different numbers of GPUs. You can find examples of scripts that do exactly this sort of testing in the scripts directory -- for example, the series of scripts scripts/slurm/frontier-1node.submit, frontier-8node.submit, frontier-64node.submit are intended to do a weak scaling test where we run the exact same problem (in this case the test_hydro3d_blast problem) using a 512^3 grid on one node, a 1024^3 grid on 8 nodes (so work per node is fixed), a 2048^3 grid on 64 nodes, etc. If you want to do speed testing, you should follow this pattern.

[tianninglyu]

Thanks a lot! My supercomputer uses LSF system. But it doesn't matter. I will modify the slurm script to LSF.

[tianninglyu]

Sorry one more question. Is it normal to run out of memory when running ctest with 3D gpu?

[markkrumholz]

That depends on how big a problem you're running, and on how much memory you have on your GPU cards. The smallest machine we use for regression testing has a single V100 card with 32 GB of memory, so the standard tests should fit within that memory footprint. Are you seeing a failure at that memory size on one of the tests?

[tianninglyu]

I didn't modify anything, just ran ctest. Most of the tasks showed out of memory issues.

[tianninglyu]

I think this may have something to do with all the tests launched by ctest only using a single core. Because I run the hydro3dblast tests in parallel using the following script:

#!/bin/bash

#SBATCH -J quokka_test_hydroblast3d
#SBATCH -o 1node-%x-%j.out
#SBATCH -t 01:00:00
#SBATCH -p Nvidia_A800
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=8
#SBATCH --gpus-per-node=8
#SBATCH -N 1

set -e

source /public/home/lvtianning/init/init.sh
source /public/home/lvtianning/miniconda3/etc/profile.d/conda.sh
conda activate quokka
export MPICH_GPU_SUPPORT_ENABLED=1

srun --mpi=pmix build_3d_gpu/src/problems/HydroBlast3D/test_hydro3d_blast inputs/benchmark_unigrid_256.in

And it works very well.

[tianninglyu]

Here is my script to run ctest:
For V100 machine using LSF:

#!/bin/bash
#BSUB -J quokka_3d_gpu_test
#BSUB -q gpu-v100
#BSUB -n 24
#BSUB -R "span[ptile=24]"
#BSUB -gpu "num=4"
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -R "select[hname='g10']"

echo "--- Task Information ---"
echo "Task ID: $LSB_JOBID"
echo "Task starts on node $(hostname)"
echo "Initial work dir: $(pwd)"
echo "----------------------"
echo ""

echo "Setting environment..."

source "/public/home/users/ssctmf038t/anaconda3/etc/profile.d/conda.sh"

conda activate quokka

echo "Go to build dir..."

cd /public/home/users/ssctmf038t/Software/quokka/build_3d_gpu

echo "start ctest..."

/public/home/users/ssctmf038t/anaconda3/envs/quokka/bin/ctest -j24

echo "--- Task completed ---"

For A800 machine using SLURM:

#!/bin/bash

#SBATCH -J quokka_ctest
#SBATCH -o 1node-%x-%j.out
#SBATCH -t 03:00:00
#SBATCH -p Nvidia_A800
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=8
#SBATCH --gpus-per-node=8
#SBATCH -N 1

source /public/home/lvtianning/init/init.sh
source /public/home/lvtianning/miniconda3/etc/profile.d/conda.sh
conda activate quokka
export MPICH_GPU_SUPPORT_ENABLED=1

ctest -j8

Both of them shows 'out of memory' issue.

[markkrumholz]

Ok, so the problem here is that if you do ctest -j N, then ctest will run N test jobs in parallel, and each of those will request memory on the GPU simultaneously. You’re running out of memory because you’re trying to launch every GPU test at once.