Skip to content

Commit 9d8ecf9

Browse files
roelof-groenewaldroelof-groenewald
authored
Add install instructions for ALCF's Polaris (#4636)
* add polaris machine files * add doc page for Polaris
1 parent 206b081 commit 9d8ecf9

File tree

5 files changed

+398
-0
lines changed

5 files changed

+398
-0
lines changed

‎Docs/source/install/hpc.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ This section documents quick-start guides for a selection of supercomputers that
4646
hpc/lxplus
4747
hpc/ookami
4848
hpc/perlmutter
49+
hpc/polaris
4950
hpc/quartz
5051
hpc/spock
5152
hpc/summit

‎Docs/source/install/hpc/polaris.rst‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
.. _building-polaris:
2+
3+
Polaris (ALCF)
4+
==============
5+
6+
The `Polaris cluster <https://docs.alcf.anl.gov/polaris/getting-started/>`__ is located at ALCF.
7+
8+
9+
Introduction
10+
------------
11+
12+
If you are new to this system, **please see the following resources**:
13+
14+
* `ALCF user guide <https://docs.alcf.anl.gov/>`__
15+
* Batch system: `PBS <https://docs.alcf.anl.gov/running-jobs/job-and-queue-scheduling/>`__
16+
* `Filesystems <https://docs.alcf.anl.gov/data-management/filesystem-and-storage/file-systems/>`__
17+
18+
.. _building-polaris-preparation:
19+
20+
Preparation
21+
-----------
22+
23+
Use the following commands to download the WarpX source code:
24+
25+
.. code-block:: bash
26+
27+
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
28+
29+
On Polaris, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.
30+
31+
.. tab-set::
32+
33+
.. tab-item:: A100 GPUs
34+
35+
We use system software modules, add environment hints and further dependencies via the file ``$HOME/polaris_gpu_warpx.profile``.
36+
Create it now:
37+
38+
.. code-block:: bash
39+
40+
cp $HOME/src/warpx/Tools/machines/polaris-alcf/polaris_gpu_warpx.profile.example $HOME/polaris_gpu_warpx.profile
41+
42+
.. dropdown:: Script Details
43+
:color: light
44+
:icon: info
45+
:animate: fade-in-slide-down
46+
47+
.. literalinclude:: ../../../../Tools/machines/polaris-alcf/polaris_gpu_warpx.profile.example
48+
:language: bash
49+
50+
Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
51+
For example, if you are member of the project ``proj_name``, then run ``nano $HOME/polaris_gpu_warpx.profile`` and edit line 2 to read:
52+
53+
.. code-block:: bash
54+
55+
export proj="proj_name"
56+
57+
Exit the ``nano`` editor with ``Ctrl`` + ``O`` (save) and then ``Ctrl`` + ``X`` (exit).
58+
59+
.. important::
60+
61+
Now, and as the first step on future logins to Polaris, activate these environment settings:
62+
63+
.. code-block:: bash
64+
65+
source $HOME/polaris_gpu_warpx.profile
66+
67+
Finally, since Polaris does not yet provide software modules for some of our dependencies, install them once:
68+
69+
.. code-block:: bash
70+
71+
bash $HOME/src/warpx/Tools/machines/polaris-alcf/install_gpu_dependencies.sh
72+
source ${CFS}/${proj%_g}/${USER}/sw/polaris/gpu/venvs/warpx/bin/activate
73+
74+
.. dropdown:: Script Details
75+
:color: light
76+
:icon: info
77+
:animate: fade-in-slide-down
78+
79+
.. literalinclude:: ../../../../Tools/machines/polaris-alcf/install_gpu_dependencies.sh
80+
:language: bash
81+
82+
83+
.. tab-item:: CPU Nodes
84+
85+
*Under construction*
86+
87+
88+
.. _building-polaris-compilation:
89+
90+
Compilation
91+
-----------
92+
93+
Use the following :ref:`cmake commands <building-cmake>` to compile the application executable:
94+
95+
.. tab-set::
96+
97+
.. tab-item:: A100 GPUs
98+
99+
.. code-block:: bash
100+
101+
cd $HOME/src/warpx
102+
rm -rf build_pm_gpu
103+
104+
cmake -S . -B build_pm_gpu -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
105+
cmake --build build_pm_gpu -j 16
106+
107+
The WarpX application executables are now in ``$HOME/src/warpx/build_pm_gpu/bin/``.
108+
Additionally, the following commands will install WarpX as a Python module:
109+
110+
.. code-block:: bash
111+
112+
cd $HOME/src/warpx
113+
rm -rf build_pm_gpu_py
114+
115+
cmake -S . -B build_pm_gpu_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
116+
cmake --build build_pm_gpu_py -j 16 --target pip_install
117+
118+
.. tab-item:: CPU Nodes
119+
120+
*Under construction*
121+
122+
Now, you can :ref:`submit Polaris compute jobs <running-cpp-polaris>` for WarpX :ref:`Python (PICMI) scripts <usage-picmi>` (:ref:`example scripts <usage-examples>`).
123+
Or, you can use the WarpX executables to submit Polaris jobs (:ref:`example inputs <usage-examples>`).
124+
For executables, you can reference their location in your :ref:`job script <running-cpp-polaris>` or copy them to a location in ``$PSCRATCH``.
125+
126+
127+
.. _building-polaris-update:
128+
129+
Update WarpX & Dependencies
130+
---------------------------
131+
132+
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
133+
134+
.. code-block:: bash
135+
136+
cd $HOME/src/warpx
137+
138+
# read the output of this command - does it look ok?
139+
git status
140+
141+
# get the latest WarpX source code
142+
git fetch
143+
git pull
144+
145+
# read the output of these commands - do they look ok?
146+
git status
147+
git log # press q to exit
148+
149+
And, if needed,
150+
151+
- :ref:`update the polaris_gpu_warpx.profile or polaris_cpu_warpx files <building-polaris-preparation>`,
152+
- log out and into the system, activate the now updated environment profile as usual,
153+
- :ref:`execute the dependency install scripts <building-polaris-preparation>`.
154+
155+
As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_pm_*`` and rebuild WarpX.
156+
157+
158+
.. _running-cpp-polaris:
159+
160+
Running
161+
-------
162+
163+
.. tab-set::
164+
165+
.. tab-item:: A100 (40GB) GPUs
166+
167+
The batch script below can be used to run a WarpX simulation on multiple nodes (change ``<NODES>`` accordingly) on the supercomputer Polaris at ALCF.
168+
169+
Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``.
170+
Note that we run one MPI rank per GPU.
171+
172+
.. literalinclude:: ../../../../Tools/machines/polaris-alcf/polaris_gpu.pbs
173+
:language: bash
174+
:caption: You can copy this file from ``$HOME/src/warpx/Tools/machines/polaris-alcf/polaris_gpu.pbs``.
175+
176+
To run a simulation, copy the lines above to a file ``polaris_gpu.pbs`` and run
177+
178+
.. code-block:: bash
179+
180+
qsub polaris_gpu.pbs
181+
182+
to submit the job.
183+
184+
185+
.. tab-item:: CPU Nodes
186+
187+
*Under construction*

‎Tools/machines/polaris-alcf/install_gpu_dependencies.sh‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
#!/bin/bash
2+
#
3+
# Copyright 2024 The WarpX Community
4+
#
5+
# This file is part of WarpX.
6+
#
7+
# Author: Axel Huebl (edited by Roelof Groenewald for Polaris)
8+
# License: BSD-3-Clause-LBNL
9+
10+
# Exit on first error encountered #############################################
11+
#
12+
set -eu -o pipefail
13+
14+
# Check: ######################################################################
15+
#
16+
# Was polaris_gpu_warpx.profile sourced and configured correctly?
17+
if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your polaris_gpu_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi
18+
19+
# Remove old dependencies #####################################################
20+
#
21+
SW_DIR="/home/${USER}/sw/polaris/gpu"
22+
rm -rf ${SW_DIR}
23+
mkdir -p ${SW_DIR}
24+
25+
# remove common user mistakes in python, located in .local instead of a venv
26+
python3 -m pip uninstall -qq -y pywarpx
27+
python3 -m pip uninstall -qq -y warpx
28+
python3 -m pip uninstall -qqq -y mpi4py 2>/dev/null || true
29+
30+
# General extra dependencies ##################################################
31+
#
32+
33+
# c-blosc (I/O compression)
34+
if [ -d $HOME/src/c-blosc ]
35+
then
36+
cd $HOME/src/c-blosc
37+
git fetch --prune
38+
git checkout v1.21.1
39+
cd -
40+
else
41+
git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git $HOME/src/c-blosc
42+
fi
43+
rm -rf $HOME/src/c-blosc-pm-gpu-build
44+
cmake -S $HOME/src/c-blosc -B $HOME/src/c-blosc-pm-gpu-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1
45+
cmake --build $HOME/src/c-blosc-pm-gpu-build --target install --parallel 16
46+
rm -rf $HOME/src/c-blosc-pm-gpu-build
47+
48+
# ADIOS2
49+
if [ -d $HOME/src/adios2 ]
50+
then
51+
cd $HOME/src/adios2
52+
git fetch --prune
53+
git checkout v2.8.3
54+
cd -
55+
else
56+
git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git $HOME/src/adios2
57+
fi
58+
rm -rf $HOME/src/adios2-pm-gpu-build
59+
cmake -S $HOME/src/adios2 -B $HOME/src/adios2-pm-gpu-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3
60+
cmake --build $HOME/src/adios2-pm-gpu-build --target install -j 16
61+
rm -rf $HOME/src/adios2-pm-gpu-build
62+
63+
# BLAS++ (for PSATD+RZ)
64+
if [ -d $HOME/src/blaspp ]
65+
then
66+
cd $HOME/src/blaspp
67+
git fetch --prune
68+
git checkout master
69+
git pull
70+
cd -
71+
else
72+
git clone https://github.com/icl-utk-edu/blaspp.git $HOME/src/blaspp
73+
fi
74+
rm -rf $HOME/src/blaspp-pm-gpu-build
75+
CXX=$(which CC) cmake -S $HOME/src/blaspp -B $HOME/src/blaspp-pm-gpu-build -Duse_openmp=OFF -Dgpu_backend=cuda -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-master
76+
cmake --build $HOME/src/blaspp-pm-gpu-build --target install --parallel 16
77+
rm -rf $HOME/src/blaspp-pm-gpu-build
78+
79+
# LAPACK++ (for PSATD+RZ)
80+
if [ -d $HOME/src/lapackpp ]
81+
then
82+
cd $HOME/src/lapackpp
83+
git fetch --prune
84+
git checkout master
85+
git pull
86+
cd -
87+
else
88+
git clone https://github.com/icl-utk-edu/lapackpp.git $HOME/src/lapackpp
89+
fi
90+
rm -rf $HOME/src/lapackpp-pm-gpu-build
91+
CXX=$(which CC) CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S $HOME/src/lapackpp -B $HOME/src/lapackpp-pm-gpu-build -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-master
92+
cmake --build $HOME/src/lapackpp-pm-gpu-build --target install --parallel 16
93+
rm -rf $HOME/src/lapackpp-pm-gpu-build
94+
95+
# Python ######################################################################
96+
#
97+
python3 -m pip install --upgrade pip
98+
python3 -m pip install --upgrade virtualenv
99+
python3 -m pip cache purge
100+
rm -rf ${SW_DIR}/venvs/warpx
101+
python3 -m venv --system-site-packages ${SW_DIR}/venvs/warpx
102+
source ${SW_DIR}/venvs/warpx/bin/activate
103+
python3 -m pip install --upgrade pip
104+
python3 -m pip install --upgrade build
105+
python3 -m pip install --upgrade packaging
106+
python3 -m pip install --upgrade wheel
107+
python3 -m pip install --upgrade setuptools
108+
python3 -m pip install --upgrade cython
109+
python3 -m pip install --upgrade numpy
110+
python3 -m pip install --upgrade pandas
111+
python3 -m pip install --upgrade scipy
112+
# MPICC="cc -target-accel=nvidia80 -shared" python3 -m pip install --upgrade mpi4py --no-cache-dir --no-build-isolation --no-binary mpi4py
113+
python3 -m pip install --upgrade openpmd-api
114+
python3 -m pip install --upgrade matplotlib
115+
python3 -m pip install --upgrade yt
116+
# install or update WarpX dependencies such as picmistandard
117+
python3 -m pip install --upgrade -r $HOME/src/warpx/requirements.txt
118+
python3 -m pip install cupy-cuda11x # CUDA 11.7 compatible wheel
119+
# optional: for libEnsemble
120+
python3 -m pip install -r $HOME/src/warpx/Tools/LibEnsemble/requirements.txt
121+
# optional: for optimas (based on libEnsemble & ax->botorch->gpytorch->pytorch)
122+
python3 -m pip install --upgrade torch # CUDA 11.7 compatible wheel
123+
python3 -m pip install -r $HOME/src/warpx/Tools/optimas/requirements.txt

‎Tools/machines/polaris-alcf/polaris_gpu.pbs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/bin/bash -l
2+
3+
#PBS -A <proj>
4+
#PBS -l select=<NODES>:system=polaris
5+
#PBS -l place=scatter
6+
#PBS -l walltime=0:10:00
7+
#PBS -l filesystems=home:eagle
8+
#PBS -q debug
9+
#PBS -N test_warpx
10+
11+
# Set required environment variables
12+
# support gpu-aware-mpi
13+
# export MPICH_GPU_SUPPORT_ENABLED=1
14+
15+
# Change to working directory
16+
echo Working directory is $PBS_O_WORKDIR
17+
cd ${PBS_O_WORKDIR}
18+
19+
echo Jobid: $PBS_JOBID
20+
echo Running on host `hostname`
21+
echo Running on nodes `cat $PBS_NODEFILE`
22+
23+
# executable & inputs file or python interpreter & PICMI script here
24+
EXE=./warpx
25+
INPUTS=input1d
26+
27+
# MPI and OpenMP settings
28+
NNODES=`wc -l < $PBS_NODEFILE`
29+
NRANKS_PER_NODE=4
30+
NDEPTH=1
31+
NTHREADS=1
32+
33+
NTOTRANKS=$(( NNODES * NRANKS_PER_NODE ))
34+
echo "NUM_OF_NODES= ${NNODES} TOTAL_NUM_RANKS= ${NTOTRANKS} RANKS_PER_NODE= ${NRANKS_PER_NODE} THREADS_PER_RANK= ${NTHREADS}"
35+
36+
mpiexec -np ${NTOTRANKS} ${EXE} ${INPUTS} > output.txt

0 commit comments

Comments
 (0)