Skip to content

Doc: WarpX no-MPI Perlmutter Container #6422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: development
Choose a base branch
from
+282 −0
Open

Doc: WarpX no-MPI Perlmutter Container #6422

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 ...s/machines/perlmutter-nersc/Containerfile → ...ter-nersc/container/gpu/mpi/Containerfile
View file
Open in desktop
File renamed without changes.
0 ...s/machines/perlmutter-nersc/entrypoint.sh → ...ter-nersc/container/gpu/mpi/entrypoint.sh
View file
Open in desktop
File renamed without changes.
269 changes: 269 additions & 0 deletions Tools/machines/perlmutter-nersc/container/gpu/nompi/Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,269 @@
# Build this container from its current directory:
# podman build --build-arg NJOBS=6 -t warpx-perlmutter-nompi .
# Adjust NJOBS to the number of processes to use for parallel compilation.
#
# WarpX executables are installed in /opt/warpx/bin
# WarpX Python bindings are installed in /opt/venv
#
# On Perlmutter, run WarpX like this:
# podman-hpc run --rm --gpu -v $PWD:/opt/pwd -it registry.nersc.gov/m558/superfacility/warpx-perlmutter-nompi:latest warpx.2d /opt/pwd/inputs_base_2d
# Prefix this line with "srun ..." in job scripts, as usual:
# srun --cpu-bind=cores bash -c "
# export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID));
# podman run ... ${INPUTS}" \
# > output.txt
#

# Base System and Essential Tools Installation
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04 AS warpx-dev

# WarpX version
ARG warpx_version=25.11

# Build parallelism
ARG NJOBS
ENV NJOBS=$NJOBS

# Perlmutter GPU: A100
# Perlmutter CPU: Zen v3
ENV AMREX_CUDA_ARCH=8.0 \
CUDAARCHS=80 \
CXXFLAGS="-march=znver3" \
CFLAGS="-march=znver3"

# Ubuntu apt global options
ENV DEBIAN_FRONTEND=noninteractive

# CMake global options
ENV CMAKE_GENERATOR="Ninja"

# Autotools global options
ENV FORCE_UNSAFE_CONFIGURE=1

# Pip global options
ENV PYTHONDONTWRITEBYTECODE=1

# Custom install location
ENV PATH="/opt/warpx/bin:${PATH}"
ENV LD_LIBRARY_PATH="/opt/warpx/lib:${LD_LIBRARY_PATH}"
ENV CMAKE_PREFIX_PATH="/opt/warpx:${CMAKE_PREFIX_PATH}"

# Install essential system dependencies (development)
RUN apt-get update && \
apt-get install \
-y \
--no-install-recommends \
autoconf \
build-essential \
ca-certificates \
cmake \
coreutils \
curl \
g++ \
git \
pkg-config \
python3 \
python3-pip \
python3-dev \
python3-venv \
unzip \
libblas-dev \
libbz2-dev \
libhdf5-dev \
libpng-dev \
liblapack-dev \
libzstd-dev \
ninja-build \
zlib1g-dev && \
rm -rf /var/lib/apt/lists/*

# Install c-blosc from source
RUN git clone \
-b v2.14.4 \
https://github.com/Blosc/c-blosc2.git \
/tmp/c-blosc2 && \
cmake \
-S /tmp/c-blosc2 \
-B /tmp/c-blosc2-build \
-DCMAKE_INSTALL_PREFIX=/opt/warpx \
-DBUILD_TESTS=OFF \
-DBUILD_BENCHMARKS=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_FUZZERS=OFF \
-DDEACTIVATE_AVX2=OFF \
&& \
cmake --build /tmp/c-blosc2-build \
--target install \
-j${NJOBS} && \
rm -rf /tmp/c-blosc2*

# Install ADIOS2 from source
# TODO: -DADIOS2_USE_SST=ON
RUN git clone \
-b v2.10.2 \
https://github.com/ornladios/ADIOS2.git \
/tmp/adios2 \
&& \
cmake -S /tmp/adios2 \
-B /tmp/adios2-build \
-DADIOS2_USE_Blosc2=ON \
-DADIOS2_USE_BZip2=ON \
-DADIOS2_USE_Fortran=OFF \
-DADIOS2_USE_SST=OFF \
-DADIOS2_USE_SZ=OFF \
-DADIOS2_USE_MGARD=OFF \
-DADIOS2_USE_MPI=OFF \
-DADIOS2_USE_PNG=ON \
-DADIOS2_USE_Python=OFF \
-DADIOS2_USE_ZeroMQ=OFF \
-DADIOS2_USE_ZFP=OFF \
-DCMAKE_INSTALL_PREFIX=/opt/warpx \
&& \
cmake --build /tmp/adios2-build \
--target install -j${NJOBS} && \
rm -rf /tmp/adios2*

# Install BLAS++ and LAPACK++
RUN git clone \
-b v2024.05.31 \
https://github.com/icl-utk-edu/blaspp.git \
/tmp/blaspp && \
cd /tmp/blaspp && \
cmake -S /tmp/blaspp \
-B /tmp/blaspp-build \
-Duse_openmp=OFF \
-Dgpu_backend=cuda \
-DCMAKE_CXX_STANDARD=17 \
-DCMAKE_INSTALL_PREFIX=/opt/warpx \
-DBLAS_LIBRARIES=/usr/lib/x86_64-linux-gnu/libblas.so \
-DLAPACK_LIBRARIES=/usr/lib/x86_64-linux-gnu/liblapack.so \
&& \
cmake --build /tmp/blaspp-build \
--target install \
-j${NJOBS} && \
rm -rf /tmp/blaspp*

RUN git clone \
-b v2024.05.31 \
https://github.com/icl-utk-edu/lapackpp.git \
/tmp/lapackpp \
&& \
cmake -S /tmp/lapackpp \
-B /tmp/lapackpp-build \
-DCMAKE_CXX_STANDARD=17 \
-Dbuild_tests=OFF \
-DCMAKE_INSTALL_PREFIX=/opt/warpx \
-DLAPACK_LIBRARIES=/usr/lib/x86_64-linux-gnu/liblapack.so \
&& \
cmake --build /tmp/lapackpp-build \
--target install \
-j${NJOBS} && \
rm -rf /tmp/lapackpp*

# Create and activate Python virtual environment
ENV PATH="/opt/venv/bin:${PATH}"
RUN python3 -m venv /opt/venv && \
pip install \
--no-cache-dir \
--upgrade \
pip \
uv \
&& \
uv pip install \
--no-cache-dir \
--upgrade \
build \
cmake \
cupy-cuda12x \
cython \
h5py \
matplotlib \
numpy \
openpmd-api \
openpmd-viewer \
pandas \
scipy \
packaging \
setuptools[core] \
wheel

# TODO: if we build cupy ourselves, we can likely shrink
# the container size

# Optional: PyTorch (can be added on top as well in a warpx-ml container)
# \
# && \
# uv pip install \
# --no-cache-dir \
# --upgrade \
# --index-url https://download.pytorch.org/whl/cu126 \
# torch
# optimas[all]

# Build WarpX
# TODO: -DWarpX_DIMS="1;2;RCYLINDER;RSPHERE;RZ;3"
# TODO: -DWarpX_QED_TABLE_GEN=ON (needs boost math)
RUN git clone \
-b ${warpx_version} \
https://github.com/BLAST-WarpX/warpx.git \
/tmp/warpx-src && \
cmake \
-S /tmp/warpx-src \
-B /tmp/warpx-build \
-DCMAKE_INSTALL_PREFIX=/opt/warpx \
-DWarpX_COMPUTE=CUDA \
-DWarpX_DIMS="1;2;RZ;3" \
-DWarpX_FFT=ON \
-DWarpX_MPI=OFF \
-DWarpX_PYTHON=ON \
-DWarpX_QED_TABLE_GEN=OFF \
&& \
cmake --build /tmp/warpx-build \
--target install \
-j${NJOBS} \
&& \
cmake --build /tmp/warpx-build \
--target pip_install \
-j${NJOBS} \
&& \
rm -rf /tmp/warpx* && \
ln -s /opt/warpx/bin/warpx.1d* /opt/warpx/bin/warpx.1d && \
ln -s /opt/warpx/bin/warpx.2d* /opt/warpx/bin/warpx.2d && \
ln -s /opt/warpx/bin/warpx.rz* /opt/warpx/bin/warpx.rz && \
ln -s /opt/warpx/bin/warpx.3d* /opt/warpx/bin/warpx.3d

# Reduce to runtime
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04 AS warpx

# Install essential system dependencies (runtime)
RUN apt-get update && \
apt-get install \
-y \
--no-install-recommends \
ca-certificates \
coreutils \
python3 \
python3-pip \
python3-venv \
libblas3 \
libbz2-1.0 \
libhdf5-103-1 \
libpng16-16 \
liblapack3 \
libzstd1 \
zlib1g && \
rm -rf /var/lib/apt/lists/*

# Copy installed software from previous stage
# TODO: Check BLAS still works
# TODO: Check this does not copy CUDA devel packages
COPY --from=warpx-dev /opt/warpx /opt/warpx
COPY --from=warpx-dev /opt/venv /opt/venv

# Set up entrypoint
COPY entrypoint.sh /opt/entrypoint.sh
RUN chmod +x /opt/entrypoint.sh
ENTRYPOINT ["/opt/entrypoint.sh"]

# Default command
CMD ["/bin/bash", "-l"]
13 changes: 13 additions & 0 deletions Tools/machines/perlmutter-nersc/container/gpu/nompi/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash --login

# Hint custom install location
export WARPX_ROOT=/opt/warpx
export CMAKE_PREFIX_PATH=${WARPX_ROOT}:${CMAKE_PREFIX_PATH}
export PATH=${WARPX_ROOT}/bin:${PATH}
export LD_LIBRARY_PATH=${WARPX_ROOT}/lib:${LD_LIBRARY_PATH}

# Activate python venv
source /opt/venv/bin/activate

# Execute the provided command
exec "$@"
Loading