Simulation doesn't run on Karolina HPC

[spadova-a]

Hello there,
I compiled WarpX with GPU support on Karolina supercomputer according to these instructions. Now, when I try to run the code using one of the examples on the web page (2d lwfa) I get this error message:

amrex::Abort::0::CUDA error 34 in file /home/spadoalz/src/warpx/build_gpu/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 164: CUDA driver is a stub library !!!
SIGABRT
--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[55073,1],0] (PID 8220)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
See Backtrace.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 6.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
bash: line 2: input: command not found
bash: line 2: input: command not found
bash: line 2: input: command not found
bash: line 2: input: command not found
bash: line 2: input: command not found
bash: line 2: input: command not found
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
bash: line 2: input: command not found

and this is the Backtrace file:

Host Name: acn27.karolina.it4i.cz
=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: ./warpx.2d() [0x99d3fe]
    amrex::BLBackTrace::print_backtrace_info(_IO_FILE*) at ??:?

 1: ./warpx.2d() [0x99f65b]
    amrex::BLBackTrace::handler(int) at ??:?

 2: ./warpx.2d() [0xa41ba4]
    amrex::Gpu::Device::Initialize() at ??:?

 3: ./warpx.2d() [0x928bea]
    amrex::Initialize(int&, char**&, bool, ompi_communicator_t*, std::function<void ()> const&, std::ostream&, std::ostream&, void (*)(char const*)) at ??:?

 4: ./warpx.2d() [0x5e7bf6]
    warpx::initialization::amrex_init(int&, char**&, bool) at ??:?

 5: ./warpx.2d() [0x4ab41a]
    main at ??:?

 6: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b2811745555]

 7: ./warpx.2d() [0x4f9ebe]
    _start at ??:?


===== TinyProfilers ======

Do you know, what could be the problem? Did something go wrong during the installation?

[ax3l]

cc @berceanu can you help ? :)

[ax3l]

@spadova-a can you post your submission script?
The error

bash: line 2: input: command not found

might point to something missing or broken in it.

Did you source $HOME/karolina_gpu_warpx.profile before submitting the script?

[berceanu]

Hi @spadova-a , since I am also planning to use WarpX on Karolina and have access there, I will try to help. The instructions we wrote are a bit outdated, as that was before Karolina changed scheduler from PBS to SLURM.
First off, which version of WarpX are you using, ie did you clone the latest development branch via

git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx

[spadova-a]

Hi @berceanu, yes I used the git clone exactly this way.... I also tried to recompile it today with newer modules with the help of the Karolina's IT support.

[spadova-a]

The input script is here: (I actually had to modify it compared to the one in the instructions because the cluster migrated to SLURM.)

#!/bin/bash -l
#SBATCH --job-name run.sh
#SBATCH --account OPEN-28-44
#SBATCH --partition qgpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --gpus=8
#SBATCH --time 0:10:00
#SBATCH --gpu-bind=map_gpu:1
#SBATCH -o out.%j
#SBATCH -e err.%j

WORK_DIR=/home/spadoalz/src
BIN=/home/spadoalz/src/build_gpu/bin
OUTPUT=/scratch/project/open-28-44/warpx/test_2d
date

cd $WORK_DIR

# executable & inputs file or python interpreter & PICMI script here
EXE=warpx.2d
INPUTS=input

# OpenMP threads per MPI rank
export OMP_NUM_THREADS=16

mpirun -np 8 bash -c " export CUDA_VISIBLE_DEVICES=\${OMPI_COMM_WORLD_LOCAL_RANK}; ./warpx.2d /scratch/project/open-28-44/warpx/2d_test/input "

#I also tried only "mpirun -np 8 ./warpx.2d /scratch/project/open-28-44/warpx/2d_test/input " and some other modifications like mpirun -np 8 ...

Nonetheless, I spent some time this afternoon with the HPC administrator and we think the problem is here:

(warpx-gpu) [[email protected] warpx]$ ldd build_gpu/bin/warpx.2d
        linux-vdso.so.1 =>  (0x00007ffd10104000)
        libcurand.so.10 => /apps/all/CUDA/12.2.0/lib/libcurand.so.10 (0x00002ae87dfd6000)
        libnvToolsExt.so.1 => /apps/all/CUDA/12.2.0/lib/libnvToolsExt.so.1 (0x00002ae88446d000)
        libcufft.so.11 => /apps/all/CUDA/12.2.0/lib/libcufft.so.11 (0x00002ae884677000)
        libmpi_cxx.so.40 => /apps/all/OpenMPI/4.1.5-NVHPC-23.5-CUDA-12.2.0/lib/libmpi_cxx.so.40 (0x00002ae87ddd8000)
        libhdf5.so.310 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/hdf5-1.14.1.2/lib/libhdf5.so.310 (0x00002ae88f3a8000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002ae88fcff000)
        libadios2_cxx11_mpi.so.2 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/libadios2_cxx11_mpi.so.2 (0x00002ae87de4d000)
        libadios2_cxx11.so.2 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/libadios2_cxx11.so.2 (0x00002ae87de56000)
        libmpi.so.40 => /apps/all/OpenMPI/4.1.5-NVHPC-23.5-CUDA-12.2.0/lib/libmpi.so.40 (0x00002ae88ff03000)
        librt.so.1 => /lib64/librt.so.1 (0x00002ae89003d000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ae890245000)
        libstdc++.so.6 => /apps/all/GCCcore/12.2.0/lib64/libstdc++.so.6 (0x00002ae890461000)
        libm.so.6 => /lib64/libm.so.6 (0x00002ae890684000)
        libgcc_s.so.1 => /apps/all/GCCcore/12.2.0/lib64/libgcc_s.so.1 (0x00002ae890986000)
        libc.so.6 => /lib64/libc.so.6 (0x00002ae8909a7000)
        /lib64/ld-linux-x86-64.so.2 (0x00002ae87ddb2000)
        libopen-rte.so.40 => /apps/all/OpenMPI/4.1.5-NVHPC-23.5-CUDA-12.2.0/lib/libopen-rte.so.40 (0x00002ae890d75000)
        libopen-pal.so.40 => /apps/all/OpenMPI/4.1.5-NVHPC-23.5-CUDA-12.2.0/lib/libopen-pal.so.40 (0x00002ae890e3d000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00002ae890f19000)
        libhwloc.so.15 => /apps/all/hwloc/2.8.0-GCCcore-12.2.0/lib/libhwloc.so.15 (0x00002ae89111c000)
        libpciaccess.so.0 => /apps/all/libpciaccess/0.17-GCCcore-12.2.0/lib/libpciaccess.so.0 (0x00002ae87dfc0000)
        libxml2.so.2 => /apps/all/libxml2/2.10.3-GCCcore-12.2.0/lib/libxml2.so.2 (0x00002ae89117d000)
        libz.so.1 => /apps/all/zlib/1.2.12-GCCcore-12.2.0/lib/libz.so.1 (0x00002ae8912e7000)
        liblzma.so.5 => /apps/all/XZ/5.2.7-GCCcore-12.2.0/lib/liblzma.so.5 (0x00002ae891302000)
        libevent_core-2.1.so.7 => /apps/all/libevent/2.1.12-GCCcore-12.2.0/lib/libevent_core-2.1.so.7 (0x00002ae89132b000)
        libevent_pthreads-2.1.so.7 => /apps/all/libevent/2.1.12-GCCcore-12.2.0/lib/libevent_pthreads-2.1.so.7 (0x00002ae87dfce000)
        libnvomp.so => /apps/all/NVHPC/23.5/Linux_x86_64/23.5/compilers/lib/libnvomp.so (0x00002ae891362000)
        libnvhpcatm.so => /apps/all/NVHPC/23.5/Linux_x86_64/23.5/compilers/lib/libnvhpcatm.so (0x00002ae892364000)
        libatomic.so.1 => /apps/all/GCCcore/12.2.0/lib/../lib64/libatomic.so.1 (0x00002ae89256f000)
        libnvcpumath.so => /apps/all/NVHPC/23.5/Linux_x86_64/23.5/compilers/lib/libnvcpumath.so (0x00002ae892579000)
        libnvc.so => /apps/all/NVHPC/23.5/Linux_x86_64/23.5/compilers/lib/libnvc.so (0x00002ae892992000)
        libadios2_core_mpi.so.2 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/libadios2_core_mpi.so.2 (0x00002ae892bf7000)
        libadios2_core.so.2 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/libadios2_core.so.2 (0x00002ae892c84000)
        libnvcpumath-avx2.so => /apps/all/NVHPC/23.5/Linux_x86_64/23.5/compilers/lib/libnvcpumath-avx2.so (0x00002ae8933c2000)
        libadios2_perfstubs.so => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/libadios2_perfstubs.so (0x00002ae8937db000)
        libadios2_core_cuda.so => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/../lib64/libadios2_core_cuda.so (0x00002ae8937e2000)
        libcudart.so.12 => /apps/all/CUDA/12.2.0/lib/libcudart.so.12 (0x00002ae893d55000)
        libcuda.so.1 => /apps/all/CUDA/12.2.0/lib/stubs/libcuda.so.1 (0x00002ae893fff000)
        libblosc.so.1 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/c-blosc-1.21.1/lib/libblosc.so.1 (0x00002ae89420f000)
        libbz2.so.1.0 => /apps/all/bzip2/1.0.8-GCCcore-12.2.0/lib/libbz2.so.1.0 (0x00002ae89433e000)
        libfabric.so.1 => /apps/all/libfabric/1.16.1-GCCcore-12.2.0/lib/libfabric.so.1 (0x00002ae894355000)
        libadios2_evpath.so => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/../lib64/libadios2_evpath.so (0x00002ae894bc2000)
        libadios2_ffs.so.1 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/../lib64/libadios2_ffs.so.1 (0x00002ae894c38000)
        libadios2_atl.so.2 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/../lib64/libadios2_atl.so.2 (0x00002ae894ca2000)
        librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00002ae894cb0000)
        libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00002ae894ec8000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00002ae8950e1000)
        libnuma.so.1 => /apps/all/numactl/2.0.16-GCCcore-12.2.0/lib/libnuma.so.1 (0x00002ae8952e6000)
        libadios2_dill.so.2 => /mnt/proj1/open-28-44/spadoalz/warpx/sw/karolina/gpu/adios2-2.8.3/lib64/../lib64/../lib64/libadios2_dill.so.2 (0x00002ae8952f5000)
        libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00002ae895328000)
        libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00002ae895596000)
        libffi.so.8 => /apps/all/libffi/3.4.4-GCCcore-12.2.0/lib64/libffi.so.8 (0x00002ae8957b7000)

Specifically here:
libcuda.so.1 => /apps/all/CUDA/12.2.0/lib/stubs/libcuda.so.1 (0x00002ae893fff000)

We believe that libcuda.so.1 shouldn't be loaded from this place, but we were not able to fix it.

[berceanu]

OK, I am working on this, and I expect it may take a couple of days, but we should be able to get it running.
I will reach back soon with additional info.

[spadova-a]

Thanks, we will keep working on it as well and let you know if we figure something up.

[spadova-a]

Hi,
so we did fix the problem with the compilation. The error occurred because of the wrongly loaded library libcuda.so. The way to the library was set in RUNPATH in the compiled binary file, so we were not able to change it. What ultimately fixed it was two things - 1) slightly changing the karolina_gpu_warpx_test.profile file 2) we compiled the file not on a login node but as an interactive job on a GPU node.

[berceanu]

Hi @spadova-a , I just made a pull request with the latest instructions for WarpX on Karolina: #4477
All the dependencies are now automatically installed by spack, and a single module from Karolina is used, the one that loads CUDA 11.7, GCC 11.3 and OpenMPI 4.1.4.
Compilation works on the login node, and all the packages were optimized for the zen3 microarchitecture.