HIP + MPI Compilation Question

[denisbertini]

Hi
Trying to build with CMake 3.20.5 i got the following problem:

Singularity> cmake -S . -B build -DWarpX_DIMS=2 -DWarpX_COMPUTE=HIP -DAMReX_GPU_BACKEND=HIP -DCMAKE_CXX_COMPILER=hipcc -DAMReX_AMD_ARCH="gfx908" -DMPI_C_COMPILER=mpicc -DMPI_CXX_COMPILER=mpicxx -DCMAKE_PREFIX_PATH=/usr/lib/x86_64-linux-gnu/openmpi -DCMAKE_INSTALL_PREFIX=../warpx_install ../warpx_src
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rocm/llvm/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rocm/bin/hipcc - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find CCache
-- Downloading AMReX ...
-- AMReX repository: https://github.com/AMReX-Codes/amrex.git (18d0a2861d31c52c65752a1d5856f54e08699ce3)
-- CMake version: 3.20.5
-- AMReX installation directory: /lustre/rz/dbertini/gpu/warpx/warpx_install
-- Build type set by user to 'Release'.
-- Building AMReX with AMReX_SPACEDIM = 2
-- Configuring AMReX with the following options enabled: 
--    AMReX_PRECISION = DOUBLE
--    AMReX_GPU_BACKEND = HIP
--    AMReX_MPI
--    AMReX_MPI_THREAD_MULTIPLE
--    AMReX_LINEAR_SOLVERS
--    AMReX_PARTICLES
--    AMReX_PARTICLES_PRECISION = DOUBLE
--    AMReX_TINY_PROFILE
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1") 
-- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS) 
CMake Error at /usr/local/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_CXX_FOUND CXX) (found version "3.1")
Call Stack (most recent call first):
  /usr/local/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/share/cmake-3.20/Modules/FindMPI.cmake:1742 (find_package_handle_standard_args)
  /lustre/rz/dbertini/gpu/warpx/warpx_build/build/_deps/fetchedamrex-src/Tools/CMake/AMReXParallelBackends.cmake:21 (find_package)
  /lustre/rz/dbertini/gpu/warpx/warpx_build/build/_deps/fetchedamrex-src/Src/CMakeLists.txt:14 (include)

Any idea why the MPI_CXX compiler is claimed to be not found ?

[ax3l]

Hi @denisbertini,

Thanks for reaching out.

Any idea why the MPI_CXX compiler is claimed to be not found ?

This error means you do not have MPI installed or fully configured. Your output above has a few more lines, which hint to a CMakeError.log file that would describe why the compiler does not work. My guess it's the current configuration options and compiler settings that you pass that cause a mini-MPI example compilation to fail during configure.

You can either:

• run without MPI on a single node: pass -DWarpX_MPI=OFF to the CMake configuration line for this

or you provide and hint the location of MPI on the machine.
For the latter case, can you please share details on the machine? Is this a desktop (in that case you don't need MPI) or an HPC system?
Which HPC system:

• system name
• operating system and
• modules loaded?

On which instructions did you base your current CMake line? Because we compile in a simpler way on HIP systems with regards to compilers and MPI hints and usually don't need to use the hipcc compiler wrapper:

• Cray Example: https://warpx.readthedocs.io/en/latest/install/hpc/crusher.html
• Ubuntu Example: https://github.com/ECP-WarpX/WarpX/blob/ee1548f32e406512076825daded6057d279abd4d/.github/workflows/hip.yml#L41-L59

[denisbertini]

@ax3l
I am building on HPC on an environment very similar than SPOCK.
We use singularity containers to build which contain all the dependencies to build warpx including MPI and ROCm4.3.
In fact i use this container to run simulation based on PiconGPU code.
I used the following warpx.profile:

#
# Environment #################################################################
#

# Compilers
export CC=/opt/rocm/llvm/bin/clang
export CXX=/opt/rocm/llvm/bin/clang++

# Main environment variables
export ADIOS2_DIR=/opt/adios/2.7.1
export openPMD_DIR=/opt/openPMD-api/0.14.3/

## add hip+adios
export LD_LIBRARY_PATH=/opt/rocm/hiprand/lib:/opt/rocm/rocrand/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$ADIOS2_DIR/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$openPMD_DIR/lib:$LD_LIBRARY_PATH

export PATH=/opt/rocm/llvm/bin:$PATH
export PATH=/opt/rocm/hiprand/bin:$PATH
export PATH=$ADIOS2_DIR/bin:$PATH

# output data
export SCRATCH=/lustre/rz/dbertini/gpu/data
export WORKDIR=/lustre/rz/dbertini/gpu

export OMPI_MCA_io=^ompio
export OMPI_MCA_mpi_leave_pinned=0

export AMRex_AMD_ARCH=gfx908

trying to build gives me such errors:

Singularity> cmake -S . -B build -DWarpX_DIMS=3 -DWarpX_COMPUTE=HIP -DAMReX_AMD_ARCH=gfx908 -DMPI_CXX_COMPILER=$(which CXX) -DMPI_C_COMPILER=$(which CC) 
-- The C compiler identification is Clang 13.0.0
-- The CXX compiler identification is Clang 13.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rocm/llvm/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rocm/llvm/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find CCache
-- Downloading AMReX ...
-- AMReX repository: https://github.com/AMReX-Codes/amrex.git (18d0a2861d31c52c65752a1d5856f54e08699ce3)
-- CMake version: 3.20.5
-- AMReX installation directory: /usr/local
-- Build type set by user to 'Release'.
-- Building AMReX with AMReX_SPACEDIM = 3
-- Configuring AMReX with the following options enabled: 
--    AMReX_PRECISION = DOUBLE
--    AMReX_GPU_BACKEND = HIP
--    AMReX_MPI
--    AMReX_MPI_THREAD_MULTIPLE
--    AMReX_LINEAR_SOLVERS
--    AMReX_PARTICLES
--    AMReX_PARTICLES_PRECISION = DOUBLE
--    AMReX_TINY_PROFILE
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS) 
-- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS) 
CMake Error at /usr/local/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND C CXX)
Call Stack (most recent call first):
  /usr/local/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/share/cmake-3.20/Modules/FindMPI.cmake:1742 (find_package_handle_standard_args)
  build/_deps/fetchedamrex-src/Tools/CMake/AMReXParallelBackends.cmake:21 (find_package)
  build/_deps/fetchedamrex-src/Src/CMakeLists.txt:14 (include)


-- Configuring incomplete, errors occurred!
See also "/lustre/rz/dbertini/gpu/warpx/warpx_src/build/CMakeFiles/CMakeOutput.log".
See also "/lustre/rz/dbertini/gpu/warpx/warpx_src/build/CMakeFiles/CMakeError.log".

MPI is not found at all during the build, but MPI is installed as system library within my singularity container.
Any idea what could go wrong here?