Tutorial codes build error with CUDA 12.1

[cityadmirer]

Host compiler: gcc/g++ 12.2.1
OS: Manjaro Linux
GPU: RTX 3060Ti
Driver:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+

Repo:
https://github.com/AMReX-Codes/amrex-tutorials

configuration:

$ cd amrex-tutorials/ExampleCodes
$ mkdir build && cd build
$ cmake -DAMReX_GPU_BACKEND=CUDA ..
$ make

output:

...
[ 62%] Linking CUDA static library libamrex.a
[ 62%] Built target amrex
[ 62%] Building CUDA object Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AdvancePhiAllLevels.cpp.o
[ 63%] Building CUDA object Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AdvancePhiAtLevel.cpp.o
[ 64%] Building CUDA object Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o
nvcc error   : 'cicc' died due to signal 11 (Invalid memory reference)
nvcc error   : 'cicc' core dumped
make[2]: *** [Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/build.make:107: Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o] Error 139
make[1]: *** [CMakeFiles/Makefile2:590: Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[WeiqunZhang]

Unfortunately, I cannot reproduce the issue with cuda 12.0.140, gcc 12.2.0 and cmake 3.25.3 on Ubuntu 20.04. What's the version of your cmake? Could you try to compile with make VERBOSE=ON and show the command used to compile amrex and the failed test?

[cityadmirer]

cmake version 3.25.3
Verbose output

[ 64%] Building CUDA object Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o
cd /home/citya/Downloads/amrex-tutorials/ExampleCodes/build/Amr/Advection_AmrCore && /opt/cuda/bin/nvcc -forward-unknown-to-host-compiler  --options-file CMakeFiles/Advection_AmrCore.dir/includes_CUDA.rsp -O3 -DNDEBUG --generate-code=arch=compute_86,code=[compute_86,sm_86] --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -Xcudafe --display_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info --source-in-ptx -std=c++17 -MD -MT Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o -MF CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o.d -x cu -rdc=true -c /home/citya/Downloads/amrex-tutorials/ExampleCodes/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp -o CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o
nvcc error   : 'cicc' died due to signal 11 (Invalid memory reference)
nvcc error   : 'cicc' core dumped
make[2]: *** [Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/build.make:107: Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o] Error 139
make[2]: Leaving directory '/home/citya/Downloads/amrex-tutorials/ExampleCodes/build'
make[1]: *** [CMakeFiles/Makefile2:590: Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/all] Error 2
make[1]: Leaving directory '/home/citya/Downloads/amrex-tutorials/ExampleCodes/build'
make: *** [Makefile:91: all] Error 2

[WeiqunZhang]

I installed cuda 12.1 and now I can see the error too. This is clear a compiler bug. If you can, try a different version of cuda. We will try to see if we can find a workaround for cuda 12.1.

[ax3l]

I reported this to Nvidia as part of NV bug 4088095. They see a different error as part of the lto-wrapper:

[ 87%] Building CUDA object Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o
cd /localhome/local-yni/yni/4088095/amrex-tutorials/ExampleCodes/build3/Amr/Advection_AmrCore && /usr/local/cuda-12.1/bin/nvcc -forward-unknown-to-host-compiler --options-file CMakeFiles/Advection_AmrCore.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++17 --generate-code=arch=compute_80,code=[compute_80,sm_80] --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -Xcudafe --display_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -flto=auto -ffat-lto-objects -MD -MT Amr/Advection_AmrCore/CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o -MF CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o.d -x cu -rdc=true -c /localhome/local-yni/yni/4088095/amrex-tutorials/ExampleCodes/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp -o CMakeFiles/Advection_AmrCore.dir/Source/AmrCoreAdv.cpp.o
/tmp/ccSvVNzr.s: Assembler messages:
/tmp/ccSvVNzr.s:25687: Error: symbol `fatbinData' is already defined

I personally cannot reproduce an issue yet. Is this only with CUDA 12.0 but not with 12.1?

My setup:

...
-- The CXX compiler identification is GNU 11.3.0
...
-- The CUDA compiler identification is NVIDIA 12.1.105
...
-- Found CUDAToolkit: /usr/local/cuda/include (found version "12.1.105") 
...
-- CMake version: 3.22.1
..
-- Autodetected CUDA architecture(s):  8.6

• Driver Version: 530.30.02
• NVIDIA RTX A2000 8GB Laptop GPU

[yukini2009]

ltoerr.txt

I added the full log for the lto-wrapper failure . It still reproduces for me on same config

-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
...
-- CMake version: 3.26.3
....
-- Found CUDAToolkit: /usr/local/cuda-12.1/include (found version "12.1.105")
-- Autodetected CUDA architecture(s): 8.0
-- AMReX configuration summary:
-- Build type = Release
-- Install directory = /usr/local
-- C++ compiler = /usr/bin/c++
-- CUDA compiler = /usr/local/cuda-12.1/bin/nvcc
-- C++ defines =
-- C++ flags = -O3 -DNDEBUG -fno-lto -flto=auto -ffat-lto-objects
-- CUDA flags = -O3 -DNDEBUG --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -Xcudafe --display_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -flto=auto -ffat-lto-objects

Is there a quick way I can disable the lto both for C++ and CUDA in the cmake configuration ? I only found a 'CMAKE_CXX_FLAGS' and 'CXXFLAGS' can append more flags .

[WeiqunZhang]

Try -DAMReX_CUDA_LTO=OFF.

[yukini2009]

Looks not effective :(

$ cmake -DAMReX_GPU_BACKEND=CUDA -DAMReX_CUDA_LTO=OFF ..
-- The C compiler identification is GNU 11.3.0
-- The CXX compiler identification is GNU 11.3.0
....
-- Found MPI_C: /usr/lib/x86_64-linux-gnu/libmpich.so (found version "4.0") 
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/libmpichcxx.so (found version "4.0") 
-- Found MPI: TRUE (found version "4.0") found components: C CXX 
-- Found CUDAToolkit: /usr/local/cuda-12.1/include (found version "12.1.105") 
-- Autodetected CUDA architecture(s):  8.0
-- AMReX configuration summary: 
--    Build type               = Release
--    Install directory        = /usr/local
--    C++ compiler             = /usr/bin/c++
--    CUDA compiler            = /usr/local/cuda-12.1/bin/nvcc
--    C++ defines              = 
--    C++ flags                = -O3 -DNDEBUG  -flto=auto -ffat-lto-objects -Werror=return-type
--    CUDA flags               = -O3 -DNDEBUG  --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -Xcudafe --display_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -flto=auto -ffat-lto-objects

[WeiqunZhang]

I cannot reproduce what you see in the cmake configuration summary unless I explicitly enable IPO with -DAMReX_IPO=ON. Maybe there is something in your environment?

[yukini2009]

Sorry for the delay . It looks like related to the MPI . If I disable MPI via -DAMReX_MPI=OFF , the LTO flags are disabled and no more lto-wrapper error . (I was previously enabled with mpich by default).
But then , It just interestingly passes well to me .

cmake -DAMReX_GPU_BACKEND=CUDA -DAMReX_MPI=OFF ..

[ 98%] Built target GPU_CNS_Sod
[ 99%] Linking CUDA executable NodalPoisson
[ 99%] Built target GPU_CNS_RT
[ 99%] Built target NodalPoisson
[ 99%] Linking CUDA device code CMakeFiles/Advection_AmrCore.dir/cmake_device_link.o
[100%] Linking CUDA executable Advection_AmrCore
[100%] Built target Advection_AmrCore
local-yni@ipp1-0221:~/yni/4088095/amrex-tutorials/ExampleCodes/build1$ gcc --version
gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

local-yni@ipp1-0221:~/yni/4088095/amrex-tutorials/ExampleCodes/build1$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
local-yni@ipp1-0221:~/yni/4088095/amrex-tutorials/ExampleCodes/build1$ cmake --version
cmake version 3.22.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
local-yni@ipp1-0221:~/yni/4088095/amrex-tutorials/ExampleCodes/build1$ 

[yukini2009]

If I use our HPC MPI wrapper . With MPI enabled , I can also pass the full building . No LTO flag enabled by default .

-- Found Threads: TRUE  
-- Found MPI_C: /opt/nvidia/hpc_sdk/Linux_x86_64/23.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /opt/nvidia/hpc_sdk/Linux_x86_64/23.3/comm_libs/openmpi/openmpi-3.1.5/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components: C CXX 
-- Found CUDAToolkit: /usr/local/cuda-12.1/include (found version "12.1.105") 
-- Autodetected CUDA architecture(s):  8.0
-- AMReX configuration summary: 
--    Build type               = Release
--    Install directory        = /usr/local
--    C++ compiler             = /usr/bin/c++
--    CUDA compiler            = /usr/local/cuda-12.1/bin/nvcc
--    C++ defines              = 
--    C++ flags                = -O3 -DNDEBUG  -Werror=return-type
--    CUDA flags               = -O3 -DNDEBUG  --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=esa_on_defaulted_function_ignored -maxrregcount=255 -Xcudafe --display_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info
....

 99%] Built target NodeTensorLap
[ 99%] Built target ABecLaplacian_C
[ 99%] Linking CUDA executable NodalPoisson
[ 99%] Built target NodalPoisson
[ 99%] Linking CUDA device code CMakeFiles/Advection_AmrCore.dir/cmake_device_link.o
[100%] Linking CUDA executable Advection_AmrCore
[100%] Built target Advection_AmrCore
local-yni@ipp1-0221:~/yni/4088095/amrex-tutorials/ExampleCodes/build2$ 

[BenWibking]

This bug affects our code as well. Is it known if it's fixed in CUDA 12.2?

[BenWibking]

It's still broken in CUDA 12.2.