Forward Compatibility CUDA bug with CMake 3.17+

[cameronrutherford]

In HiOp, we link against CUDA in the following fashion:

find_package(CUDAToolkit REQUIRED)

if(HIOP_BUILD_SHARED)
  target_link_libraries(hiop_cuda INTERFACE
    CUDA::cusolver
    CUDA::cusparse
    CUDA::cudart
    CUDA::cublasLt
    )
endif()
if(HIOP_BUILD_STATIC)
  target_link_libraries(hiop_cuda INTERFACE 
    CUDA::cusolver_static
    CUDA::cusparse_static
    CUDA::cudart_static
    CUDA::cublasLt_static
    )
endif()

However, when building camp@2022.03.0 with RAJA@2022.03.0 and Umpire@2022.03.1, the camp CMake exported configuration ends up creating the camp target as follows (in campTargets.cmake):

# Create imported target camp
add_library(camp STATIC IMPORTED)

set_target_properties(camp PROPERTIES
  INTERFACE_COMPILE_FEATURES "cxx_std_14"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "cuda_runtime"
)

Per CMake documentation https://cmake.org/cmake/help/latest/module/FindCUDAToolkit.html#module:FindCUDAToolkit, this breaks forward compatibility for including and linking CUDA libraries.

When building and running with this configuration of camp, RAJA and Umpire, I end up getting the following adding to my link line:

.../libumpire.a .../libRAJA.so .../libcamp.a -lcuda_runtime /share/apps/cuda/11.4/lib64/libcudart_static.a /usr/lib64/librt.so

Which generates an ld error:

/usr/bin/ld: cannot find -lcuda_runtime

Simply commenting out the line INTERFACE_LINK_LIBRARIES "cuda_runtime" in campTargets.cmake temporarily resolves the issue, but this should probably be updated for forward compatibility.

@davidbeckingsale @pelesh

[trws]

Do you not see this issue with RAJA or Umpire otherwise? That dependency comes through BLT, so I'm trying to track down how best to solve this.

Also, where in that documentation does it mention forward compatibility or this problem? Mislink?

[cameronrutherford]

https://cmake.org/cmake/help/latest/module/FindCUDA.html - this is the link to the old way of finding cuda that is deprecated in favor of the link that I sent through. The old targets here are called CUDA_cudart_static_LIBRARY, while the newer targets configured through find_package(CUDAToolkit) are instead CUDA::cudart and CUDA::cudart_static. I still am not sure why/when linking against cuda_runtime worked, but I vaguely remember having that in HiOp initially.

I didn't see this issue with RAJA 0.14.0 and Umpire 6.0.0, and I suspect that it is related to some of the C++ 14 changes that were made. That is largely a hunch though. Changing back to those versions with C++11 lets everything build just fine.

In terms of where this dependency comes from, I narrowed it down to this section of camp CMake https://github.com/LLNL/camp/blob/main/CMakeLists.txt#L69:

set (camp_runtime_backends cuda hip)

foreach (backend ${camp_backends})
  string(TOUPPER "${backend}" suffix)
  if ("${ENABLE_${suffix}}")
    set ("CAMP_ENABLE_${suffix}" On)
  endif()
  if (${CAMP_ENABLE_${suffix}})
    if (backend IN_LIST camp_runtime_backends)
      set (backend ${backend}_runtime)  
    endif()
    if (TARGET blt::${backend})
      set (backend blt::${backend})
    endif()
    list (APPEND camp_depends ${backend})
  endif()
endforeach()
# ...
blt_add_library (
  NAME camp
  HEADERS ${camp_headers}
  SOURCES ./src/errors.cpp
  DEPENDS_ON ${camp_depends}
  )

Parsing this manually, it seems like the string literal cuda_runtime is appended to the list passed to DEPENDS_ON in blt_add_library, and so I assume that this is where the dependency is being introduced. Perhaps the target blt::cuda_runtime should exist, but isn't being found for some reason?

I was tempted to make this sort of change myself, but I am not as familiar with camp as I would like, but I would imagine something like the following would acheive the same thing? It is a bit difficult (as we have seen in HiOp) to pick the correct static/shared library target to use:

set (camp_runtime_backends cuda hip)

foreach (backend ${camp_backends})
  string(TOUPPER "${backend}" suffix)
  if ("${ENABLE_${suffix}}")
    set ("CAMP_ENABLE_${suffix}" On)
  endif()
  if (${CAMP_ENABLE_${suffix}})
    if (backend IN_LIST camp_runtime_backends)
      if(backend STREQUAL cuda)
        find_package(CUDAToolkit)
        # Linking only static could cause issues?
        set(backend CUDA::cudart_static)
      else()
        set (backend ${backend}_runtime)  
      endif()
    endif()
    if (TARGET blt::${backend})
      set (backend blt::${backend})
    endif()
    list (APPEND camp_depends ${backend})
  endif()
endforeach()
# ...
blt_add_library (
  NAME camp
  HEADERS ${camp_headers}
  SOURCES ./src/errors.cpp
  DEPENDS_ON ${camp_depends}
  )

Hopefully that clarifies.

[trws]

It’s because BLT is porting from `{backend}_runtime` to `{backend}::runtime`, starting with hip. The cuda one isn’t done, and at present `cuda_runtime` is the correct dependence to request the cuda runtime from blt, it actually has nothing to do with the underlying cuda finding implementation other than that BLT currently still supports versions of CMake from before the new one existed, so we’re a bit stuck with it. Docs for BLT here: https://llnl-blt.readthedocs.io/en/develop/tutorial/common_hpc_dependencies.html#cuda Ideally we should update the handling in BLT to make this work better, but we may need a stopgap to get you working again. Would you be alright with something like “CAMP_NO_INTERFACE_RUNTIMES” or similar to remove that type of dependency from the interface? That’s the fastest thing I can think of to work around this problem generically. Sadly the change you suggested would break our BLT integration. That’s actually almost exactly how camp used to work until we became BLT integrated to fix some integration issues with RAJA and Umpire in the last release.

[rhornung67]

@trws FWIW, Chris is working on the BLT updates for CUDA support to align what is done for HIP. He's run into some issues ripping out the object library stuff, which has issues with CUDA. So it's taking longer than we expected.

[trws]

Thanks @rhornung67, I'm glad to hear that's going forward, but thankfully I don't think we're necessarily dependent on that rework to fix this. I can put in the stopgap in camp for now at least, and we may be able to do something in BLT to either select the static runtime or allow the runtime interface deps to not propagate.

[cameronrutherford]

I'm satisfied with whatever is the most practical solution. I think a CMake CAMP_NO_INTERFACE_RUNTIMES could work, but I don't fully understand a lot of the implications of these decisions.

If there is a fix coming through BLT that we are just waiting on, then I think we can wait for that. We are fine using the older RAJA and Umpire versions for the moment, but a temporary stopgap would be fine. I think this would have to be added to the camp spack package in the end, or I can use a specific camp branch for temporary development.

Thanks for the quick updates!

[rhornung67]

@cameronrutherford we'll keep you posted when the BLT updates and related changes in RAJA, Umpire, Camp, etc. are available.

[trws]

@cameronrutherford, in working on some updates to the cmake here and in RAJA (partly spurred by this but also to fix other issues) I think we may already support this by use of BLT_EXPORT_THIRDPARTY=Off in the cmake configure. I'm not 100% sure that will remove the library dependency, but it will remove the target from the export. If that doesn't do it, we'll have to add something to completely prevent camp from linking to the library at all, which is a bit more complicated.

[jaelynlitz]

I've been working with @cameronrutherford on testing out this suggested fix and am still running into some issues.

These are the changes I made to the RAJA spack package to add a toggle for the BLT_EXPORT_THIRDPARTY

--- a/var/spack/repos/builtin/packages/raja/package.py
+++ b/var/spack/repos/builtin/packages/raja/package.py
@@ -52,6 +52,7 @@ class Raja(CachedCMakePackage, CudaPackage, ROCmPackage):
     # TODO: figure out gtest dependency and then set this default True
     # and remove the +tests conflict below.
     variant("tests", default=False, description="Build tests")
+    variant('enable_blt_export', default=False, description='Turn on/off blt third party export')

     depends_on("blt")
     depends_on("blt@0.5.0:", type="build", when="@0.14.1:")
@@ -138,6 +139,9 @@ def initconfig_package_entries(self):
         option_prefix = "RAJA_" if spec.satisfies("@0.14.0:") else ""

         entries.append(cmake_cache_path("BLT_SOURCE_DIR", spec["blt"].prefix))
+
+        entries.append(cmake_cache_option("BLT_EXPORT_THIRDPARTY", "+enable_blt_export" in spec))
+
         if "camp" in self.spec:
             entries.append(cmake_cache_path("camp_DIR", spec["camp"].prefix))
         entries.append(cmake_cache_option("BUILD_SHARED_LIBS", "+shared" in spec))

With the suggested BLT_EXPORT_THIRDPARTY=Off, we run into an OpenMP compiler compatibility error in RAJA. We ran into this error with gcc8.5.0 and gcc10.2.0. Error snippet:

>> 221    /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/config.hpp:243:2: error: #error RAJA configured wit
            h RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     222      243 | #error RAJA configured with RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     223          |  ^~~~~

(full output below the cut)

Details

[+]  g2wnmqi  ^raja@2022.03.0%gcc@10.2.0+cuda~enable_blt_export+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo cuda_arch=70 arch=linux-centos8-power9le
[+]  zxsdyft          ^blt@0.5.1%gcc@10.2.0 arch=linux-centos8-power9le
[+]  kkcgbwm          ^camp@2022.03.0%gcc@10.2.0+cuda~ipo~rocm~tests build_type=RelWithDebInfo cuda_arch=70 arch=linux-centos8-power9le
[+]  ckaaip3              ^cub@1.16.0%gcc@10.2.0 arch=linux-centos8-power9le
[+]  xf4cspj      ^suite-sparse@5.10.1%gcc@10.2.0~cuda~graphblas~openmp+pic~tbb arch=linux-centos8-power9le
[+]  oyjx6bm          ^gmp@6.2.1%gcc@10.2.0 libs=shared,static arch=linux-centos8-power9le
[+]  5musg74              ^autoconf@2.69%gcc@10.2.0 patches=35c4492,7793209,a49dd5b arch=linux-centos8-power9le
[+]  ypvbhca                  ^m4@1.4.19%gcc@10.2.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-centos8-power9le
[+]  ndiphp5                      ^libsigsegv@2.13%gcc@10.2.0 arch=linux-centos8-power9le
[+]  risyxq5              ^automake@1.16.5%gcc@10.2.0 arch=linux-centos8-power9le
[+]  g3p6oh2              ^libtool@2.4.7%gcc@10.2.0 arch=linux-centos8-power9le
[+]  xmffy4h          ^mpfr@4.1.0%gcc@10.2.0 libs=shared,static arch=linux-centos8-power9le
[+]  uztmion              ^autoconf-archive@2022.02.11%gcc@10.2.0 patches=139214f arch=linux-centos8-power9le
[+]  oegakup              ^texinfo@6.5%gcc@10.2.0 patches=12f6edb,1732115 arch=linux-centos8-power9le
[+]  4c6ggfn      ^umpire@2022.03.1%gcc@10.2.0+c+cuda+device_alloc~deviceconst+examples~fortran~ipo~numa~openmp~rocm~shared build_type=RelWithDebInfo cuda_arch=70 tests=none arch=linux-centos8-power9le

...

13 errors found in build log:
     215    cd /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm/src/LinAlg && /share/apps/cuda/11.4/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/share/a
            pps/gcc/10.2.0/bin/g++ -DCAMP_HAVE_CUDA -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/tpl/eigen -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emj
            zcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Interface -I/tmp/litz372/spack-stage/spack-stage-hi
            op-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Optimization -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg -I/tmp/litz372/spack-st
            age/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Utils -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/_Internals -I/qfs/p
            rojects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/magma-2.6.2-r5aqzf7jljbynhqspm5sx6qdcn5pmwjk/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-cen
            tos8-power9le/gcc-10.2.0/suite-sparse-5.10.1-xf4cspjtnyq2ggzqsxtaoszmr63guhhr/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/metis-5.1.0-xwpjsqkotvvc
            brm36amq5jg3jnihwmdm/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/coinhsl-2015.06.23-4g6hpo3tyx6ye247hl25hbgn4ao23zbk/include -isystem=/qfs/project
            s/ops/rh8p9/openmpi/4.1.4/gcc/10.2.0/include -isystem=/share/apps/openmpi/4.1.4/gcc/10.2.0/include -isystem=/share/apps/cuda/11.4/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack
            /linux-centos8-power9le/gcc-10.2.0/camp-2022.03.0-kkcgbwmh7iet3yis7sqipo4ueo7nthum/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/umpire-2022.
            03.1-4c6ggfna4htr6ix5vwhwi5yncxkurp7w/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include -
            isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/cub-1.16.0-ckaaip3lazkpt3jxqyqrgeceu2xlwe42/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack
            /opt/spack/linux-centos8-power9le/gcc-10.2.0/ginkgo-glu_experimental-j5uhuhtwey37ash7ga2i4tt6ygsvrqbv/include --expt-extended-lambda --expt-relaxed-constexpr -O2 -g -DNDEBUG --generate-code=arch=compu
            te_70,code=[compute_70,sm_70] -Xcompiler=-fPIC -Xcompiler -pthread -std=c++14 -MD -MT src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopMatrixRajaDense.cpp.o -MF CMakeFiles/hiopLinAlg.dir/hiopMatrixRajaDense.c
            pp.o.d -x cu -rdc=true -c /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopMatrixRajaDense.cpp -o CMakeFiles/hiopLinAlg.dir/hiopMatrixRajaDen
            se.cpp.o
     216    cd /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm/src/LinAlg && /share/apps/cuda/11.4/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/share/a
            pps/gcc/10.2.0/bin/g++ -DCAMP_HAVE_CUDA -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/tpl/eigen -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emj
            zcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Interface -I/tmp/litz372/spack-stage/spack-stage-hi
            op-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Optimization -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg -I/tmp/litz372/spack-st
            age/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Utils -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/_Internals -I/qfs/p
            rojects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/magma-2.6.2-r5aqzf7jljbynhqspm5sx6qdcn5pmwjk/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-cen
            tos8-power9le/gcc-10.2.0/suite-sparse-5.10.1-xf4cspjtnyq2ggzqsxtaoszmr63guhhr/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/metis-5.1.0-xwpjsqkotvvc
            brm36amq5jg3jnihwmdm/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/coinhsl-2015.06.23-4g6hpo3tyx6ye247hl25hbgn4ao23zbk/include -isystem=/qfs/project
            s/ops/rh8p9/openmpi/4.1.4/gcc/10.2.0/include -isystem=/share/apps/openmpi/4.1.4/gcc/10.2.0/include -isystem=/share/apps/cuda/11.4/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack
            /linux-centos8-power9le/gcc-10.2.0/camp-2022.03.0-kkcgbwmh7iet3yis7sqipo4ueo7nthum/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/umpire-2022.
            03.1-4c6ggfna4htr6ix5vwhwi5yncxkurp7w/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include -
            isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/cub-1.16.0-ckaaip3lazkpt3jxqyqrgeceu2xlwe42/include -isystem=/qfs/projects/exasgd/src/jaelyn-spack/spack
            /opt/spack/linux-centos8-power9le/gcc-10.2.0/ginkgo-glu_experimental-j5uhuhtwey37ash7ga2i4tt6ygsvrqbv/include --expt-extended-lambda --expt-relaxed-constexpr -O2 -g -DNDEBUG --generate-code=arch=compu
            te_70,code=[compute_70,sm_70] -Xcompiler=-fPIC -Xcompiler -pthread -std=c++14 -MD -MT src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopMatrixRajaSparseTriplet.cpp.o -MF CMakeFiles/hiopLinAlg.dir/hiopMatrixRaj
            aSparseTriplet.cpp.o.d -x cu -rdc=true -c /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopMatrixRajaSparseTriplet.cpp -o CMakeFiles/hiopLinA
            lg.dir/hiopMatrixRajaSparseTriplet.cpp.o
     217    cd /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm/src/LinAlg && /qfs/projects/exasgd/src/jaelyn-spack/spack/lib/spack/env/gcc/g++ -DCAMP_HAVE_CU
            DA -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/tpl/eigen -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-b
            uild-emjzcpm -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Interface -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534v
            vk5d5g/spack-src/src/Optimization -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme
            6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Utils -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/_Internals -I/qfs/projects/exasgd/src/jaelyn-spack/spack
            /opt/spack/linux-centos8-power9le/gcc-10.2.0/magma-2.6.2-r5aqzf7jljbynhqspm5sx6qdcn5pmwjk/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/suite-sparse
            -5.10.1-xf4cspjtnyq2ggzqsxtaoszmr63guhhr/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/metis-5.1.0-xwpjsqkotvvcbrm36amq5jg3jnihwmdm/include -I/qfs/p
            rojects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/coinhsl-2015.06.23-4g6hpo3tyx6ye247hl25hbgn4ao23zbk/include -isystem /qfs/projects/ops/rh8p9/openmpi/4.1.4/gcc/10.2.0/
            include -isystem /share/apps/openmpi/4.1.4/gcc/10.2.0/include -isystem /share/apps/cuda/11.4/include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/ca
            mp-2022.03.0-kkcgbwmh7iet3yis7sqipo4ueo7nthum/include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/umpire-2022.03.1-4c6ggfna4htr6ix5vwhwi5yncxkurp7w
            /include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include -isystem /qfs/projects/exasgd/src/jael
            yn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/cub-1.16.0-ckaaip3lazkpt3jxqyqrgeceu2xlwe42/include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc
            -10.2.0/ginkgo-glu_experimental-j5uhuhtwey37ash7ga2i4tt6ygsvrqbv/include -O2 -g -DNDEBUG -fPIC -pthread -fopenmp -std=gnu++14 -MD -MT src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopLinSolverSymDenseMagma.cp
            p.o -MF CMakeFiles/hiopLinAlg.dir/hiopLinSolverSymDenseMagma.cpp.o.d -o CMakeFiles/hiopLinAlg.dir/hiopLinSolverSymDenseMagma.cpp.o -c /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2
            bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopLinSolverSymDenseMagma.cpp
     218    cd /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm/src/LinAlg && /qfs/projects/exasgd/src/jaelyn-spack/spack/lib/spack/env/gcc/g++ -DCAMP_HAVE_CU
            DA -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/tpl/eigen -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-b
            uild-emjzcpm -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Interface -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534v
            vk5d5g/spack-src/src/Optimization -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme
            6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Utils -I/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/_Internals -I/qfs/projects/exasgd/src/jaelyn-spack/spack
            /opt/spack/linux-centos8-power9le/gcc-10.2.0/magma-2.6.2-r5aqzf7jljbynhqspm5sx6qdcn5pmwjk/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/suite-sparse
            -5.10.1-xf4cspjtnyq2ggzqsxtaoszmr63guhhr/include -I/qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/metis-5.1.0-xwpjsqkotvvcbrm36amq5jg3jnihwmdm/include -I/qfs/p
            rojects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/coinhsl-2015.06.23-4g6hpo3tyx6ye247hl25hbgn4ao23zbk/include -isystem /qfs/projects/ops/rh8p9/openmpi/4.1.4/gcc/10.2.0/
            include -isystem /share/apps/openmpi/4.1.4/gcc/10.2.0/include -isystem /share/apps/cuda/11.4/include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/ca
            mp-2022.03.0-kkcgbwmh7iet3yis7sqipo4ueo7nthum/include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/umpire-2022.03.1-4c6ggfna4htr6ix5vwhwi5yncxkurp7w
            /include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include -isystem /qfs/projects/exasgd/src/jael
            yn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/cub-1.16.0-ckaaip3lazkpt3jxqyqrgeceu2xlwe42/include -isystem /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc
            -10.2.0/ginkgo-glu_experimental-j5uhuhtwey37ash7ga2i4tt6ygsvrqbv/include -O2 -g -DNDEBUG -fPIC -pthread -fopenmp -std=gnu++14 -MD -MT src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopLinSolverUMFPACKZ.cpp.o -
            MF CMakeFiles/hiopLinAlg.dir/hiopLinSolverUMFPACKZ.cpp.o.d -o CMakeFiles/hiopLinAlg.dir/hiopLinSolverUMFPACKZ.cpp.o -c /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5
            g/spack-src/src/LinAlg/hiopLinSolverUMFPACKZ.cpp
     219    In file included from /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/RAJA.hpp:29,
     220                     from /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopMatrixRajaDense.cpp:68:
  >> 221    /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/config.hpp:243:2: error: #error RAJA configured wit
            h RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     222      243 | #error RAJA configured with RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     223          |  ^~~~~
     224    In file included from /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/RAJA.hpp:29,
     225                     from /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopVectorIntRaja.cpp:60:
  >> 226    /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/config.hpp:243:2: error: #error RAJA configured wit
            h RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     227      243 | #error RAJA configured with RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     228          |  ^~~~~
     229    In file included from /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/RAJA.hpp:29,
     230                     from /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/Utils/RajaUmpireUtils.cpp:63:
  >> 231    /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/config.hpp:243:2: error: #error RAJA configured wit
            h RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     232      243 | #error RAJA configured with RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     233          |  ^~~~~
     234    In file included from /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/RAJA.hpp:29,
     235                     from /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopVectorRajaPar.cpp:82:
  >> 236    /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/config.hpp:243:2: error: #error RAJA configured wit
            h RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     237      243 | #error RAJA configured with RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     238          |  ^~~~~
     239    In file included from /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/RAJA.hpp:29,
     240                     from /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopMatrixRajaSparseTriplet.cpp:64:
  >> 241    /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/raja-2022.03.0-g2wnmqidgmhejbbrypmm6crxpqpwswox/include/RAJA/config.hpp:243:2: error: #error RAJA configured wit
            h RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     242      243 | #error RAJA configured with RAJA_ENABLE_OPENMP, but OpenMP not supported by current compiler
     243          |  ^~~~~
     244    make[2]: Leaving directory '/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm'
     245    [ 36%] Built target hiopInterface
  >> 246    make[2]: *** [src/Utils/CMakeFiles/hiopUtils.dir/build.make:121: src/Utils/CMakeFiles/hiopUtils.dir/RajaUmpireUtils.cpp.o] Error 1
     247    make[2]: *** Waiting for unfinished jobs....
  >> 248    make[2]: *** [src/LinAlg/CMakeFiles/hiopLinAlg.dir/build.make:373: src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopVectorIntRaja.cpp.o] Error 1
     249    make[2]: *** Waiting for unfinished jobs....
  >> 250    make[2]: *** [src/LinAlg/CMakeFiles/hiopLinAlg.dir/build.make:359: src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopVectorRajaPar.cpp.o] Error 1
  >> 251    make[2]: *** [src/LinAlg/CMakeFiles/hiopLinAlg.dir/build.make:401: src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopMatrixRajaSparseTriplet.cpp.o] Error 1
  >> 252    make[2]: *** [src/LinAlg/CMakeFiles/hiopLinAlg.dir/build.make:387: src/LinAlg/CMakeFiles/hiopLinAlg.dir/hiopMatrixRajaDense.cpp.o] Error 1
     253    /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopLinSolverSparseGinkgo.cpp: In function 'std::shared_ptr<gko::matrix::Csr<double, int> > hiop
            ::{anonymous}::transferTripletToCSR(std::shared_ptr<gko::Executor>, int, hiop::hiopMatrixSparse*, int**, int**)':
     254    /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopLinSolverSparseGinkgo.cpp:175:42: warning: 'using Array = class gko::array<double>' is depre
            cated: please use array [-Wdeprecated-declarations]
     255      175 |     auto val_array = gko::Array<double>::view(exec, nnz_, kVal_);
     256          |                                          ^~~~
     257    In file included from /qfs/projects/exasgd/src/jaelyn-spack/spack/opt/spack/linux-centos8-power9le/gcc-10.2.0/ginkgo-glu_experimental-j5uhuhtwey37ash7ga2i4tt6ygsvrqbv/include/ginkgo/ginkgo.hpp:41,
     258                     from /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/src/LinAlg/hiopLinSolverSparseGinkgo.hpp:55,

     ...

     373          |                                                                          ^~
     374    /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/tpl/eigen/Eigen/src/Core/PlainObjectBase.h(426): warning: calling a __host__ function("Eigen::internal::con
            servative_resize_like_impl< ::Eigen::Matrix<int, (int)-1, (int)1, (int)0, (int)-1, (int)1> ,  ::Eigen::Matrix<int, (int)-1, (int)1, (int)0, (int)-1, (int)1> , (bool)1> ::run( ::Eigen::DenseBase< ::Eig
            en::Matrix<int, (int)-1, (int)1, (int)0, (int)-1, (int)1> >  &, long)") from a __host__ __device__ function("Eigen::PlainObjectBase< ::Eigen::Matrix<int, (int)-1, (int)1, (int)0, (int)-1, (int)1> > ::
            conservativeResize") is not allowed
     375
     376    /tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-src/tpl/eigen/Eigen/src/Core/AssignEvaluator.h(890): warning: calling a __host__ function("void Eigen::internal
            ::Assignment< ::Eigen::SparseMatrix<double, (int)0, int> ,  ::Eigen::SparseSelfAdjointView< ::Eigen::Map< ::Eigen::SparseMatrix<double, (int)1, int> , (int)0,  ::Eigen::Stride<(int)0, (int)0> > , (uns
            igned int)2u> ,  ::Eigen::internal::assign_op<double, double> ,  ::Eigen::internal::SparseSelfAdjoint2Sparse, void> ::run<double, (int)0> ( ::Eigen::SparseMatrix<T1, T2, int>  &, const  ::Eigen::Spars
            eSelfAdjointView< ::Eigen::Map< ::Eigen::SparseMatrix<double, (int)1, int> , (int)0,  ::Eigen::Stride<(int)0, (int)0> > , (unsigned int)2u>  &, const  ::Eigen::internal::assign_op<double, double>  &)"
            ) from a __host__ __device__ function("Eigen::internal::call_assignment_no_alias< ::Eigen::SparseMatrix<double, (int)0, int> ,  ::Eigen::SparseSelfAdjointView< ::Eigen::Map< ::Eigen::SparseMatrix<doub
            le, (int)1, int> , (int)0,  ::Eigen::Stride<(int)0, (int)0> > , (unsigned int)2u> ,  ::Eigen::internal::assign_op<double, double> > ") is not allowed
     377
     378    make[2]: Leaving directory '/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm'
  >> 379    make[1]: *** [CMakeFiles/Makefile2:462: src/Utils/CMakeFiles/hiopUtils.dir/all] Error 2
     380    make[1]: *** Waiting for unfinished jobs....
     381    make[2]: Leaving directory '/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm'
     382    [ 36%] Built target hiopOptimization
     383    make[2]: Leaving directory '/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm'
  >> 384    make[1]: *** [CMakeFiles/Makefile2:436: src/LinAlg/CMakeFiles/hiopLinAlg.dir/all] Error 2
     385    make[1]: Leaving directory '/tmp/litz372/spack-stage/spack-stage-hiop-develop-emjzcpme6d5u4nk2bwfrai534vvk5d5g/spack-build-emjzcpm'
  >> 386    make: *** [Makefile:149: all] Error 2

[cameronrutherford]

Thanks for debugging this @jaelynlitz. We can probably disable OpenMP to avoid this issue for the moment, but I think this is worth debugging.

We tried to debug the OpenMP compatibility with our compiler by looking at docs here, but struggled to interpret what version we should use.

It is also unclear through spack and this build configuration what version of OpenMP we are pulling in in the first place.

[rhornung67]

When CMake is invoked, is ENABLE_OPENMP (no RAJA_ prefix) being set to on?

[jaelynlitz]

@rhornung67 How do I check for that? I don't see it at hiop's cmake level

[rhornung67]

This may be a red herring, but I thought it may spark a thought. It looks like ENABLE_OPENMP must be on for RAJA_ENABLE_OPENMP to be on since RAJA_ENABLE_OPENMP is a CMake dependent option in RAJA. It doesn't make sense to me that gcc would not support OpenMP, or at least define the variable _OPENMP when OpenMP is enabled, which is standard. The logic here indicates that _OPENMP is not defined https://github.com/LLNL/RAJA/blob/develop/include/RAJA/config.hpp.in#L244

[trws]

I'm not sure if it will be helpful or harmful for this issue, but the target export setup and various issues around it have been substantially reworked in the most recent releases of camp and umpire, and on develop in RAJA. It may be worth trying with those, and if you are using spack ensuring that you use the packages for those three from the most recent develop version of spack, to see if the rework helped or not.

For the OpenMP issue, what compiler and version is this with? It's extremely strange that the build would be able to get to that error, since cmake should have prevented it before that point, unless it's happening in a project including RAJA that is building without the OpenMP flags somehow.