Tpetra: Compilation error with Serial backend not enabled - "undefined reference to `Tpetra::Details::FixedHashTable<...>::hasDuplicateKeys()"

[ndellingwood]

Bug Report

@trilinos/tpetra

Description

In OpenMP builds with Tpetra enabled the following undefined reference errors are occurring with FixedHashTable:

FAILED: packages/tpetra/core/test/HashTable/TpetraCore_FixedHashTableTest.exe
: && /projects/x86-64-icelake-rocky8/external-packages/openmpi/4.1.5/gcc/11.3.0/bin/mpicxx -pedantic -Wall -Wno-long-long -Wwrite-strings   -Wno-inline -Wno-deprecated-declarations -g   -O3 -DNDEBUG -DKOKKOS_DEPENDENCE -march=skylake-avx512 -mtune=skylake-avx512 packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/__/__/__/__/teuchos/core/test/UnitTest/Teuchos_StandardUnitTestMain.cpp.o -o packages/tpetra/core/test/HashTable/TpetraCore_FixedHashTableTest.exe  packages/tpetra/core/ext/libtpetraext.a  packages/tpetra/core/inout/libtpetrainout.a  packages/tpetra/core/src/libtpetra.a  packages/tpetra/core/compat/libtpetraclassic.a  packages/tpetra/tsqr/src/libkokkostsqr.a  packages/teuchos/remainder/src/libteuchosremainder.a  packages/teuchos/kokkoscomm/src/libteuchoskokkoscomm.a  packages/teuchos/kokkoscompat/src/libteuchoskokkoscompat.a  packages/kokkos-kernels/libkokkoskernels.a  packages/teuchos/numerics/src/libteuchosnumerics.a  packages/teuchos/comm/src/libteuchoscomm.a  packages/teuchos/parameterlist/src/libteuchosparameterlist.a  packages/teuchos/parser/src/libteuchosparser.a  packages/teuchos/core/src/libteuchoscore.a  packages/kokkos/containers/src/libkokkoscontainers.a  packages/kokkos/algorithms/src/libkokkosalgorithms.a  packages/kokkos/core/src/libkokkoscore.a  -ldl  /projects/x86-64-icelake-rocky8/compilers/gcc/11.3.0/gcc/8.5.0/base/3kidadg/lib64/libgomp.so  packages/kokkos/simd/src/libkokkossimd.a  /projects/x86-64-icelake-rocky8/tpls/openblas/0.3.23/gcc/11.3.0/base/ucufbom/lib/libopenblas.so  /projects/x86-64-icelake-rocky8/tpls/openblas/0.3.23/gcc/11.3.0/base/ucufbom/lib/libopenblas.so && :
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::TestCopyCtor<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >::test(std::ostream&, bool&, Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> >&, Kokkos::View<long long const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, Kokkos::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:770: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::TestCopyCtor<long long, int, Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::test(std::ostream&, bool&, Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >&, Kokkos::View<long long const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> > const&, Kokkos::View<int const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:769: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_T_ContigKeysStartingValue_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:302: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:314: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_K_ContigKeysStartingValue_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:374: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Kokkos::View<long long const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> > const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:386: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_T_NoncontigKeysStartingValue_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:446: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:458: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_K_NoncontigKeysStartingValue_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:519: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Kokkos::View<long long const*, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> > const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:531: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_T_NoncontigKeysAndVals_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:591: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, Teuchos::ArrayView<int const> const&)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:603: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_T_DuplicateKeys_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:667: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:682: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:699: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_T_CopyCtorNoDupKeys_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:841: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:846: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_T_CopyCtorDupKeys_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:985: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:990: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: In function `(anonymous namespace)::FixedHashTable_Empty_UnitTest<int, long long, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::runUnitTestImpl(Teuchos::basic_FancyOStream<char, std::char_traits<char> >&, bool&) const':
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:213: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::FixedHashTable(Teuchos::ArrayView<long long const> const&, int)'
/home/ndellin/trilinos/Trilinos-pristine/packages/tpetra/core/test/HashTable/FixedHashTableTest.cpp:219: undefined reference to `Tpetra::Details::FixedHashTable<long long, int, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace> >::hasDuplicateKeys()'
collect2: error: ld returned 1 exit status

Seems to have begun after #14427

Steps to Reproduce

1. SHA1: e721625
2. Configure script: Blake testbed, "all" queue

ssh blake
salloc -N 1 -p all

module purge
module load cmake gcc/11.3.0 openmpi/4.1.5 openblas/0.3.23 ninja

export TRILINOS_DIR=<path-to-source>

cmake \
  -B build \
  -G Ninja \
  -D CMAKE_INSTALL_PREFIX="${PWD}/install" \
  -D CMAKE_CXX_COMPILER="`which mpicxx`" \
  -D CMAKE_C_COMPILER="`which mpicc`" \
  -D CMAKE_Fortran_COMPILER="`which mpifort`" \
  -D CMAKE_CXX_FLAGS="-g" \
  -D CMAKE_C_FLAGS="-g" \
  -D CMAKE_CXX_STANDARD=20 \
  -DTPL_ENABLE_MPI=ON \
  -DTPL_ENABLE_BLAS:BOOL=ON \
   -DBLAS_LIBRARY_DIRS:FILEPATH=${OPENBLAS_ROOT}/lib \
   -DBLAS_LIBRARY_NAMES:STRING="openblas" \
  -DTPL_ENABLE_LAPACK:BOOL=ON \
   -DLAPACK_INCLUDE_DIRS:FILEPATH=${OPENBLAS_ROOT}/include \
   -DLAPACK_LIBRARY_DIRS:FILEPATH=${OPENBLAS_ROOT}/lib \
   -DLAPACK_LIBRARY_NAMES:STRING="openblas" \
  -DTrilinos_ENABLE_ALL_PACKAGES=OFF \
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
  -DTrilinos_ENABLE_TESTS=ON \
  -DTrilinos_MUST_FIND_ALL_TPL_LIBS=TRUE \
  -DTrilinos_ENABLE_COMPLEX=ON \
  -DTrilinos_ENABLE_OpenMP=ON \
  -DTrilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_OPENMP=ON \
   -D Kokkos_ENABLE_TESTS=ON \
  -D Kokkos_ARCH_SKX=ON \
   -D Kokkos_ENABLE_IMPL_VIEW_LEGACY=ON \
  -DTrilinos_ENABLE_KokkosKernels=ON \
   -D KokkosKernels_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Tpetra=ON \
   -D Tpetra_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Sacado=ON \
   -D Sacado_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Stokhos=ON \
   -D Stokhos_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Intrepid2=ON \
   -D Intrepid2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Amesos2=ON \
   -D Amesos2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Ifpack2=ON \
   -D Ifpack2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Zoltan2=ON \
   -D Zoltan2_ENABLE_TESTS=ON \
\
  -DTPL_ENABLE_Matio=OFF \
$TRILINOS_DIR

cmake --build build --parallel 32

[ndellingwood]

Extra snip from config output in case useful:

Processing enabled top-level package: Kokkos (Libs, Tests, Examples)
-- Setting default Kokkos CXX standard to 20
-- Kokkos version: 4.7.0
-- The project name is: Kokkos
-- Using internal gtest for testing
-- Using -std=c++20 for C++20 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: NoTypeDefined
--     Host Parallel: Kokkos::OpenMP
--       Host Serial: SERIAL
--
-- Architectures:
--  SKX
-- Found TPLLIBDL: /usr/include
-- Found OpenMP_CXX: -fopenmp (found suitable version "4.5", minimum required is "3.0")
-- Found OpenMP: TRUE (found suitable version "4.5", minimum required is "3.0") found components: CXX
-- Using internal desul_atomics copy
-- Performing Test KOKKOS_LINK_OPTIONS_CHECK
-- Performing Test KOKKOS_LINK_OPTIONS_CHECK - Success
-- Performing Test KOKKOS_COMPILE_OPTIONS_CHECK
-- Performing Test KOKKOS_COMPILE_OPTIONS_CHECK - Success
-- Experimental mdspan support is enabled
-- Using internal mdspan directory /home/ndellin/trilinos/Trilinos-pristine/packages/kokkos/tpls/mdspan/include
-- Found Python3: /usr/bin/python3.11 (found version "3.11.5") found components: Interpreter
-- Sources TestOpenMP.cpp
-- Kokkos Backends: OPENMP;SERIAL
...
Processing enabled top-level package: Tpetra (TSQR, Core, Tests, Examples)
-- Tpetra: Using internal Kokkos
-- Tpetra: Enabling deprecated code
-- NOTE: Kokkos::Serial is ON (the CMake option Kokkos_ENABLE_SERIAL is ON), but the corresponding Tpetra Node type is disabled.  If you want to enable instantiation and use of Kokkos::Serial in Tpetra, please also set the CMake option Tpetra_INST_SERIAL:BOOL=ON.  If you use the Kokkos::Serial Node type in Tpetra without doing this, you will get link errors!
-- Determine whether Tpetra will assume that MPI is GPU aware:
--   - Tpetra_INST_CUDA, Tpetra_INST_HIP and Tpetra_INST_SYCL are OFF, so Tpetra will assume that MPI is not GPU aware.
-- Tpetra: defaulting to Tpetra_ASSUME_GPU_AWARE_MPI: OFF
-- Tpetra execution space availability (ON means available):
--   - Serial:  OFF
--   - Threads: OFF
--   - OpenMP:  ON
--   - Cuda:    OFF
--   - HIP:     OFF
--   - SYCL:    OFF
-- Tpetra: Tpetra_INST_INT_LONG_LONG is enabled by default.
-- Tpetra: Tpetra_INST_INT_UNSIGNED is disabled by default.
-- Tpetra: Tpetra_INST_INT_UNSIGNED_LONG is disabled by default.
-- Tpetra: Tpetra_INST_INT_INT is disabled by default.
-- Tpetra: Tpetra_INST_INT_LONG is disabled by default.
--
-- Tpetra: Validate global ordinal setting ...
-- Tpetra: global ordinal setting is OK
--
-- Setting default Node to Tpetra::KokkosCompat::KokkosOpenMPWrapperNode.
-- Tpetra: Using Full ETI
-- Tpetra: Performance-CGSolve test ENABLED
-- TpetraCore_TpetraUtils_UnitTests: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- TpetraCore_TpetraUtils_merge: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- TpetraCore_createPrefix_no_MPI: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!

[jhux2]

@csiefer2

[ndellingwood]

fwiw I retested and explicitly set -D Tpetra_INST_SERIAL=ON as part of the configuration and the build completes without issue. I'm not sure how to translate that to a fix, I tried hacking around with this code block to explicitly add template instantiation for the Serial backend when both SERIAL and OPENMP are enabled (my guess was the Serial-or-OpenMP branch defers to the default execution space when both are present?) but I still had the undefined references

[ndellingwood]

I'm seeing this issues with other builds as well.
Retesting at 2d6e90d , which includes the #14495 fixes, I'm still seeing issues for example in a Cuda/12.6 build:

FAILED: packages/tpetra/core/test/HashTable/TpetraCore_FixedHashTableTest.exe
: && /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/openmpi-4.1.6-d324p7nf3cta4slauvgaaed3iska4une/bin/mpicxx -pedantic -Wall -Wno-long-long -Wwrite-strings   -Wno-inline -Wno-deprecated-declarations  -O3 -DNDEBUG -DKOKKOS_DEPENDENCE -arch=sm_70 packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/__/__/__/__/teuchos/core/test/UnitTest/Teuchos_StandardUnitTestMain.cpp.o -o packages/tpetra/core/test/HashTable/TpetraCore_FixedHashTableTest.exe -L/usr/lib/gcc/x86_64-redhat-linux/11 -Wl,-rpath,/local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/openblas-0.3.27-7yscom52bradcrzxaaaclxoc6wxicn2a/lib  packages/tpetra/core/ext/libtpetraext.a  packages/tpetra/core/inout/libtpetrainout.a  packages/tpetra/core/src/libtpetra.a  packages/tpetra/core/compat/libtpetraclassic.a  packages/tpetra/tsqr/src/libkokkostsqr.a  packages/teuchos/remainder/src/libteuchosremainder.a  packages/teuchos/kokkoscomm/src/libteuchoskokkoscomm.a  packages/teuchos/kokkoscompat/src/libteuchoskokkoscompat.a  packages/kokkos-kernels/libkokkoskernels.a  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libcusolver.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libcusparse.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libnvJitLink.so  packages/teuchos/numerics/src/libteuchosnumerics.a  packages/teuchos/comm/src/libteuchoscomm.a  packages/teuchos/parameterlist/src/libteuchosparameterlist.a  packages/teuchos/parser/src/libteuchosparser.a  packages/teuchos/core/src/libteuchoscore.a  packages/kokkos/containers/src/libkokkoscontainers.a  packages/kokkos/algorithms/src/libkokkosalgorithms.a  packages/kokkos/core/src/libkokkoscore.a  packages/kokkos/simd/src/libkokkossimd.a  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/openblas-0.3.27-7yscom52bradcrzxaaaclxoc6wxicn2a/lib/libopenblas.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/openblas-0.3.27-7yscom52bradcrzxaaaclxoc6wxicn2a/lib/libopenblas.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libcufft.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libcublas.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libcublasLt.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libculibos.a  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/lib64/libcudart.so  /local-projects/spack/opt/spack/linux-rhel9-skylake_avx512/gcc-11.4.1/cuda-12.6.1-p3irw4cfnueq7oiefmxv4urqrikdczem/targets/x86_64-linux/lib/stubs/libcuda.so  -ldl  /usr/lib64/librt.a && :
/usr/bin/ld: packages/tpetra/core/test/HashTable/CMakeFiles/TpetraCore_FixedHashTableTest.dir/FixedHashTableTest.cpp.o: in function `(anonymous namespace)::InitExecSpace<Kokkos::Cuda>::InitExecSpace()':
tmpxft_0000f38a_00000000-6_FixedHashTableTest.cudafe1.cpp:(.text._ZN61_GLOBAL__N__a1f913eb_22_FixedHashTableTest_cpp_24f11f15_6241813InitExecSpaceIN6Kokkos4CudaEEC2Ev[_ZN61_GLOBAL__N__a1f913eb_22_FixedHashTableTest_cpp_24f11f15_6241813InitExecSpaceIN6Kokkos4CudaEEC5Ev]+0x15): undefined reference to `(anonymous namespace)::InitExecSpace<Kokkos::Serial>::registeredExitHook_'
/usr/bin/ld: tmpxft_0000f38a_00000000-6_FixedHashTableTest.cudafe1.cpp:(.text._ZN61_GLOBAL__N__a1f913eb_22_FixedHashTableTest_cpp_24f11f15_6241813InitExecSpaceIN6Kokkos4CudaEEC2Ev[_ZN61_GLOBAL__N__a1f913eb_22_FixedHashTableTest_cpp_24f11f15_6241813InitExecSpaceIN6Kokkos4CudaEEC5Ev]+0x254): undefined reference to `(anonymous namespace)::InitExecSpace<Kokkos::Serial>::registeredExitHook_'
collect2: error: ld returned 1 exit status

Here is the reproducer configuration (tested on kokkos-dev-2) for that build:

module load cmake gcc/11.4.0 cuda/12.6.1-gcc-11.4.1-p3irw4c openmpi/4.1.6 openblas/0.3.27 hwloc/2.11.1-gcc-11.4.1-64pe434 ninja/1.12.1

export TRILINOS_DIR=<path-to-source>
export KOKKOS_DIR=$TRILINOS_DIR/packages/kokkos
export OMPI_CXX=$KOKKOS_DIR/bin/nvcc_wrapper

cmake \
  -B build \
  -GNinja \
  -D CMAKE_INSTALL_PREFIX="${PWD}/install" \
  -D CMAKE_CXX_STANDARD=20 \
  -D CMAKE_BUILD_TYPE:STRING=RELEASE \
  -D BUILD_SHARED_LIBS:BOOL=OFF \
  -DTPL_ENABLE_MPI=ON \
  -DTPL_ENABLE_BLAS:STRING=ON \
   -DBLAS_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
   -DBLAS_LIBRARY_NAMES:STRING="openblas" \
  -DTPL_ENABLE_LAPACK:STRING=ON \
   -DLAPACK_INCLUDE_DIRS:FILEPATH="$OPENBLAS_ROOT/include" \
   -DLAPACK_LIBRARY_DIRS:FILEPATH="$OPENBLAS_ROOT/lib" \
   -DLAPACK_LIBRARY_NAMES:STRING="openblas" \
  -DTPL_ENABLE_CUDA=ON \
  -DTPL_ENABLE_CUSPARSE:BOOL=ON \
  -DTrilinos_ENABLE_ALL_PACKAGES=OFF \
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
  -DTrilinos_ENABLE_TESTS=ON \
  -DTrilinos_MUST_FIND_ALL_TPL_LIBS=TRUE \
  -DTrilinos_ENABLE_COMPLEX=ON \
  -DTrilinos_ENABLE_OpenMP=OFF \
  -DTrilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_SERIAL=ON \
  -D Kokkos_ENABLE_CUDA=ON \
   -D Kokkos_ENABLE_TESTS=ON \
  -D Kokkos_ARCH_VOLTA70=ON \
  -D Kokkos_ENABLE_DEPRECATED_CODE_4=ON \
   -D Kokkos_ENABLE_IMPL_VIEW_LEGACY=ON \
  -DTrilinos_ENABLE_KokkosKernels=ON \
   -D KokkosKernels_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Tpetra=ON \
   -D Tpetra_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Sacado=ON \
   -D Sacado_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Stokhos=ON \
   -D Stokhos_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Adelus=ON \
   -D Adelus_ENABLE_TESTS=OFF \
  -DTrilinos_ENABLE_Compadre=ON \
   -D Compadre_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Amesos2=ON \
   -D Amesos2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Zoltan2=ON \
   -D Zoltan2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Ifpack2=ON \
   -D Ifpack2_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Belos=ON \
   -D Belos_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Phalanx=ON \
   -D Phalanx_ENABLE_TESTS=ON \
  -DTrilinos_ENABLE_Intrepid2=ON \
   -D Intrepid2_ENABLE_TESTS=OFF \
\
  -DTPL_ENABLE_Matio=OFF \
$TRILINOS_DIR

cmake --build build --parallel 20

Edit: adding configure output snips

Processing enabled top-level package: Kokkos (Libs, Tests, Examples)
-- Setting default Kokkos CXX standard to 20
-- Kokkos version: 4.7.0
-- The project name is: Kokkos
-- Using internal gtest for testing
-- Compiler Version: 12.6.68
-- Using -std=c++20 for C++20 standard as feature
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::Cuda
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
--
-- Architectures:
--  VOLTA70
-- Using internal desul_atomics copy
-- Experimental mdspan support is enabled
-- Using internal mdspan directory /ascldap/users/ndellin/Trilinos/packages/kokkos/tpls/mdspan/include
-- Found Python3: /usr/bin/python3.9 (found version "3.9.21") found components: Interpreter
-- Sources TestCuda.cpp
-- Kokkos Backends: SERIAL;CUDA

...
Processing enabled top-level package: Tpetra (TSQR, Core, Tests, Examples)
-- Tpetra: Using internal Kokkos
-- Tpetra: Enabling deprecated code
-- NOTE: Kokkos::Serial is ON (the CMake option Kokkos_ENABLE_SERIAL is ON), but the corresponding Tpetra Node type is disabled.  If you want to enable instantiation and use of Kokkos::Serial in Tpetra, please also set the CMake option Tpetra_INST_SERIAL:BOOL=ON.  If you use the Kokkos::Serial Node type in Tpetra without doing this, you will get link errors!
-- Determine whether Tpetra will assume that MPI is GPU aware:
--   - You explicitly set the CMake variable Tpetra_ASSUME_GPU_AWARE_MPI=ON.  This means that you assume that the MPI implementation you want to use with Trilinos is GPU aware.  If this is NOT true, then solvers will segfault or otherwise fail.
-- Tpetra: defaulting to Tpetra_ASSUME_GPU_AWARE_MPI: ON
-- Tpetra execution space availability (ON means available):
--   - Serial:  OFF
--   - Threads: OFF
--   - OpenMP:  OFF
--   - Cuda:    ON
--   - HIP:     OFF
--   - SYCL:    OFF
-- Tpetra: Tpetra_INST_INT_LONG_LONG is enabled by default.
-- Tpetra: Tpetra_INST_INT_UNSIGNED is disabled by default.
-- Tpetra: Tpetra_INST_INT_UNSIGNED_LONG is disabled by default.
-- Tpetra: Tpetra_INST_INT_INT is disabled by default.
-- Tpetra: Tpetra_INST_INT_LONG is disabled by default.
--
-- Tpetra: Validate global ordinal setting ...
-- Tpetra: global ordinal setting is OK
--
-- User-set default Node: Tpetra::KokkosCompat::KokkosCudaWrapperNode
-- Tpetra: Using Full ETI
-- Tpetra: Performance-CGSolve test ENABLED
-- Tpetra: Performance-CGSolve CUDA_LAUNCH_BLOCKING test ENABLED
-- TpetraCore_TpetraUtils_UnitTests: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- TpetraCore_TpetraUtils_merge: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- TpetraCore_createPrefix_no_MPI: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!

[ndellingwood]

The configuration above has similar setup to the original post, where -D Tpetra_INST_SERIAL=ON is not explicitly set

Edit: Expanding on this comment, explicitly setting -D Tpetra_INST_SERIAL=ON bypasses the issue