Amesos2: SuperLU_dist segfaulting?

[berselius]

Bug Report

Pinging @srajama1 as the listed Amesos2 package lead.

Description

I am converting our project from Amesos to Amesos2. We have used SuperLU_Dist as a direct solver for quite some time with Amesos as the interface to use SuperLU_dist as a direct solver. After moving to Amesos2, our simple test that uses the SuperLU_Dist solver fails with a segfault. After poking through the changes I made, I didn't see anything obviously wrong with my vector/CRS matrix setup, so I made a small reproducer that linked in trilinos/superlu and saw the same segfault.

We're not necessarily married to SuperLU_dist as a direct solver, but would still like to clear this up to be able to compare to previous results (and other options) as we finish the Tpetra conversion.

Steps to Reproduce

• SHA1: I am using the released trilinos@16.0.0 built via Spack. The full config line in spack was

-DTrilinos_VERBOSE_CONFIGURE:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=ON -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -DTrilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -DTrilinos_ENABLE_CXX11:BOOL=ON -DTrilinos_ENABLE_DEBUG:BOOL=OFF -DTrilinos_ENABLE_EXAMPLES:BOOL=OFF -DTrilinos_ENABLE_SECONDARY_TESTED_CODE:BOOL=ON -DTrilinos_ENABLE_TESTS:BOOL=OFF -DTrilinos_ENABLE_Fortran:BOOL=ON -DTrilinos_ENABLE_OpenMP:BOOL=OFF -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON  -DTrilinos_ENABLE_PyTrilinos2:BOOL=OFF -DTrilinos_ENABLE_TESTS:BOOL=OFF -DCMAKE_CXX_STANDARD:STRING=17 -DTrilinos_ENABLE_Adelus:BOOL=OFF -DTrilinos_ENABLE_Amesos:BOOL=OFF -DTrilinos_ENABLE_Amesos2:BOOL=ON -DTrilinos_ENABLE_Anasazi:BOOL=ON -DTrilinos_ENABLE_AztecOO:BOOL=OFF -DTrilinos_ENABLE_Belos:BOOL=ON -DTrilinos_ENABLE_Epetra:BOOL=OFF -DTrilinos_ENABLE_EpetraExt:BOOL=OFF -DTrilinos_ENABLE_FEI:BOOL=OFF -DTrilinos_ENABLE_Gtest:BOOL=OFF -DTrilinos_ENABLE_Ifpack:BOOL=OFF -DTrilinos_ENABLE_Ifpack2:BOOL=ON -DTrilinos_ENABLE_Intrepid:BOOL=OFF -DTrilinos_ENABLE_Intrepid2:BOOL=OFF -DTrilinos_ENABLE_Isorropia:BOOL=OFF -DTrilinos_ENABLE_Kokkos:BOOL=ON -DTrilinos_ENABLE_MiniTensor:BOOL=OFF -DTrilinos_ENABLE_Mesquite:BOOL=OFF -DTrilinos_ENABLE_ML:BOOL=OFF -DTrilinos_ENABLE_MueLu:BOOL=ON -DTrilinos_ENABLE_NOX:BOOL=ON -DTrilinos_ENABLE_Pamgen:BOOL=OFF -DTrilinos_ENABLE_Panzer:BOOL=OFF -DTrilinos_ENABLE_Pike:BOOL=OFF -DTrilinos_ENABLE_Piro:BOOL=OFF -DTrilinos_ENABLE_Phalanx:BOOL=OFF -DTrilinos_ENABLE_ROL:BOOL=ON -DTrilinos_ENABLE_Rythmos:BOOL=OFF -DTrilinos_ENABLE_Sacado:BOOL=ON -DTrilinos_ENABLE_SCOREC:BOOL=OFF -DTrilinos_ENABLE_Shards:BOOL=OFF -DTrilinos_ENABLE_ShyLU:BOOL=OFF -DTrilinos_ENABLE_STK:BOOL=OFF -DTrilinos_ENABLE_Stokhos:BOOL=OFF -DTrilinos_ENABLE_Stratimikos:BOOL=OFF -DTrilinos_ENABLE_Teko:BOOL=OFF -DTrilinos_ENABLE_Tempus:BOOL=OFF -DTrilinos_ENABLE_Thyra:BOOL=OFF -DTrilinos_ENABLE_Tpetra:BOOL=ON -DTrilinos_ENABLE_TrilinosCouplings:BOOL=OFF -DTrilinos_ENABLE_Triutils:BOOL=ON -DTrilinos_ENABLE_Zoltan:BOOL=ON -DTrilinos_ENABLE_Zoltan2:BOOL=ON -DEpetraExt_BUILD_BTF:BOOL=OFF -DEpetraExt_BUILD_EXPERIMENTAL:BOOL=OFF -DEpetraExt_BUILD_GRAPH_REORDERINGS:BOOL=OFF -DAmesos2_ENABLE_Basker:BOOL=OFF -DAmesos2_ENABLE_LAPACK:BOOL=ON -DAmesos2_ENABLE_MUMPS:BOOL=OFF -DTrilinos_ENABLE_SEACASExodus:BOOL=OFF -DTrilinos_ENABLE_SEACASIoss:BOOL=OFF -DTrilinos_ENABLE_SEACASChaco:BOOL=OFF -DTrilinos_ENABLE_SEACASNemslice:BOOL=OFF -DTPL_ENABLE_ADIOS2:BOOL=OFF -DTPL_ENABLE_Boost:BOOL=OFF -DTPL_ENABLE_CUDA:BOOL=OFF -DTPL_ENABLE_HDF5:BOOL=OFF -DTPL_ENABLE_HYPRE:BOOL=ON -DHYPRE_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/hypre-2.31.0-ii6fywr/include -DHYPRE_ROOT:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/hypre-2.31.0-ii6fywr -DHYPRE_LIBRARY_NAMES:STRING=HYPRE -DHYPRE_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/hypre-2.31.0-ii6fywr/lib -DTPL_ENABLE_MUMPS:BOOL=OFF -DTPL_ENABLE_AMD:BOOL=OFF -DTPL_ENABLE_UMFPACK:BOOL=OFF -DTPL_ENABLE_SuperLU:BOOL=OFF -DTPL_ENABLE_SuperLUDist:BOOL=ON -DSuperLUDist_INCLUDE_DIRS:STRING=/usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/superlu-dist-9.1.0-lgsyg4s/include -DSuperLUDist_ROOT:STRING=/usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/superlu-dist-9.1.0-lgsyg4s -DSuperLUDist_LIBRARY_NAMES:STRING=superlu_dist -DSuperLUDist_LIBRARY_DIRS:STRING=/usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/superlu-dist-9.1.0-lgsyg4s/lib -DTPL_ENABLE_X11:BOOL=OFF -DTPL_ENABLE_STRUMPACK:BOOL=OFF -DTPL_ENABLE_BLAS:BOOL=ON -DBLAS_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/intel-oneapi-mkl-2024.2.2-4cdod2w/mkl/2024.2/include -DBLAS_ROOT:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/intel-oneapi-mkl-2024.2.2-4cdod2w -DBLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;m;dl -DBLAS_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/intel-oneapi-mkl-2024.2.2-4cdod2w/mkl/2024.2/lib;/usr/lib64 -DTPL_ENABLE_CGNS:BOOL=OFF -DTPL_ENABLE_LAPACK:BOOL=ON -DLAPACK_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/intel-oneapi-mkl-2024.2.2-4cdod2w/mkl/2024.2/include -DLAPACK_ROOT:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/intel-oneapi-mkl-2024.2.2-4cdod2w -DLAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;m;dl -DLAPACK_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/intel-oneapi-mkl-2024.2.2-4cdod2w/mkl/2024.2/lib;/usr/lib64 -DTPL_ENABLE_Matio:BOOL=OFF -DTPL_ENABLE_METIS:BOOL=ON -DMETIS_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/metis-5.1.0-pgso6fm/include -DMETIS_ROOT:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/metis-5.1.0-pgso6fm -DMETIS_LIBRARY_NAMES:STRING=metis -DMETIS_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/metis-5.1.0-pgso6fm/lib -DTPL_ENABLE_Netcdf:BOOL=OFF -DTPL_ENABLE_SCALAPACK:BOOL=OFF -DTPL_ENABLE_Zlib:BOOL=ON -DZlib_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/zlib-1.3.1-julpya3/include -DZlib_ROOT:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/zlib-1.3.1-julpya3 -DZlib_LIBRARY_NAMES:STRING=z -DZlib_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/zlib-1.3.1-julpya3/lib -DTPL_ENABLE_Pnetcdf:BOOL=OFF -DTPL_ENABLE_HWLOC:BOOL=ON -DHWLOC_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/hwloc-2.11.1-ujaqbx6/include -DHWLOC_ROOT:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/hwloc-2.11.1-ujaqbx6 -DHWLOC_LIBRARY_NAMES:STRING=hwloc -DHWLOC_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/hwloc-2.11.1-ujaqbx6/lib -DTPL_ENABLE_Kokkos:BOOL=ON -DTPL_ENABLE_KokkosKernels:BOOL=ON -DTPL_ENABLE_MPI:BOOL=ON -DCMAKE_C_COMPILER:STRING=/cpe/24.07/mpich/8.1.30/ofi/gnu/12.3/bin/mpicc -DCMAKE_CXX_COMPILER:STRING=/cpe/24.07/mpich/8.1.30/ofi/gnu/12.3/bin/mpicxx -DCMAKE_Fortran_COMPILER:STRING=/cpe/24.07/mpich/8.1.30/ofi/gnu/12.3/bin/mpif90 -DMPI_BASE_DIR:STRING=/cpe/24.07/mpich/8.1.30/ofi/gnu/12.3 -DTPL_ENABLE_ParMETIS:BOOL=ON -DParMETIS_LIBRARY_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/parmetis-4.0.3-rkcqm6m/lib;/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/metis-5.1.0-pgso6fm/lib -DParMETIS_LIBRARY_NAMES:STRING=parmetis;metis -DTPL_ParMETIS_INCLUDE_DIRS:STRING=/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/parmetis-4.0.3-rkcqm6m/include;/usr/projects/shavano/spack-environments/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/metis-5.1.0-pgso6fm/include -DHAVE_SUPERLUDIST_LUSTRUCTINIT_2ARG:BOOL=ON -DTPL_ENABLE_Cholmod:BOOL=OFF -DTeuchos_ENABLE_COMPLEX:BOOL=OFF -DTeuchos_ENABLE_FLOAT:BOOL=OFF -DTpetra_INST_OPENMP:BOOL=OFF -DTpetra_INST_DOUBLE:BOOL=ON -DTpetra_INST_COMPLEX_DOUBLE:BOOL=OFF -DTpetra_INST_COMPLEX_FLOAT:BOOL=OFF -DTpetra_INST_FLOAT:BOOL=OFF -DTpetra_INST_SERIAL:BOOL=ON -DTpetra_INST_INT_INT:BOOL=OFF -DTpetra_INST_INT_LONG:BOOL=OFF -DTpetra_INST_INT_LONG_LONG:BOOL=ON -DKokkos_ARCH_SKX:BOOL=ON -DKokkos_ENABLE_CUDA:BOOL=OFF -DKokkos_ENABLE_OPENMP:BOOL=OFF -DCMAKE_INSTALL_NAME_DIR:STRING=/usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/trilinos-16.0.0-gshs2gf/lib

• Build log:

spack-build-out.txt.gz

• Reproducer:

CMakeLists.txt

superlu_amesos_small_example copy.txt

• change name of cpp file from .txt to .cc since gitlab wouldn't let me upload it
• set up a build dir, run cmake (with whatever MPI you're using)
• Trilnos_dir etc is already in my CMAKE_PREFIX_PATH from my env
• Trilinos spack config line in the env "should" be

trilinos: variants: ['+amesos2', '~epetra', '+ifpack2', '+kokkos', '+muelu', '+nox', '+rol', '+superlu-dist', '+tpetra', '+zoltan', '+zoltan2', '~amesos', '~aztec', '~epetraext', '~ifpack', '~isorroppia', '~ml', 'cxxstd=17']

gdb backtrace of segfault is

Program received signal SIGSEGV, Segmentation fault.
0x00001555536afa41 in pdgstrs ()
   from /usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/superlu-dist-9.1.0-lgsyg4s/lib/libsuperlu_dist.so.9
Missing separate debuginfos, use: zypper install krb5-debuginfo-1.20.1-150500.3.9.1.x86_64 libbrotlicommon1-debuginfo-1.0.7-3.3.1.x86_64 libbrotlidec1-debuginfo-1.0.7-3.3.1.x86_64 libcom_err2-debuginfo-1.46.4-150400.3.9.2.x86_64 libcurl4-debuginfo-8.0.1-150400.5.59.1.x86_64 libidn2-0-debuginfo-2.2.0-3.6.1.x86_64 libjitterentropy3-debuginfo-3.4.1-150000.1.12.1.x86_64 libjson-c3-debuginfo-0.13-3.3.1.x86_64 libkeyutils1-debuginfo-1.6.3-5.6.1.x86_64 libldap-2_4-2-debuginfo-2.4.46-150200.14.17.1.x86_64 libnghttp2-14-debuginfo-1.40.0-150200.17.1.x86_64 libnl3-200-debuginfo-3.3.0-1.29.x86_64 libopenssl1_1-debuginfo-1.1.1l-150500.17.37.1.x86_64 libpcre1-debuginfo-8.45-150000.20.13.1.x86_64 libpsl5-debuginfo-0.20.1-150000.3.3.1.x86_64 libsasl2-3-debuginfo-2.1.28-150500.1.1.x86_64 libselinux1-debuginfo-3.1-150400.1.69.x86_64 libunistring2-debuginfo-0.9.10-1.1.x86_64 libzstd1-debuginfo-1.5.0-150400.3.3.1.x86_64
(gdb) bt
#0  0x00001555536afa41 in pdgstrs ()
   from /usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/superlu-dist-9.1.0-lgsyg4s/lib/libsuperlu_dist.so.9
#1  0x0000155553950405 in Amesos2::Superludist<Tpetra::CrsMatrix<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >, Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::solve_impl(Teuchos::Ptr<Amesos2::MultiVecAdapter<Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > >, Teuchos::Ptr<Amesos2::MultiVecAdapter<Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > > const>) const ()
   from /usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/trilinos-16.0.0-gshs2gf/lib64/libamesos2.so.16
#2  0x000015555396a576 in Amesos2::SolverCore<Amesos2::Superludist, Tpetra::CrsMatrix<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >, Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::solve(Teuchos::Ptr<Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >, Teuchos::Ptr<Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const>) const ()
   from /usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/trilinos-16.0.0-gshs2gf/lib64/libamesos2.so.16
#3  0x0000155553945bb3 in Amesos2::SolverCore<Amesos2::Superludist, Tpetra::CrsMatrix<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >, Tpetra::MultiVector<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::solve() ()
   from /usr/projects/capsaicin/devs/haack/spack-environments.20250714-tpetra/ATS3G/2.6.0-1/spack/opt/spack/linux-sles15-sapphirerapids/gcc-12.3.0/trilinos-16.0.0-gshs2gf/lib64/libamesos2.so.16
#4  0x000000000041450d in main ()

This is the same location in superlu_dist where the segfault was happening in our main project with amesos2.

[cgcgcg]

@berselius Thank you for taking the time to construct a simple reproducer!

[jhux2]

@ndellingwood @iyamazaki Do either of you have time to investigate?

[iyamazaki]

I ran Amesos2 example and the provided example, and I seem to be seeing the same issue (in pdgstrs) with 9.1, which I don't see with an older version of superlu_dist.

[cgcgcg]

So a SuperLU_dist issue? Maybe this one? xiaoyeli/superlu_dist#180

[jhux2]

So a SuperLU_dist issue? Maybe this one? xiaoyeli/superlu_dist#180

Looks that way. Maybe try superlu_dist 9.2? Issue #180 appears to have been fixed between 9.1 & 9.2.

[iyamazaki]

It seems to work with 9.2. @berselius, are you seeing this issue only with Amesos2? Is Amesos working with SuperLU_DIst 9.1?

[berselius]

No problems with OG Amesos, we've been using it with 9.1 for quite a while.

[berselius]

It looks like 9.2 hasn't made its way into spack yet but it should be easy to hack in for development purposes, I'll take a look next week. Thanks for your help!

[iyamazaki]

I will see if I can spot the difference between our Amesos & Amesos2 interfaces, maybe next week.