MiniEM: PanzerMiniEM_Maxwell_MueLu_order1_tpl_MPI_4 test failing on AT2/cuda12 #14569
Description
Bug Report
PanzerMiniEM_Maxwell_MueLu_order1_tpl_MPI_4 is failing in the AT2/cuda12 PR build (this is now a required check to merge PRs)
Description
Relevant part of the output:
2025-10-14T17:53:02.8793225Z Start 1172: PanzerMiniEM_Maxwell_MueLu_order1_tpl_MPI_4
2025-10-14T17:53:02.8798581Z 1172: Test command: /home/runner/spack/opt/spack/linux-x86_64/openmpi-4.1.6-bealcilgsqez2jpdt6qwakomg2fmxjae/bin/mpiexec "--bind-to" "none --mca btl ^smcuda" "-np" "4" "/home/Trilinos/build/packages/panzer/mini-em/example/BlockPrec/PanzerMiniEM_BlockPrec.exe" "--solver=MueLu" "--numTimeSteps=1" "--linAlgebra=Tpetra" "--tpl"
2025-10-14T17:53:02.8800191Z 1172: Working Directory: /home/Trilinos/build/packages/panzer/mini-em/example/BlockPrec
...
2025-10-14T17:53:15.7618430Z 1172: Using iterative solver = "Belos::BlockGmresSolMgr":
2025-10-14T17:53:15.7618727Z 1172: Template parameters:
2025-10-14T17:53:15.7618942Z 1172: ScalarType: double
2025-10-14T17:53:15.7619182Z 1172: MV: Thyra::MultiVectorBase<double>
2025-10-14T17:53:15.7619438Z 1172: OP: Thyra::LinearOpBase<double>
2025-10-14T17:53:15.7619672Z 1172: Flexible: true
2025-10-14T17:53:15.7619863Z 1172: Num Blocks: 10
2025-10-14T17:53:15.7620065Z 1172: Maximum Iterations: 10
2025-10-14T17:53:15.7620284Z 1172: Maximum Restarts: 20
2025-10-14T17:53:15.7620497Z 1172: Convergence Tolerance: 1e-08
2025-10-14T17:53:15.7620794Z 1172: With #Eqns=22320, #RHSs=1 ...
2025-10-14T17:53:15.8222749Z 1172:
2025-10-14T17:53:15.8225714Z 1172: Starting iterations with Belos:
2025-10-14T17:53:15.8229791Z 1172: Using forward operator = Thyra::TpetraLinearOp<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaSpace> >{rangeDim=10800,domainDim=10800}
2025-10-14T17:53:15.8230947Z 1172: Using iterative solver = Belos::BlockCGSolMgr<...,double>{Ortho Type='ICGS', Block Size=1}
2025-10-14T17:53:15.8231341Z 1172: With #Eqns=10800, #RHSs=1 ...
2025-10-14T17:53:15.8353039Z 1172:
2025-10-14T17:53:15.8355395Z 1172: The Belos solver "CG Q_B" of type "Belos::BlockCGSolMgr<...,double>{Ortho Type='ICGS', Block Size=1}" returned a solve status of "SOLVE_STATUS_CONVERGED" in 0 iterations with total CPU time of 0.0128791 sec
2025-10-14T17:53:15.8437342Z 1172:
2025-10-14T17:53:15.8440986Z 1172: Starting iterations with Belos:
2025-10-14T17:53:15.8444807Z 1172: Using forward operator = Thyra::TpetraLinearOp<double, int, long long, Tpetra::KokkosCompat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaSpace> >{rangeDim=11520,domainDim=11520}
2025-10-14T17:53:15.8445732Z 1172: Using iterative solver = Belos::BlockCGSolMgr<...,double>{Ortho Type='ICGS', Block Size=1}
2025-10-14T17:53:15.8446127Z 1172: With #Eqns=11520, #RHSs=1 ...
2025-10-14T17:53:15.9886521Z 1172: terminate called after throwing an instance of 'Thyra::CatastrophicSolveFailure'
2025-10-14T17:53:15.9888295Z 1172: what(): ../../runner/_work/Trilinos/Trilinos/packages/stratimikos/adapters/belos/src/Thyra_BelosLinearOpWithSolve_def.hpp:661:
2025-10-14T17:53:15.9888914Z 1172:
2025-10-14T17:53:15.9889090Z 1172: Throw number = 3
2025-10-14T17:53:15.9889284Z 1172:
2025-10-14T17:53:15.9889475Z 1172: Throw test that evaluated to true: true
2025-10-14T17:53:15.9889723Z 1172:
2025-10-14T17:53:15.9890057Z 1172: ../../runner/_work/Trilinos/Trilinos/packages/belos/src/BelosCGSingleRedIter.hpp:620:
2025-10-14T17:53:15.9890422Z 1172:
2025-10-14T17:53:15.9890621Z 1172: Throw number = 2
2025-10-14T17:53:15.9890798Z 1172:
2025-10-14T17:53:15.9891040Z 1172: Throw test that evaluated to true: SCT::real(alpha) <= zero
2025-10-14T17:53:15.9891315Z 1172:
2025-10-14T17:53:15.9891612Z 1172: Belos::CGSingleRedIter::iterate(): non-positive value for p^H*A*p encountered!
2025-10-14T17:53:15.9892009Z 1172: [trilogpu02:111763] *** Process received signal ***
2025-10-14T17:53:15.9892301Z 1172: [trilogpu02:111763] Signal: Aborted (6)