Summary
Building Palace with strumpack+cuda and running any Device: "GPU" case with Solver.Linear.Type: "STRUMPACK" aborts during multigrid-hierarchy assembly with a CUDA error (CUDA_ERROR_INVALID_HANDLE). The STRUMPACK coarse-solver construction appears to clobber libCEED's active CUDA context / JIT'd module, so the next CUDA operation fails.
This reproduces on a completely fresh build of main and is independent of any other work — CPU STRUMPACK and GPU cuDSS/SuperLU_DIST on the same build are unaffected.
Environment
- Palace
main (commit 9b9d6524d), built via Spack (fully fresh, no reused artifacts):
spack install palace@develop +cuda +strumpack cuda_arch=80
strumpack@8.0.0+cuda → slate+cuda, alongside libceed+cuda (/gpu/cuda/magma).
- 1× NVIDIA A100 (p4d.24xlarge), CUDA 13, GCC 13.3, OpenMPI 5.0.
Minimal reproduction
Take the shipped example examples/cpw/cpw_coax_uniform.json and change exactly two fields in the "Solver" block:
- "Device": "CPU",
+ "Device": "GPU",
...
"Linear": {
- "Type": "Default",
+ "Type": "STRUMPACK",Run:
mpirun -np 2 palace cpw_coax_uniform.json
Observed
libCEED backend: /gpu/cuda/magma
Assembling multigrid hierarchy:
It 1/15: ω/2π = 2.000e+00 GHz ...
MFEM abort: .../backends/cuda/ceed-cuda-compile.cpp:579 in
CeedRunKernelDimSharedCore_Cuda(): CUDA_ERROR_INVALID_HANDLE
signal 6 (Aborted).
Aborts at the first multigrid-hierarchy solve (~2 s). The exact failing CUDA call varies run-to-run — sometimes the libCEED kernel launch above, sometimes cudaMemcpyDeviceToHost (general/cuda.cpp:162) — consistent with a CUDA context/module invalidation rather than a specific kernel bug.
What isolates it
Device: "CPU" + STRUMPACK → works.Device: "GPU" with cuDSS or SuperLU_DIST → works on the same build. Only STRUMPACK triggers it.- The abort happens at coarse-solver construction during hierarchy assembly, before any factorization. Palace already calls
STRUMPACKSolver::DisableGPU() (palace/linalg/strumpack.cpp), so a +cuda-compiled STRUMPACK/SLATE seems to initialize CUDA state (device queue / BLAS handle) at construction regardless of the disable-GPU option, invalidating libCEED's live CUDA context / JIT module.
Workaround
Build strumpack~cuda (CPU STRUMPACK; avoids linking slate+cuda). Palace does not use GPU STRUMPACK anyway (DisableGPU()), so this is functionally equivalent for current usage.
Summary
Building Palace with
strumpack+cudaand running anyDevice: "GPU"case withSolver.Linear.Type: "STRUMPACK"aborts during multigrid-hierarchy assembly with a CUDA error (CUDA_ERROR_INVALID_HANDLE). The STRUMPACK coarse-solver construction appears to clobber libCEED's active CUDA context / JIT'd module, so the next CUDA operation fails.This reproduces on a completely fresh build of
mainand is independent of any other work — CPU STRUMPACK and GPU cuDSS/SuperLU_DIST on the same build are unaffected.Environment
main(commit9b9d6524d), built via Spack (fully fresh, no reused artifacts):strumpack@8.0.0+cuda→slate+cuda, alongsidelibceed+cuda(/gpu/cuda/magma).Minimal reproduction
Take the shipped example
examples/cpw/cpw_coax_uniform.jsonand change exactly two fields in the"Solver"block:Run:
Observed
Aborts at the first multigrid-hierarchy solve (~2 s). The exact failing CUDA call varies run-to-run — sometimes the libCEED kernel launch above, sometimes
cudaMemcpyDeviceToHost(general/cuda.cpp:162) — consistent with a CUDA context/module invalidation rather than a specific kernel bug.What isolates it
Device: "CPU"+ STRUMPACK → works.Device: "GPU"with cuDSS or SuperLU_DIST → works on the same build. Only STRUMPACK triggers it.STRUMPACKSolver::DisableGPU()(palace/linalg/strumpack.cpp), so a+cuda-compiled STRUMPACK/SLATE seems to initialize CUDA state (device queue / BLAS handle) at construction regardless of the disable-GPU option, invalidating libCEED's live CUDA context / JIT module.Workaround
Build
strumpack~cuda(CPU STRUMPACK; avoids linkingslate+cuda). Palace does not use GPU STRUMPACK anyway (DisableGPU()), so this is functionally equivalent for current usage.