Add cuDSS support via mfem::CuDSSSolver by Pennycook · Pull Request #717 · awslabs/palace

Pennycook · 2026-04-29T09:44:26Z

cuDSS is a high-performance CUDA library for Direct Sparse Solvers.

This commit enables the use of cuDSS in Palace by:

Wrapping mfem::CuDSSSolver in a new palace::CuDSSSolver;
Enabling the use of CuDSSSolver for KSP and WavePort; and
Adding cuDSS configuration options to CMake, for both Palace and MFEM.

I've opened this as a draft for now because it depends on mfem/mfem#5124, which has not yet been merged.

An example of building with cuDSS:

cmake .. -DPALACE_WITH_CUDA=ON -DPALACE_WITH_CUDSS=ON -DCUDSS_DIR="/path/to/cudss/"

Pennycook · 2026-04-30T07:43:58Z

I fixed the formatting, but I'm not sure what to do about the other failures.

The license check is failing because the test assumes the header will match exactly, but I added a new copyright line to two files. One possible solution would be to rewrite the test to check for the copyright notice and SPDX license identifier separately, but I think that's outside the scope of this PR.

I think the other tests are failing because they're using the Spack route of installation, which I didn't update. It's not obvious to me that there's a way to make this work, besides waiting for the mfem Spack package to expose cuDSS support.

hughcars

We'll take this for a spin, one small comment on the patching process but generally looks good.

hughcars

Hi @Pennycook,

I took a stab at cleaning things up as mentioned in the comments and there's a draft PR onto your fork at Pennycook#1. Most of that is noise from merging origin/main into your branch. If you merge into your fork it'll reduce down to a much smaller diff. The main thing was assembling the diffs.

cuDSS is a high-performance CUDA library for Direct Sparse Solvers. This commit enables the use of cuDSS in Palace by: - Wrapping mfem::CuDSSSolver in a new palace::CuDSSSolver; - Enabling the use of CuDSSSolver for KSP and WavePort; and - Adding cuDSS configuration options to CMake, for both Palace and MFEM.

Pennycook · 2026-06-30T12:34:22Z

I took a stab at cleaning things up as mentioned in the comments and there's a draft PR onto your fork at Pennycook#1. Most of that is noise from merging origin/main into your branch. If you merge into your fork it'll reduce down to a much smaller diff. The main thing was assembling the diffs.

Thanks again for this, @hughcars. I rebased my PR on top of origin/main and then cherry-picked these changes.

I'm seeing some runtime failures locally that I didn't see before. Would you mind taking a look to see if the merged code looks as you expected? There were a few changes that didn't show up as merge conflicts (e.g., the JSON processing moved) that I had to fix, and so I might have missed something.

hughcars · 2026-07-01T16:07:30Z

Took cuDSS for a spin on GPU and benchmarked it against SuperLU_DIST. Short version: it's a solid win and the advantage grows with problem size.

Setup

p4d.24xlarge, 8× A100-40GB, -np 8, "Device": "GPU"
Built palace@develop +cuda +cudss via spack (^cuda@13.1, cuDSS 0.7.1, OpenMPI 5.0)
Case: examples/cpw/cpw_lumped_uniform (driven sweep, 7 frequencies/level, Order: 2) with Model.Refinement.MaxIts AMR to grow the mesh; field output disabled to isolate solve time
Only Solver.Linear.Type changed between runs (cuDSS vs SuperLU). Both drove AMR to the same mesh at every level and produced identical S-parameters (~11 digits).
cuDSS/SuperLU run as the geometric-multigrid coarse solver, i.e. they factorize the p1 coarse level; the fine (p2) space is the full problem size.

Results

┌─────────────────────────┬────────────────────────┬──────────────────┬──────────────┬────────────┬───────────────┐
│                         │ p1 coarse (factorized) │ p2 fine (global) │ Coarse Solve │ Total wall │ Peak mem/rank │
├─────────────────────────┼────────────────────────┼──────────────────┼──────────────┼────────────┼───────────────┤
│ cuDSS (MaxIts=2)        │ 58,714                 │ 330,414          │ 16.5 s       │ 162.7 s    │ —             │
├─────────────────────────┼────────────────────────┼──────────────────┼──────────────┼────────────┼───────────────┤
│ SuperLU_DIST (MaxIts=2) │ 58,714                 │ 330,414          │ 48.9 s       │ 191.9 s    │ —             │
├─────────────────────────┼────────────────────────┼──────────────────┼──────────────┼────────────┼───────────────┤
│ cuDSS (MaxIts=5)        │ 869,228                │ 5,254,370        │ 98.3 s       │ 889.7 s    │ 2.8 GB        │
├─────────────────────────┼────────────────────────┼──────────────────┼──────────────┼────────────┼───────────────┤
│ SuperLU_DIST (MaxIts=5) │ 869,228                │ 5,254,370        │ 883.7 s      │ 1648.3 s   │ 4.3 GB        │
└─────────────────────────┴────────────────────────┴──────────────────┴──────────────┴────────────┴───────────────┘

So ~3× faster on the coarse (direct) solve at the smaller size, growing to ~9× at the 869k-DOF coarse grid (5.25M-DOF fine problem), which is ~18% → ~46% off total wall time. cuDSS also appears to use less memory (2.8 vs 4.3 GB/rank), and both MaxIts=5 runs completed with no OOM.

…solvers Follow-up to cuDSS support (#717). Two related build/recipe fixes kept out of the feature PR: - Vendor mfem PR #5389 (extern/patch/mfem/mfem_pr5389_cudss.diff, applied after mfem_pr5124_cudss.diff in both the CMake superbuild and the spack recipe): select the cuDSS communication layer matching the linked MPI so MPICH builds no longer abort at CuDSSSolver construction (cudssSetCommLayer returning CUDSS_STATUS_INVALID_VALUE). Remove once merged upstream and the pinned mfem is bumped. - Spack recipe: stop forcing +cuda on superlu-dist and strumpack. Palace does not use their GPU builds (strumpack.cpp calls DisableGPU(); SuperLU offload is off). Use mfem's conditional cuda_arch-propagation pattern instead, so superlu-dist defaults to its CPU build and strumpack works with whatever the mfem dependency provides. (Fixes the SuperLU device-alloc hang; the separate strumpack+cuda/libCEED crash is tracked in #803.)

hughcars

Took it for a spin, worked great! There's an issue with building with mpich in mfem, but that will fail loudly and I have a patch on its way to mfem, so i think we can merge. I have one final commit on hughcars/pr717-cudss-followup with some small fix ups. If you cherry-pick that, rebase, and maybe bump the SchemaVer again (if another bump snuck in underneath), then we're good to go. Thank you for the submission! This'll be a meaningful improvement in nvidia gpu performance.

hughcars · 2026-07-13T19:38:30Z

+
+#if defined(MFEM_USE_CUDSS)
+
+#include "utils/iodata.hpp"


This file only uses MatrixSymmetry/SymbolicFactorization (defined in utils/labels.hpp) and nothing from iodata.hpp — the feedback commit switches this include to utils/labels.hpp, the actual definer, for include-what-you-use.

hughcars · 2026-07-13T19:38:30Z

+
+}  // namespace
+
+CuDSSSolver::CuDSSSolver(MPI_Comm comm, MatrixSymmetry sym, SymbolicFactorization reorder,


cuDSS exposes only reordering reuse, not a strategy selection, so the SymbolicFactorization reorder arg is unused — the feedback commit marks it /*reorder*/ with a one-line note, kept only for parity with the other direct-solver wrappers.

hughcars · 2026-07-13T19:38:30Z

+namespace palace
+{
+
+class IoData;


Unused IoData forward-decl — the feedback commit removes it.

hughcars · 2026-07-13T19:38:30Z

 #include "linalg/ams.hpp"
 #include "linalg/arpack.hpp"
 #include "linalg/blockprecond.hpp"
+#include "linalg/cudss.hpp"


linalg/cudss.hpp isn't used in this file — the feedback commit removes it.

hughcars · 2026-07-13T19:38:30Z

+      pc = MakeWrapperSolver<OperType, CuDSSSolver>(linear, comm, pc_mat_sym,
+                                                    linear.sym_factorization,
+                                                    linear.reorder_reuse, print);
+#else


The feedback commit drops this #else MFEM_ABORT so the case matches the empty-guarded MUMPS one; the single "not built with cuDSS" check moves to an MFEM_VERIFY in iodata.cpp alongside the SuperLU/STRUMPACK/MUMPS guards.

hughcars · 2026-07-13T19:38:30Z

          "description": "Use the Jacobi preconditioner."
+        },
+        {
+          "const": "cuDSS",


New allowed enum value ⇒ backward-compatible SchemaVer REVISION. main is now at 1-1-0 (RationalImpedance, #770), so the feedback commit bumps the schema $id 1-1-0 → 1-2-0 and tags the CHANGELOG entry SchemaVer 1-2-0 (check-schema-version enforces this). You might need to bump it further if something else gets in first.

hughcars reviewed Apr 30, 2026

View reviewed changes

Comment thread cmake/ExternalGitTags.cmake

hughcars added a commit that referenced this pull request Jun 16, 2026

Merge origin/main into PR #717 branch

5dd4bc4

hughcars added a commit that referenced this pull request Jun 16, 2026

Reapply cuDSS support after merging origin/main into PR #717 branch

ba9c382

hughcars mentioned this pull request Jun 17, 2026

Fixes for PR717 Pennycook/palace#1

Draft

hughcars requested changes Jun 17, 2026

View reviewed changes

hughcars added a commit that referenced this pull request Jun 22, 2026

Fix PR #717 MFEM patch stack and Spack cuDSS support

5038017

hughcars added a commit that referenced this pull request Jun 22, 2026

Fix PR #717 MFEM patch stack and Spack cuDSS support

fe5c897

hughcars added a commit that referenced this pull request Jun 22, 2026

Fix PR #717 MFEM patch stack and Spack cuDSS support

e6f2324

Pennycook and others added 2 commits June 30, 2026 11:12

Fix PR awslabs#717 MFEM patch stack and Spack cuDSS support

ebc2555

Pennycook force-pushed the cudss branch from cabb01f to ebc2555 Compare June 30, 2026 12:31

Pennycook marked this pull request as ready for review July 2, 2026 05:27

hughcars added a commit that referenced this pull request Jul 13, 2026

Fix PR #717 MFEM patch stack and Spack cuDSS support

96c5aca

hughcars reviewed Jul 14, 2026

View reviewed changes


		} // namespace

		CuDSSSolver::CuDSSSolver(MPI_Comm comm, MatrixSymmetry sym, SymbolicFactorization reorder,

Uh oh!

Conversation

Pennycook commented Apr 29, 2026

Uh oh!

Pennycook commented Apr 30, 2026

Uh oh!

hughcars left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hughcars left a comment

Choose a reason for hiding this comment

Uh oh!

Pennycook commented Jun 30, 2026

Uh oh!

hughcars commented Jul 1, 2026

Uh oh!

hughcars left a comment

Choose a reason for hiding this comment

Uh oh!

hughcars Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hughcars Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hughcars Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hughcars Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hughcars Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hughcars Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants