Some lotus users running with FFI_USE_CUDA_SUPRASEAL=1 are seeing GPU out-of-memory panics on v1.35.0 (and later). It shows up as a Rust panic out of sppark. We've seen this both in C2 in the sealing pipeline and in generating WindowPoSt.
thread '<unnamed>' panicked at /home/admfc/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/supraseal-c2-0.1.2/src/lib.rs:151:9:
cudaGetLastError()@sppark-0.1.14/sppark/ntt/kernels/gs_mixed_radix_wide.cu:220 failed: "out of memory"
2026-02-25T23:00:55.511 INFO filecoin_proofs::api::seal > seal_commit_phase2:start: SectorId(33283)
2026-02-25T23:00:55.511 INFO filecoin_proofs::api::seal > seal_commit_phase2_circuit_proofs:start: SectorId(33283)
2026-02-25T23:00:55.511 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[68719476736]
2026-02-25T23:00:55.511 INFO filecoin_proofs::caches > found params in memory cache for STACKED[68719476736]
2026-02-25T23:00:55.528 INFO bellperson::groth16::prover::supraseal > Bellperson 0.26.0 with SupraSeal is being used!
2026-02-25T23:02:14.971 INFO bellperson::groth16::prover::supraseal > synthesis time: 79.44350428s
2026-02-25T23:02:14.971 INFO bellperson::groth16::prover::supraseal > starting proof timer
The problem seems to come from sppark, the GPU SNARK library underneath SupraSeal. We use SupraSeal via Bellperson (via the create_random_proof_batch call) for both C2 and WindowPoSt to do Groth16 when compiled with FFI_USE_CUDA_SUPRASEAL=1. When that environment variable is unset, SupraSeal doesn't get involved and neither does sppark. So the immediate workaround is to not compile with FFI_USE_CUDA_SUPRASEAL=1.
Curio users are unlikely to be impacted by this because Curio runs each of these tasks in a separate process, so any accumulation is short-lived.
The specific change looks like this commit which moved how GPU memory gets released. In practice, GPU memory isn't being reliably handed back between proofs, so a long-running lotus-miner or lotus-worker process eventually exhausts the GPU and panics. I'll take deeper technical discussion over to that repo to try and get a fix released.
For us, we can pin sppark and supraseal-c2 back to their pre-regression versions in filecoin-ffi (supraseal-c2 = 0.1.1, sppark = 0.1.11). We lose a few minor perf tweaks and there were some legitimate fixes in there too, but if none of them are biting anyone today it should be safe to wind back.
Some lotus users running with
FFI_USE_CUDA_SUPRASEAL=1are seeing GPU out-of-memory panics on v1.35.0 (and later). It shows up as a Rust panic out of sppark. We've seen this both in C2 in the sealing pipeline and in generating WindowPoSt.The problem seems to come from sppark, the GPU SNARK library underneath SupraSeal. We use SupraSeal via Bellperson (via the
create_random_proof_batchcall) for both C2 and WindowPoSt to do Groth16 when compiled withFFI_USE_CUDA_SUPRASEAL=1. When that environment variable is unset, SupraSeal doesn't get involved and neither does sppark. So the immediate workaround is to not compile withFFI_USE_CUDA_SUPRASEAL=1.Curio users are unlikely to be impacted by this because Curio runs each of these tasks in a separate process, so any accumulation is short-lived.
The specific change looks like this commit which moved how GPU memory gets released. In practice, GPU memory isn't being reliably handed back between proofs, so a long-running lotus-miner or lotus-worker process eventually exhausts the GPU and panics. I'll take deeper technical discussion over to that repo to try and get a fix released.
For us, we can pin sppark and supraseal-c2 back to their pre-regression versions in filecoin-ffi (
supraseal-c2 = 0.1.1,sppark = 0.1.11). We lose a few minor perf tweaks and there were some legitimate fixes in there too, but if none of them are biting anyone today it should be safe to wind back.