Resolve CUDA JIT libraries from pinned cu12 packages, not the host - #63
Merged
duburcqa merged 1 commit intoJul 22, 2026
Merged
Conversation
At first render the wheel JIT-compiles its device code with nvrtc and links the result with nvJitLink. nvrtc's output must not be newer than the nvJitLink that consumes it; when the two come from different CUDA majors the link fails with "ERROR 4 in nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink" (genesis-world #2089, #2133, #2814). nvrtc was previously dlopen'd at runtime, preferring libnvrtc.so.13 over .12, while nvJitLink was linked (and a copy bundled) at cu12. So on a host that also has CUDA 13 present, the cu12 wheel loaded a cu13 nvrtc and fed its PTX to the cu12 nvJitLink -> version mismatch and a crash on the first render. Adopt torch's model instead of dlopen-guessing or bundling: - Link nvrtc dynamically (DT_NEEDED libnvrtc.so.12), like nvJitLink, and bind the loader's entry points to the linked symbols. dlopen ignores DT_RUNPATH, so binding to the versioned soname is what makes the choice deterministic and keeps nvrtc on the same cu12 major as nvJitLink. - Declare nvidia-cuda-nvrtc-cu12 / nvidia-nvjitlink-cu12 (>= 12.8) as dependencies and resolve them from the sibling nvidia/*/lib packages via RPATH ($ORIGIN/../nvidia/...). Both then come from a single pinned cu12 minor and stay mutually consistent, independent of the host toolkit. - Stop bundling libnvJitLink.so.12 in the wheel and exclude both libs from auditwheel so they are not vendored (keeps the wheel within PyPI limits). - Drop the now-redundant nvrtc preload in renderer_gs. The >= 12.8 floor is intentional: nvrtc compiles the CCCL/CUB headers bundled from the 12.8 build toolkit and must target Blackwell (sm_120), so an older runtime is genuinely unsupported and should fail at install time rather than cryptically at first render. Also fix the nvrtcGetErrorString function-pointer signature (it returns const char*, not nvrtcResult).
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
At first render the wheel JIT-compiles its device code with nvrtc and links the result with nvJitLink. nvrtc's output must not be newer than the nvJitLink that consumes it — when the two come from different CUDA majors the link fails with:
Reported downstream in Genesis-Embodied-AI/genesis-world#2089, #2133, #2814.
The cause: nvrtc was
dlopen'd at runtime, preferringlibnvrtc.so.13over.12, while nvJitLink was linked (and a copy bundled) at cu12. So on a host that also has a CUDA 13 runtime present, the cu12-built wheel loaded a cu13 nvrtc and fed its PTX to the cu12 nvJitLink → version mismatch and a crash on the first render.This supersedes #59 (thanks @QuantuMope for surfacing and diagnosing the hardcoded-load issue). #59's reorder fixed the symptom on specific hosts but isn't general — it still depends on which nvrtc happens to be discoverable. This PR makes the choice deterministic.
Fix — adopt torch's model instead of
dlopen-guessing or bundlingDT_NEEDED libnvrtc.so.12), like nvJitLink, and bind the loader's entry points to the linked symbols.dlopenignoresDT_RUNPATH, so binding to the versioned soname is what makes resolution deterministic and keeps nvrtc on the same cu12 major as nvJitLink.nvidia-cuda-nvrtc-cu12/nvidia-nvjitlink-cu12(>= 12.8) as dependencies and resolve them from the siblingnvidia/*/libpackages via RPATH ($ORIGIN/../nvidia/...), exactly like torch ships its own CUDA deps. Both then come from a single pinned cu12 minor and stay mutually consistent, independent of the host toolkit.libnvJitLink.so.12in the wheel and exclude both libs from auditwheel so they aren't vendored (keeps the wheel within PyPI limits).renderer_gs.nvrtcGetErrorStringfunction-pointer signature (returnsconst char*, notnvrtcResult).torchitself is intentionally not declared, so users remain free to pick any cu12 (>= 12.8) torch variant. The>= 12.8floor is deliberate: nvrtc compiles the CCCL/CUB headers bundled from the 12.8 build toolkit and must target Blackwell (sm_120), so an older runtime is genuinely unsupported and should fail at install time rather than cryptically at first render.Validation (cluster, RTX PRO 6000 Blackwell)
gs_madrona/)DT_NEEDED=libnvrtc.so.12+libnvJitLink.so.12; RUNPATH →nvidia/*/libLD_LIBRARY_PATH→ pipnvidia-*-cu12packageslibnvrtc.so.13onLD_LIBRARY_PATH, fresh cache (the bug scenario).so.13never selected)Note on CUDA 13 / GB300 (
sm_130)Out of scope here — a cu12 wheel cannot target
sm_130. The C++ is already CUDA-major-agnostic (theDT_NEEDEDsoname follows the build toolkit); only three packaging spots are cu12-literal (the two deps, the RPATH, the auditwheel excludes). CUDA 13 will be handled as a wholesale switch when CUDA 12 support is dropped.Downstream issues
Same nvJitLink-mismatch root cause reported in genesis-world:
.so.13was being picked up (closed)All three are cu12-supported arches, so this resolves the underlying bug once a gs-madrona release carrying it is consumed by genesis-world. Reporters on a torch cu126 build (#2089, #2133) must move to a ≥ cu128 torch, since the
>= 12.8floor deliberately conflicts with the cu126-pinnednvidia-*-cu12 == 12.6.Resolves Genesis-Embodied-AI/genesis-world#2089
Resolves Genesis-Embodied-AI/genesis-world#2133
Resolves Genesis-Embodied-AI/genesis-world#2814
Closes #59