Skip to content

Resolve CUDA JIT libraries from pinned cu12 packages, not the host - #63

Merged
duburcqa merged 1 commit into
Genesis-Embodied-AI:mainfrom
duburcqa:fix-cuda-jit-resolution
Jul 22, 2026
Merged

Resolve CUDA JIT libraries from pinned cu12 packages, not the host#63
duburcqa merged 1 commit into
Genesis-Embodied-AI:mainfrom
duburcqa:fix-cuda-jit-resolution

Conversation

@duburcqa

@duburcqa duburcqa commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Problem

At first render the wheel JIT-compiles its device code with nvrtc and links the result with nvJitLink. nvrtc's output must not be newer than the nvJitLink that consumes it — when the two come from different CUDA majors the link fails with:

ERROR 4 in nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink library

Reported downstream in Genesis-Embodied-AI/genesis-world#2089, #2133, #2814.

The cause: nvrtc was dlopen'd at runtime, preferring libnvrtc.so.13 over .12, while nvJitLink was linked (and a copy bundled) at cu12. So on a host that also has a CUDA 13 runtime present, the cu12-built wheel loaded a cu13 nvrtc and fed its PTX to the cu12 nvJitLink → version mismatch and a crash on the first render.

This supersedes #59 (thanks @QuantuMope for surfacing and diagnosing the hardcoded-load issue). #59's reorder fixed the symptom on specific hosts but isn't general — it still depends on which nvrtc happens to be discoverable. This PR makes the choice deterministic.

Fix — adopt torch's model instead of dlopen-guessing or bundling

  • Link nvrtc dynamically (DT_NEEDED libnvrtc.so.12), like nvJitLink, and bind the loader's entry points to the linked symbols. dlopen ignores DT_RUNPATH, so binding to the versioned soname is what makes resolution deterministic and keeps nvrtc on the same cu12 major as nvJitLink.
  • Declare nvidia-cuda-nvrtc-cu12 / nvidia-nvjitlink-cu12 (>= 12.8) as dependencies and resolve them from the sibling nvidia/*/lib packages via RPATH ($ORIGIN/../nvidia/...), exactly like torch ships its own CUDA deps. Both then come from a single pinned cu12 minor and stay mutually consistent, independent of the host toolkit.
  • Stop bundling libnvJitLink.so.12 in the wheel and exclude both libs from auditwheel so they aren't vendored (keeps the wheel within PyPI limits).
  • Drop the now-redundant nvrtc preload in renderer_gs.
  • Fix the nvrtcGetErrorString function-pointer signature (returns const char*, not nvrtcResult).

torch itself is intentionally not declared, so users remain free to pick any cu12 (>= 12.8) torch variant. The >= 12.8 floor is deliberate: nvrtc compiles the CCCL/CUB headers bundled from the 12.8 build toolkit and must target Blackwell (sm_120), so an older runtime is genuinely unsupported and should fail at install time rather than cryptically at first render.

Validation (cluster, RTX PRO 6000 Blackwell)

Check Result
Build + import (no bundled JIT libs in gs_madrona/)
DT_NEEDED = libnvrtc.so.12 + libnvJitLink.so.12; RUNPATH → nvidia/*/lib
Resolution with clean LD_LIBRARY_PATH → pip nvidia-*-cu12 packages
Control render, fresh kernel cache
Render with libnvrtc.so.13 on LD_LIBRARY_PATH, fresh cache (the bug scenario) ✅ renders (host .so.13 never selected)

Note on CUDA 13 / GB300 (sm_130)

Out of scope here — a cu12 wheel cannot target sm_130. The C++ is already CUDA-major-agnostic (the DT_NEEDED soname follows the build toolkit); only three packaging spots are cu12-literal (the two deps, the RPATH, the auditwheel excludes). CUDA 13 will be handled as a wholesale switch when CUDA 12 support is dropped.

Downstream issues

Same nvJitLink-mismatch root cause reported in genesis-world:

All three are cu12-supported arches, so this resolves the underlying bug once a gs-madrona release carrying it is consumed by genesis-world. Reporters on a torch cu126 build (#2089, #2133) must move to a ≥ cu128 torch, since the >= 12.8 floor deliberately conflicts with the cu126-pinned nvidia-*-cu12 == 12.6.

Resolves Genesis-Embodied-AI/genesis-world#2089
Resolves Genesis-Embodied-AI/genesis-world#2133
Resolves Genesis-Embodied-AI/genesis-world#2814

Closes #59

At first render the wheel JIT-compiles its device code with nvrtc and
links the result with nvJitLink. nvrtc's output must not be newer than
the nvJitLink that consumes it; when the two come from different CUDA
majors the link fails with "ERROR 4 in nvvmAddNVVMContainerToProgram,
may need newer version of nvJitLink" (genesis-world #2089, #2133, #2814).

nvrtc was previously dlopen'd at runtime, preferring libnvrtc.so.13 over
.12, while nvJitLink was linked (and a copy bundled) at cu12. So on a host
that also has CUDA 13 present, the cu12 wheel loaded a cu13 nvrtc and fed
its PTX to the cu12 nvJitLink -> version mismatch and a crash on the first
render.

Adopt torch's model instead of dlopen-guessing or bundling:

- Link nvrtc dynamically (DT_NEEDED libnvrtc.so.12), like nvJitLink, and
  bind the loader's entry points to the linked symbols. dlopen ignores
  DT_RUNPATH, so binding to the versioned soname is what makes the choice
  deterministic and keeps nvrtc on the same cu12 major as nvJitLink.
- Declare nvidia-cuda-nvrtc-cu12 / nvidia-nvjitlink-cu12 (>= 12.8) as
  dependencies and resolve them from the sibling nvidia/*/lib packages via
  RPATH ($ORIGIN/../nvidia/...). Both then come from a single pinned cu12
  minor and stay mutually consistent, independent of the host toolkit.
- Stop bundling libnvJitLink.so.12 in the wheel and exclude both libs from
  auditwheel so they are not vendored (keeps the wheel within PyPI limits).
- Drop the now-redundant nvrtc preload in renderer_gs.

The >= 12.8 floor is intentional: nvrtc compiles the CCCL/CUB headers
bundled from the 12.8 build toolkit and must target Blackwell (sm_120),
so an older runtime is genuinely unsupported and should fail at install
time rather than cryptically at first render.

Also fix the nvrtcGetErrorString function-pointer signature (it returns
const char*, not nvrtcResult).
@duburcqa
duburcqa merged commit c7d64d3 into Genesis-Embodied-AI:main Jul 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant