[Bug]: Batch Rendering crashing.

[nonlinearjunkie]

Bug Description

When I try to run,:

python Genesis/examples/rigid/single_franka_batch_render.py , I get the following error::

(.venv) pndguys@pndguys-B650M-AORUS-PRO-AX:~/Documents/genesis-learning/Genesis/examples/rigid$ python ./single_franka_batch_render.py
[Genesis] [16:31:56] [INFO] ╭─────────────────────────────────────────────────────────────────────────────────────╮
[Genesis] [16:31:56] [INFO] │┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉ Genesis ┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉│
[Genesis] [16:31:56] [INFO] ╰─────────────────────────────────────────────────────────────────────────────────────╯
[Genesis] [16:31:57] [INFO] Running on [NVIDIA A100 80GB PCIe] with backend gs.cuda. Device memory: 79.25 GB.
[Genesis] [16:31:57] [INFO] 🚀 Genesis initialized. 🔖 version: 0.3.10, 🎨 theme: dark, 🌱 seed: None, 🐛 debug: False, 📏 precision: 32, 🔥 performance: False, 💬 verbose: INFO
[Genesis] [16:31:58] [INFO] Scene <92a62fa> created.
[Genesis] [16:31:58] [INFO] Adding <gs.RigidEntity>. idx: 0, uid: <9268de5>, morph: <gs.morphs.Plane>, material: <gs.materials.Rigid>.
[Genesis] [16:31:58] [INFO] Adding <gs.RigidEntity>. idx: 1, uid: <3a7db23>, morph: <gs.morphs.MJCF(file='/home/pndguys/Documents/genesis-learning/Genesis/genesis/assets/xml/franka_emika_panda/panda.xml')>, material: <gs.materials.Rigid>.
[Genesis] [16:31:58] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint1`
[Genesis] [16:31:58] [WARNING] (MJCF) Actuator control gain and bias parameters cannot be reduced to a unique PD control position gain. Using max between gain and bias for joint `finger_joint1`.
[Genesis] [16:31:58] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint2`
[Genesis] [16:31:58] [WARNING] (MJCF) Actuator control gain and bias parameters cannot be reduced to a unique PD control position gain. Using max between gain and bias for joint `finger_joint2`.
[Genesis] [16:31:58] [INFO] Applying offset to base link's pose with user provided value in morph.
[Genesis] [16:31:58] [INFO] Building scene <92a62fa>...
[Genesis] [16:31:58] [WARNING] Reference robot position exceeds joint limits.
[Genesis] [16:31:58] [WARNING] Constraint solver time constant should be greater than 2*substep_dt. timeconst is changed from `0.005` to `0.02`). Decrease simulation timestep or increase timeconst to avoid altering the original value.
[Genesis] [16:32:00] [INFO] Compiling simulation kernels...
[Genesis] [16:32:14] [INFO] Building visualizer...
[Genesis] [16:32:14] [INFO] Raytracer and Rasterizer requires binding to the camera with a specific environment index. Defaulting to 'rendered_envs_idx[0]'. Please specify 'env_idx' if necessary.
Using raytracer
All Vulkan layers supported
CUDA linking Failed!
ERROR 4 in nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink library


Error at /project/src/mw/cuda_exec.cpp:1197 in auto madrona::buildBVHKernels(const CompileConfig &, int32_t, std::pair<int, int>, CudaBatchRenderConfig::RenderMode)::(anonymous class)::operator()(nvJitLinkResult) const
nvJitLink error: Internal error
Aborted (core dumped)

Environment

nvidia-smi :

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          Off |   00000000:01:00.0 Off |                    0 |
| N/A   30C    P0             43W /  300W |      14MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2111      G   /usr/lib/xorg/Xorg                        4MiB |
+-----------------------------------------------------------------------------------------+

nvcc --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Fri_Jun_14_16:34:21_PDT_2024
Cuda compilation tools, release 12.6, V12.6.20
Build cuda_12.6.r12.6/compiler.34431801_0

pip list :

Package                   Version      Editable project location
------------------------- ------------ ------------------------------------------------
absl-py                   2.3.1
addict                    2.4.0
annotated-types           0.7.0
anyio                     4.12.0
asttokens                 3.0.1
attrs                     25.4.0
av                        16.0.1
black                     25.12.0
blinker                   1.9.0
certifi                   2025.11.12
cffi                      2.0.0
charset-normalizer        3.4.4
click                     8.3.1
coacd                     1.0.7
colorama                  0.4.6
comm                      0.2.3
ConfigArgParse            1.7.1
contourpy                 1.3.2
cycler                    0.12.1
dash                      3.3.0
dataclasses-json          0.6.7
decorator                 5.2.1
Deprecated                1.3.1
dill                      0.4.0
etils                     1.13.0
exceptiongroup            1.3.1
execnet                   2.1.2
executing                 2.2.1
fast_simplification       0.1.13
fastjsonschema            2.21.2
filelock                  3.20.0
Flask                     3.1.2
fonttools                 4.61.1
freetype-py               2.5.1
fsspec                    2025.12.0
genesis-world             0.3.10       /home/pndguys/Documents/genesis-learning/Genesis
gitdb                     4.0.12
GitPython                 3.1.45
glfw                      2.10.0
gs-madrona                0.0.5
gstaichi                  4.5.0
h11                       0.16.0
hf-xet                    1.2.0
httpcore                  1.0.9
httpx                     0.28.1
huggingface_hub           1.2.3
idna                      3.11
ImageIO                   2.37.2
imageio-ffmpeg            0.6.0
importlib_metadata        8.7.0
importlib_resources       6.5.2
iniconfig                 2.3.0
ipython                   8.37.0
ipywidgets                8.1.8
itsdangerous              2.2.0
jedi                      0.19.2
Jinja2                    3.1.6
joblib                    1.5.3
jsonschema                4.25.1
jsonschema-specifications 2025.9.1
jupyter_core              5.9.1
jupyterlab_widgets        3.0.16
kiwisolver                1.4.9
lazy_loader               0.4
libigl                    2.6.1
llvmlite                  0.46.0
markdown-it-py            4.0.0
MarkupSafe                2.1.5
marshmallow               3.26.1
matplotlib                3.10.8
matplotlib-inline         0.2.1
mdurl                     0.1.2
moviepy                   2.2.1
mpmath                    1.3.0
mujoco                    3.3.7
mypy_extensions           1.1.0
nbformat                  5.10.4
nest-asyncio              1.6.0
networkx                  3.4.2
numba                     0.63.1
numpy                     2.2.6
nvidia-cublas-cu12        12.6.4.1
nvidia-cuda-cupti-cu12    12.6.80
nvidia-cuda-nvrtc-cu12    12.6.77
nvidia-cuda-runtime-cu12  12.6.77
nvidia-cudnn-cu12         9.10.2.21
nvidia-cufft-cu12         11.3.0.4
nvidia-cufile-cu12        1.11.1.6
nvidia-curand-cu12        10.3.7.77
nvidia-cusolver-cu12      11.7.1.2
nvidia-cusparse-cu12      12.5.4.2
nvidia-cusparselt-cu12    0.7.1
nvidia-nccl-cu12          2.27.3
nvidia-nvjitlink-cu12     12.6.85
nvidia-nvtx-cu12          12.6.77
open3d                    0.19.0
opencv-python             4.12.0.88
OpenEXR                   3.4.4
packaging                 25.0
pandas                    2.3.3
parso                     0.8.5
pathspec                  0.12.1
pexpect                   4.9.0
pillow                    11.3.0
pip                       22.0.2
platformdirs              4.5.1
plotly                    5.24.1
pluggy                    1.6.0
pooch                     1.8.2
proglog                   0.1.12
prompt_toolkit            3.0.52
protobuf                  6.33.2
psutil                    7.1.3
ptyprocess                0.7.0
pure_eval                 0.2.3
py                        1.11.0
py-cpuinfo                9.0.0
pybind11                  3.0.1
pybind11-global           3.0.1
pycollada                 0.9.2
pycparser                 2.23
pydantic                  2.12.5
pydantic_core             2.41.5
PyGEL3D                   0.6.1
pyglet                    2.1.11
pygltflib                 1.16.0
Pygments                  2.19.2
pymeshlab                 2025.7
PyOpenGL                  3.1.10
pyparsing                 3.2.5
pyquaternion              0.9.9
pysplashsurf              0.12.0.2
pytest                    9.0.2
pytest-forked             1.6.0
pytest-print              1.2.0
pytest-random-order       1.2.0
pytest-rerunfailures      16.1
pytest-xdist              3.8.0
python-dateutil           2.9.0.post0
python-dotenv             1.2.1
pytokens                  0.3.0
pytz                      2025.2
pyvista                   0.46.4
PyYAML                    6.0.3
referencing               0.37.0
requests                  2.32.5
retrying                  1.4.2
rich                      14.2.0
rpds-py                   0.30.0
rtree                     1.4.1
scikit-image              0.25.2
scikit-learn              1.7.2
scipy                     1.15.3
scooby                    0.11.0
sentry-sdk                2.48.0
setuptools                80.9.0
shellingham               1.5.4
six                       1.17.0
smmap                     5.0.2
stack-data                0.6.3
sympy                     1.14.0
syrupy                    5.0.0
tenacity                  9.1.2
tetgen                    0.6.7
threadpoolctl             3.6.0
tifffile                  2025.5.10
tomli                     2.3.0
torch                     2.8.0+cu126
torchaudio                2.8.0+cu126
torchvision               0.23.0+cu126
tqdm                      4.67.1
traitlets                 5.14.3
trimesh                   4.10.1
triton                    3.4.0
typer-slim                0.20.1
typing_extensions         4.15.0
typing-inspect            0.9.0
typing-inspection         0.4.2
tzdata                    2025.3
urllib3                   2.6.2
vtk                       9.5.2
wandb                     0.23.1
wcwidth                   0.2.14
Werkzeug                  3.1.4
widgetsnbextension        4.0.15
wrapt                     2.0.1
z3-solver                 4.15.4.0
zipp                      3.23.0

Python version: Python 3.10

Release version or Commit ID

v0.3.10

[duburcqa]

@nonlinearjunkie Could you try again with the latest main branch? gs-madrona has been updated very recently, and comes with a few bug fixes and with new features.

[patorrad]

I am running into the same issue. I have tried multiple flavors of genesis-world, gs-madrona, pytorch and cudatoolkit. Older versions of genesis do not line up with the method arguments in gs-madrona. I am running in a conda environment.

Environment:
OS: Ubuntu 22.04
GPU RTX 5090
CUDA 13.0
cudatoolkit 13.1
pytorch==2.9.1+cu130
genesis-world==0.3.11
gs-madrona==0.0.7.post2
nvidia-nvjitlink==13.0.39 or 13.1.80

[duburcqa]

I found the issue. We only support 12.4<=CUDA<13.0.

[duburcqa]

@nonlinearjunkie Although you are using valid CUDA Toolkit (12.6 apparently), your driver is too new (580.95.05 → CUDA 13.0). This is currently not supported. gs-madrona ends up mixing CUDA 13.0 NVVM from the driver with CUDA 12.4 nvJitLink that is bundled with gs-madrona when doing runtime CUDA JIT linking. Note that it is not possible to bundle the entire driver unfortunately as you may expect.

[duburcqa]

@nonlinearjunkie Hum... this is strange. After double checking, I just installed 580.95.05 driver as you did, with torch+cu12.6 and it works just great. So apparently the version of the driver is not a limitation... Maybe it is due to CUDA Toolkit version? I don't see any other difference....

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Tue_May_27_02:21:03_PDT_2025
Cuda compilation tools, release 12.9, V12.9.86
Build cuda_12.9.r12.9/compiler.36037853_0

Could you try upgrading your version of nvcc?

[duburcqa]

@patorrad What is your backtrace? Is it the exact same? I still think it may be related to CUDA 13.0 not being supported in you case.

[duburcqa]

@nonlinearjunkie Could you try running this before anything else?

import ctypes
from importlib.metadata import distribution, PackageNotFoundError

for package_name, lib_name in (
    ("nvidia_cuda_nvrtc_cu12", "libnvrtc.so.1"), 
    ("nvidia_nvjitlink_cu12", "libnvJitLink.so.12")
):
    try:
        dist = distribution(package_name)
        for file in dist.files:
            if file.name.startswith(lib_name):
                ctypes.CDLL(dist.locate_file(file), ctypes.RTLD_LOCAL)
                break
    except PackageNotFoundError:
        pass

[patorrad]

Here is my stack trace:

[Genesis] [14:32:01] [INFO] ╭───────────────────────────────────────────────────────────────────────────────╮
[Genesis] [14:32:01] [INFO] │┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈ Genesis ┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈┉┈│
[Genesis] [14:32:01] [INFO] ╰───────────────────────────────────────────────────────────────────────────────╯
[Genesis] [14:32:02] [INFO] Running on [NVIDIA GeForce RTX 5090] with backend gs.cuda. Device memory: 31.35 GB.
[Genesis] [14:32:02] [INFO] 🚀 Genesis initialized. 🔖 version: 0.3.11, 🎨 theme: dark, 🌱 seed: None, 🐛 debug: False, 📏 precision: 32, 🔥 performance: False, 💬 verbose: INFO
[Genesis] [14:32:02] [INFO] Scene <221fd4b> created.
[Genesis] [14:32:04] [INFO] Adding <gs.RigidEntity>. idx: 0, uid: <ed38c7c>, morph: <gs.morphs.Plane>, material: <gs.materials.Rigid>.
[Genesis] [14:32:04] [INFO] Adding <gs.RigidEntity>. idx: 1, uid: <4ac9c59>, morph: <gs.morphs.MJCF(file='/home/paolo/Downloads/Genesis/genesis/assets/xml/franka_emika_panda/panda.xml')>, material: <gs.materials.Rigid>.
[Genesis] [14:32:04] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint1`
[Genesis] [14:32:04] [WARNING] (MJCF) Actuator control gain and bias parameters cannot be reduced to a unique PD control position gain. Using max between gain and bias for joint `finger_joint1`.
[Genesis] [14:32:04] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint2`
[Genesis] [14:32:04] [WARNING] (MJCF) Actuator control gain and bias parameters cannot be reduced to a unique PD control position gain. Using max between gain and bias for joint `finger_joint2`.
[Genesis] [14:32:04] [INFO] Building scene <221fd4b>...
[Genesis] [14:32:04] [WARNING] Reference robot position exceeds joint limits.
[Genesis] [14:32:04] [WARNING] Constraint solver time constant should be greater than 2*substep_dt. timeconst is changed from `0.005` to `0.02`). Decrease simulation timestep or increase timeconst to avoid altering the original value.
[Genesis] [14:32:05] [INFO] Compiling simulation kernels...
[Genesis] [14:32:06] [INFO] Building visualizer...
[Genesis] [14:32:07] [INFO] Raytracer and Rasterizer requires binding to the camera with a specific environment index. Defaulting to 'rendered_envs_idx[0]'. Please specify 'env_idx' if necessary.
CUDA linking Failed!
ERROR 4 in nvvmAddNVVMContainerToProgram, may need newer version of nvJitLink library


Error at /project/src/mw/cuda_exec.cpp:1197 in auto madrona::buildBVHKernels(const CompileConfig &, int32_t, std::pair<int, int>, CudaBatchRenderConfig::RenderMode)::(anonymous class)::operator()(nvJitLinkResult) const
nvJitLink error: Internal error
Aborted (core dumped)

Printing out the libraries not found from the script above gives me:
Error: nvidia_cuda_nvrtc_cu12 not found
Error: nvidia_nvjitlink_cu12 not found

[duburcqa]

@patorrad This is not applicable to you because you are using CUDA 13.0, which is probably not compatible. But you can try if you want, by replacing _cu12 with _cu13

[duburcqa]

@patorrad Let me know if you tried it. Who knows, maybe it is work after all x) Just don't have too much hope...