Dr Jit Exhausted Memory with envmap?

[Disgruntoad]

Hi I've been rendering on 3.5.2 for a while without issue, but i recently started getting this bug after updating nvidia drivers. I recompiled mitsuba as well. This is a 'cuda-spectral' variant running with a 300Mb 16k hdri. Before the driver update, I wasn't having any issues rendering it.

Any insight would be appreciated! Can't seem to figure out what I've missed.

I upgraded my mitsuba environment to 3.6.4 and reran the code. The error is now:

I'm loading a relatively large xml file (>300Kb), but the scene should fit into gpu memory with about 12 gb to spare. The xml file only includes objects, bsdfs, and emitters, not the camera or integrator. I did turn off hdri for this test, replacing it with a global illuminator (custom spectrum, embedded in xml file).

[Disgruntoad]

OK, I did want to give an update here. When my runs were failing, I was on nvidia driver 570. I reverted to 565 and Dr. Jit reported an error message for incompatibility with that driver version, requesting either <565 or >=570. Sorry, didn't screen capture the error message, but it said something about unstable drivers. I reverted to 560 and the global illumination is working, but hdris are failing still.

Error Message:

4k hdri images appear to load without complaint. The 16k versions just started throwing this error. Didn't used to.

Tried to rebuild mitsuba. Got this error message:

Not sure why... I assume there's some incompatibility between the nvidia and cuda versions, but it's unclear to me how to determine what this is...

[Disgruntoad]

import mitsuba as mi
mi.set_variant('cuda_spectral')
mi.load_file('demo-0001-debug.xml')

Uploaded as text file (xml unsupported)
demo-0001-debug.txt

text contents:

<scene version="3.6.4">
    <emitter type="envmap">
        <string name="filename" value="/data/gen/hdri/polyhaven_puresky_hdri/16k/hilltop_construction_16k.exr"/>
    </emitter>
</scene>

Here's the jpg, unfortunately I cannot upload the exr file here.

This is always throwing the dr jit error message in the above comment. I do sometimes get errors on loading 4k xml files for bsdf textures. Here's the error message that throws:

Really at a loss here. I'd been using these images to render for months without issue.

[merlinND]

Hello @Disgruntoad,

There were indeed bug in certain drivers versions, that are only fixed / worked around by using both driver 570+ and Mitsuba 3.6.4+ (which includes DrJit 1.0.5+).

There could have been regressions in memory usage in recent Mitsuba versions, but it's very strange that going back to the previous driver did not resolve the issue.
If you rollback both driver, Mitsuba and DrJit to versions that previously worked for you, what exact error messages are you seeing?

[Disgruntoad]

I double-checked the updates applied. I was mistaken about an nvidia driver update, I was on 570 when it was functioning. There was a libcudnn update. Here's the list of updates I'd applied:

Start-Date: 2025-02-18  15:13:44                                                                                                                                       
Commandline: apt upgrade                                                                                                                                               
Requested-By: toad (1000)                                                                                                                                              
Install: linux-hwe-6.11-tools-6.11.0-17:amd64 (6.11.0-17.17~24.04.2, automatic), mesa-libgallium:amd64 (24.2.8-1ubuntu1~24.04.1, automatic), mesa-libgallium:i386 (24.\
2.8-1ubuntu1~24.04.1, automatic), libllvm19:amd64 (1:19.1.1-1ubuntu1~24.04.2, automatic), libllvm19:i386 (1:19.1.1-1ubuntu1~24.04.2, automatic), linux-modules-extra-6\
.11.0-17-generic:amd64 (6.11.0-17.17~24.04.2, automatic), linux-image-6.11.0-17-generic:amd64 (6.11.0-17.17~24.04.2, automatic), linux-tools-6.11.0-17-generic:amd64 (\
6.11.0-17.17~24.04.2, automatic), linux-hwe-6.11-headers-6.11.0-17:amd64 (6.11.0-17.17~24.04.2, automatic), linux-headers-6.11.0-17-generic:amd64 (6.11.0-17.17~24.04.\
2, automatic), linux-modules-6.11.0-17-generic:amd64 (6.11.0-17.17~24.04.2, automatic)                                                                                 
Upgrade: linux-headers-generic-hwe-24.04:amd64 (6.8.0-52.53, 6.11.0-17.17~24.04.2+2), linux-generic-hwe-24.04:amd64 (6.8.0-52.53, 6.11.0-17.17~24.04.2+2), linux-tools\
-common:amd64 (6.8.0-52.53, 6.8.0-53.55), udev:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), systemd-oomd:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), openssh-client:amd\
64 (1:9.6p1-3ubuntu13.5, 1:9.6p1-3ubuntu13.8), libglx-mesa0:amd64 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), libglx-mesa0:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~\
24.04.1), python3-distupgrade:amd64 (1:24.04.23, 1:24.04.26), language-pack-en-base:amd64 (1:24.04+20240817, 1:24.04+20250130), mutter-common-bin:amd64 (46.2-1ubuntu0\
.24.04.4, 46.2-1ubuntu0.24.04.5), systemd-timesyncd:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), perl:amd64 (5.38.2-3.2build2, 5.38.2-3.2build2.1), libunwind8:amd64 (1\
.6.2-3build1, 1.6.2-3build1.1), libpam-systemd:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), ssh:amd64 (1:9.6p1-3ubuntu13.5, 1:9.6p1-3ubuntu13.8), libgbm1:amd64 (24.0.9\
-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), libgbm1:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), zip:amd64 (3.0-13ubuntu0.1, 3.0-13ubuntu0.2), ubuntu-release-upgrade\
r-gtk:amd64 (1:24.04.23, 1:24.04.26), libmutter-14-0:amd64 (46.2-1ubuntu0.24.04.4, 46.2-1ubuntu0.24.04.5), libsystemd0:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), lib\
systemd0:i386 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), libnss-systemd:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), openssh-server:amd64 (1:9.6p1-3ubuntu13.5, 1:9.6p1-3ubu\
ntu13.8), language-pack-en:amd64 (1:24.04+20240817, 1:24.04+20250130), language-pack-gnome-en-base:amd64 (1:24.04+20240817, 1:24.04+20250130), libxatracker2:amd64 (24\
.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), mutter-common:amd64 (46.2-1ubuntu0.24.04.4, 46.2-1ubuntu0.24.04.5), gnome-shell:amd64 (46.0-0ubuntu6~24.04.5, 46.0-0ubuntu6\
~24.04.6), python-apt-common:amd64 (2.7.7ubuntu3, 2.7.7ubuntu4), systemd:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), libudev1:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.\
5), libudev1:i386 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), mesa-va-drivers:amd64 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), mesa-va-drivers:i386 (24.0.9-0ubuntu0.3, 2\
4.2.8-1ubuntu1~24.04.1), libgl1-mesa-dri:amd64 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), libgl1-mesa-dri:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), grub-e\
fi-amd64:amd64 (2.12-1ubuntu7, 2.12-1ubuntu7.1), systemd-dev:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), ubuntu-desktop:amd64 (1.539.1, 1.539.2), file-roller:amd64 (4\
4.1-1, 44.3-0ubuntu1), systemd-resolved:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), gir1.2-mutter-14:amd64 (46.2-1ubuntu0.24.04.4, 46.2-1ubuntu0.24.04.5), mesa-vulkan\
-drivers:amd64 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), mesa-vulkan-drivers:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), libcudnn9-cuda-12:amd64 (9.7.0.66-\
1, 9.7.1.26-1), base-files:amd64 (13ubuntu10.1, 13ubuntu10.2), grub-efi-amd64-signed:amd64 (1.202+2.12-1ubuntu7, 1.202.2+2.12-1ubuntu7.1), python3-apt:amd64 (2.7.7ubu\
ntu3, 2.7.7ubuntu4), bpftrace:amd64 (0.20.2-1ubuntu4.2, 0.20.2-1ubuntu4.3), cups-browsed:amd64 (2.0.0-0ubuntu10.2, 2.0.0-0ubuntu10.3), gnome-shell-common:amd64 (46.0-\
0ubuntu6~24.04.5, 46.0-0ubuntu6~24.04.6), linux-image-generic-hwe-24.04:amd64 (6.8.0-52.53, 6.11.0-17.17~24.04.2+2), linux-firmware:amd64 (20240318.git3b128b60-0ubunt\
u2.7, 20240318.git3b128b60-0ubuntu2.9), grub-efi-amd64-bin:amd64 (2.12-1ubuntu7, 2.12-1ubuntu7.1), libglapi-mesa:amd64 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), l\
ibglapi-mesa:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), perl-base:amd64 (5.38.2-3.2build2, 5.38.2-3.2build2.1), openssh-sftp-server:amd64 (1:9.6p1-3ubuntu13.5\
, 1:9.6p1-3ubuntu13.8), ubuntu-standard:amd64 (1.539.1, 1.539.2), ubuntu-desktop-minimal:amd64 (1.539.1, 1.539.2), libsystemd-shared:amd64 (255.4-1ubuntu8.4, 255.4-1u\
buntu8.5), language-pack-gnome-en:amd64 (1:24.04+20240817, 1:24.04+20250130), libcudnn9-dev-cuda-12:amd64 (9.7.0.66-1, 9.7.1.26-1), ubuntu-kernel-accessories:amd64 (1\
.539.1, 1.539.2), systemd-sysv:amd64 (255.4-1ubuntu8.4, 255.4-1ubuntu8.5), libcudnn9-static-cuda-12:amd64 (9.7.0.66-1, 9.7.1.26-1), libegl-mesa0:amd64 (24.0.9-0ubuntu\
0.3, 24.2.8-1ubuntu1~24.04.1), libegl-mesa0:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), ubuntu-release-upgrader-core:amd64 (1:24.04.23, 1:24.04.26), perl-modul\
es-5.38:amd64 (5.38.2-3.2build2, 5.38.2-3.2build2.1), libperl5.38t64:amd64 (5.38.2-3.2build2, 5.38.2-3.2build2.1), mesa-vdpau-drivers:amd64 (24.0.9-0ubuntu0.3, 24.2.8\
-1ubuntu1~24.04.1), mesa-vdpau-drivers:i386 (24.0.9-0ubuntu0.3, 24.2.8-1ubuntu1~24.04.1), ubuntu-minimal:amd64 (1.539.1, 1.539.2), linux-libc-dev:amd64 (6.8.0-52.53, \
6.8.0-53.55)                                                                                                                                                           
End-Date: 2025-02-18  15:15:30 

I did use the 3.5.2 build, but it just reports this error message and hangs until I kill it. (I printed the mitsuba & drjit versions.)

Updated mitsuba & dr jit to latest stable:

[Disgruntoad]

Ok, there's definitely a memory issue with host allocations. There's a leak that's causing issues on my system.

The spike & crash happens when running this (3.6.4) (it only loads the emitter):

It's not properly freeing memory. Not sure why it's unable to allocate space for a single 16k emitter any more. The disk usage is 300MB for the exr.

[njroussel]

Hi @Disgruntoad

Some of the information got scattered across multiple threads.
Let me centralize some stuff here:

Code used to work with 3.5.2 but broke with 3.6.4. Now when debugging, you ran into on 3.5.2 again but it also failed there. Is that correct? If that's the case, it's very strange. It sounds more like a system issue.

You shared a few different error logs. Once it tried to allocate 2 ** 32 bytes, that's what I'd expect for a 16k image. For some internal shuffling, it might actually need a duplicate of that so ~9GB of data.
I've tried running the tiny script and scene you provided here with a very similar setup (4090 + 64GB of RAM) and didn't encounter any issues.

Ok, there's definitely a memory issue with host allocations. There's a leak that's causing issues on my system.

I'm not sure I followed this part. What lead you to believe this?

You can also compile the project yourself in debug and set a breakpoint in jitc_malloc when the memory is HostPinned to carefully follow your VRAM consumption.

[Disgruntoad]

Hi @njroussel,

This is the rough timeline:
On the 18th, I was running mitsuba when my computer froze. Forced a shutdown, installed updated, restarted. The updates applied are logged above. On restart, I started getting the errors on 3.5.2. Tried 3.6.4, same error (more detailed message). Reinstalled nvidia, same error.

I agree with your assessment that it's likely my system, so I reinstalled the os (ubuntu 24.04) and rebuilt 5.6.4, but I'm still getting the same error messages (nvidia 570 drivers).

I believe there is / was a memory leak since I'd gradually increase memory usage while getting the error messages until my entire system became laggy and requires a restart while sitting at 64 gb used (only firefox and terminal open). I also noticed memory usage increase after opening an ipython session, getting the error, and exiting the session, whereupon it reports a memory leak.

I suspect that I may have an issue with my 14900k degrading? Possibly? I ran a stress test, but it didn't show anything. I've updated the bios to intel's latest recommended stable microcode (from December 2024). At this point, I'm not sure what else would be causing it...

I'll try compiling for debug as well. Fyi, I did have to install sudo apt install libstdc++-14-dev in order to call cmake on ubuntu 24.04. Wasn't mentioned https://mitsuba.readthedocs.io/en/stable/src/developer_guide/compiling.html.

[Disgruntoad]

I tried to rebuild 3.5.2 as well, but it seems I've misconfigured something. Same setup as 3.6.4, but it reports this error on the ninja command.

[164/1182] Building CXX object ext/embree/kernels/CMakeFiles/embree_sse42.dir/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp.o
FAILED: ext/embree/kernels/CMakeFiles/embree_sse42.dir/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp.o 
/usr/bin/clang++-19 -DEMBREE_TARGET_AVX -DEMBREE_TARGET_AVX2 -DEMBREE_TARGET_AVX512 -DEMBREE_TARGET_SSE42 -DTASKING_INTERNAL  -stdlib=libc++ -D_LIBCPP_VERSION -fcolor-diagnostics -Wall -Wformat -Wformat-security -fsigned-char -Wno-unused-variable -Wno-unused-private-field -fPIE -fPIC -std=c++11 -fvisibility=hidden -fvisibility-inlines-hidden -fno-strict-aliasing -fno-tree-vectorize -D_FORTIFY_SOURCE=2  -DNDEBUG -O3 -msse4.2 -MD -MT ext/embree/kernels/CMakeFiles/embree_sse42.dir/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp.o -MF ext/embree/kernels/CMakeFiles/embree_sse42.dir/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp.o.d -o ext/embree/kernels/CMakeFiles/embree_sse42.dir/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp.o -c /home/dev/mitsuba-v3-5-2/build/ext/embree/kernels/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp
In file included from /home/dev/mitsuba-v3-5-2/build/ext/embree/kernels/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp:1:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_intersector1_bvh4.cpp:4:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_intersector1.cpp:4:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_intersector1.h:6:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh.h:7:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_node_aabb.h:6:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_node_base.h:6:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_node_ref.h:7:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../common/alloc.h:8:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../common/scene.h:14:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../common/scene_curves.h:10:
/home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../common/../subdiv/bezier_curve.h:138:56: error: no member named 'u' in 'QuadraticBezierCurve<V>'
  138 |         return cout << "QuadraticBezierCurve ( (" << a.u.lower << ", " << a.u.upper << "), " << a.v0 << ", " << a.v1 << ", " << a.v2 << ")";
      |                                                      ~ ^
/home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../common/../subdiv/bezier_curve.h:138:77: error: no member named 'u' in 'QuadraticBezierCurve<V>'
  138 |         return cout << "QuadraticBezierCurve ( (" << a.u.lower << ", " << a.u.upper << "), " << a.v0 << ", " << a.v1 << ", " << a.v2 << ")";
      |                                                                           ~ ^
In file included from /home/dev/mitsuba-v3-5-2/build/ext/embree/kernels/bvh/bvh_intersector1_bvh4.cpp.sse42.cpp:1:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_intersector1_bvh4.cpp:4:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/bvh_intersector1.cpp:24:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../geometry/curve_intersector_virtual.h:20:
In file included from /home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../geometry/spherei_intersector.h:7:
/home/dev/mitsuba-v3-5-2/ext/embree/kernels/bvh/../geometry/pointi.h:215:51: error: no member named 'v0' in 'PointMi<M>'
  215 |       return cout << "Line" << M << "i {" << line.v0 << ", " << line.geomID() << ", " << line.primID() << "}";
      |                                              ~~~~ ^
3 errors generated.
[169/1182] Building CXX object ext/drjit/src/python/CMakeFiles/drjit-python.dir/bind_cuda_ad.cpp.o
ninja: build stopped: subcommand failed.

The error message repeats around a dozen times. It's not clear to me what the issue is. I am using clang 19, but it compiled fine before the os reinstall.

[merlinND]

The Embree compilation error you encountered can be fixed by checking out the latest Embree submodule:

cd ext/embree
git checkout master
git pull

and compiling again. In particular, this is the commit that fixes it: mitsuba-renderer/embree@ad0e6cb

[Disgruntoad]

Awesome -- thanks, that did allow for compilation. Unfortunately, I get the same jit_flush_malloc_cache(): Dr.Jit exhausted the available memory and had to flush its allocation cache to free up additional memory. error for 3.5.2, 3.6.4, and current master.

I'm going to try to reinstall my conda environment tomorrow. Maybe it was corrupted somehow. Other than that, I'm pretty much out of ideas.

[Disgruntoad]

Just an update -- never resolved this. Ended up creating lower-resolution images and using those.