silent computation error on mitsuba 3.5.2 on nvidia driver update on principled scene

[KoykL]

I have a very complicated project that depends on mitsuba 3.5.2 and drjit 0.4.6. It is not easy to migrate to newer version due to nanobind. Recently, I found mitsuba produces silent computation error. I understand mitsuba 3.5.2 is EOL. Unfortunately, migration is really not an option right now.

I am just wondering if anyone have any clue what might be the cause. After all, nvidia driver is supposed to guarantee ABI stability.

The machine is part of managed cluster, and I have no control over nvidia driver version. The nvidia driver version is always update to date. The computation error only happens recently. The current version is:
Driver Version: 580.95.05 CUDA Version: 13.0

On mitsuba 3.5.2 with path integrator, the image produced is (notice the artifacts at bottom right):

On mitsuba 3.6.0 with path integrator, the image produced is:

The scene mainly uses principled brdf. The scene (scene_principled.xml) is attached here.

Archive 2.zip

[njroussel]

Hi @KoykL

I'm a bit confused.

You mention the driver a lot, is this somehow related? Did the changes happen with a driver upgrade too? Or is is solely related to the mitsuba version.

Please provide a complete archive. It's missing all the data files currently. Without that, it's hard to understand what I'm even looking at.

I would crop the film to the region you're interested in, and trace a single ray with lots of prints. The idea being that you can hopefully figure out what changed between both versions by logging information at each interaction.

Lastly, is it expected that the floor is missing?

[KoykL]

Thank you for your help!!!

I do not have control over driver version on the machine (Machine A). However, I only noticed the rendering is broken with mitsuba 3.5.2 after a maintenance session which also updates NVIDIA driver. I have always been using mitsuba 3.5.2 for this project and it worked fine before.

I suspect the driver, because I noticed previously NVIDIA driver version often break/fix optix based project with exactly same code. (e.g. https://github.com/fudan-zvg/IRGS/tree/main/submodules/surfel_tracer)

For surfel_tracer, the error was optixcreatepipeline failed. (I also tried on multiple machines even on ones I do have control over driver and it worked on older nvidia driver versions) Also after another nvidia driver update on Machine A, the problem is magically gone. I also saw this optixcreatepipeline error on mitsuba 3.5.2 with my custom rendering logic after a particular nvidia driver version update on Machine A. (My custom rendering logic worked fine before the update.)

The problem this time is indeed different and is silent corruption. I also didn't rigorously try to isolate driver as the issue back then. I am only suspecting maybe its driver again as nothing else changes.

It is expected the floor to be missing.

I will provide complete archive and try your suggestions.

[njroussel]

Hi again @KoykL

I've rendered the scene with a few different mitsuba version and driver combinations and couldn't reproduce the error.

The update to mitsuba 3.6.0 comes with Dr.Jit' v1.0.0. There are a lot of deep internal changes to Dr.Jit in that first major version. If I had to guess what is going on here: I believe that older verisons of MItsuba/Dr.Jit maybe relied on some undefined behavior which happened to still produce correct results on older drivers.

From what you describe this issue does indeed seem to be tied to the change of drivers.

Assuming this is some undefined behavior in the code and not the driver itself, the only thing I can suggest is to build mitsuba v3.5.2 with the cmake option MI_ENABLE_OPTIX_DEBUG_VALIDATION=ON. With some hope, the new driver will pick up the source of this bug/error.

[KoykL]

Let me try that.

This is a separate issue: I am also trying to port my code to mitsuba 3.6.2 in parallel and see which route is more feasible.

After patching nanobind for multi-inheritance support, and fix a couple syntax incompatibilities, my training code does run, but will have cuda out of memory error. So far I found majority of the memory allocation comes from pytorch, but pytorch version is exactly the same compared to the mitsuba 3.5.2 environment.

I will let you know how it goes.

[KoykL]

complete archive with reference rendering generated on mitsuba 3.6.4 on CPU is attached: https://umd.box.com/s/0rnea3d1hu7nlkpj8f877jnaxf37jttp

[KoykL]

Rendered with mitsuba 3.5.2 scalar backend.
scene_principled.exr.zip