Render system fails on A10 regardless of gpu or cpu render_backend

[leonmkim]

Hello,

I am attempting to run some policy evaluations in maniskill on a compute server. This code works on my local machine but on the server, I get the error:

 File "<omitting-for-privacy>/ManiSkill/mani_skill/envs/sapien_env.py", line 1085, in _setup_scene
    sapien.Scene([physx_system, sapien.render.RenderSystem(self._render_device)])
RuntimeError: Failed to find a supported physical device "cpu"

and correspondingly to "cuda:0" if I set the render_backend to "gpu". Either works for compute nodes with A40/L40, but not for A10s which I'd still like to utilize if possible.

I'm not sure where to start in debugging this and would greatly appreciate any pointers. I don't have direct sudo access to the server but can communicate with the administrators if I need to install something. For example, libvulkan seems to not be installed but I'd need to know if this is the root issue first.

Thank you!

[leonmkim]

Resolved! Adding export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json environment variable lead to a more informative error message. Turns out it was Nvidia drivers not being up to date to support Vulkan on certain nodes (unrelated to specific GPU hardware)