loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0

[j-baker]

hi! I'm trying to use Vulkan in a Docker container. On my host, Vulkan works fine.

I'm using Ubuntu 22.04.
I'm using NVIDIA 535.129.03 drivers.
I have a Tesla T4 GPU.

I'm using container toolkit 1.14.1, and inside my containers nvidia-smi works fine. I've installed Docker using the docker.com instructions and have set up the container runtime.

In a Ubuntu container started with:

sudo docker run -it --env NVIDIA_DRIVER_CAPABILITIES=all --rm --runtime=nvidia --gpus all ubuntu

if I run apt install vulkan-tools libxext6 and then vulkaninfo I see

ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0

and no GPU is listed. This is additionally the case on nVidia drivers as old as 470 (but with container toolkit 1.14.1).

[lss0510]

Hello, I have encountered the same problem with you. Have you solved the problem? Can you share the solution with me?

[micahcc]

Also have this issue.

This is the last thing strace shows before erroring, so maybe its not mounting /dev/dri?

getpid()                                = 223
openat(AT_FDCWD, "/dev/dri", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/dri", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)

[elezar]

Hi @micahcc. You mentioned an ubuntu container. What is your host distribution? (it's Ubuntu 22.04)

Could you also provide the output for:

ls -al /dev/dri

[elezar]

As a note: https://github.com/NVIDIA/nvidia-container-toolkit/releases/tag/v1.14.4 does include a minor fix for the vulkan ICD config location.

[audiovention]

Had the same issue, eventually resolved it by installing "libegl1" inside the container
As suggested here:
https://forums.developer.nvidia.com/t/minimal-docker-vulkan-offscreen-setup/242883

[micahcc]

I eventually resolved this in mine by using the newer cdi spec for podman

[NayamAmarshe]

I'm getting the same error with Docker Engine:

ubuntu:~$ docker run -it --init --env-file .env --network="host" --rm --gpus all -e PYTHONUNBUFFERED=1 ubuntu
Thu Jul 11 10:41:19 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:00:1E.0 Off |                    0 |
| N/A   29C    P8              9W /   70W |       1MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at ./vulkaninfo/vulkaninfo.h:649:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

Is there a fix?

[powerman]

Looks like it's not related to docker, I got the same error in Obsidian:

$ obsidian
2024-07-21 12:17:41 Loading updated app package /home/powerman/.config/obsidian/obsidian-1.6.7.asar
2024-07-21 12:17:41 Checking for update using Github
2024-07-21 12:17:42 Success.
2024-07-21 12:17:42 Latest version is 1.6.7
2024-07-21 12:17:42 App is up to date.
Warning: loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
Warning: vkCreateInstance: Found no drivers!
Warning: vkCreateInstance failed with VK_ERROR_INCOMPATIBLE_DRIVER
    at CheckVkSuccessImpl (../../third_party/dawn/src/dawn/native/vulkan/VulkanError.cpp:101)
    at CreateVkInstance (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:493)
    at Initialize (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:379)
    at Create (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:301)
    at operator() (../../third_party/dawn/src/dawn/native/vulkan/BackendVk.cpp:556)

Steam games which use Vulkan works fine, Ollama in docker also works using GPU.

$ vulkaninfo --summary
==========
VULKANINFO
==========

Vulkan Instance Version: 1.3.283


Instance Extensions: count = 22
-------------------------------
VK_EXT_acquire_drm_display             : extension revision 1
VK_EXT_acquire_xlib_display            : extension revision 1
VK_EXT_debug_report                    : extension revision 10
VK_EXT_debug_utils                     : extension revision 2
VK_EXT_direct_mode_display             : extension revision 1
VK_EXT_display_surface_counter         : extension revision 1
VK_EXT_surface_maintenance1            : extension revision 1
VK_EXT_swapchain_colorspace            : extension revision 4
VK_KHR_device_group_creation           : extension revision 1
VK_KHR_display                         : extension revision 23
VK_KHR_external_fence_capabilities     : extension revision 1
VK_KHR_external_memory_capabilities    : extension revision 1
VK_KHR_external_semaphore_capabilities : extension revision 1
VK_KHR_get_display_properties2         : extension revision 1
VK_KHR_get_physical_device_properties2 : extension revision 2
VK_KHR_get_surface_capabilities2       : extension revision 1
VK_KHR_portability_enumeration         : extension revision 1
VK_KHR_surface                         : extension revision 25
VK_KHR_surface_protected_capabilities  : extension revision 1
VK_KHR_xcb_surface                     : extension revision 6
VK_KHR_xlib_surface                    : extension revision 6
VK_LUNARG_direct_driver_loading        : extension revision 1

Instance Layers: count = 5
--------------------------
VK_LAYER_NV_optimus               NVIDIA Optimus layer         1.3.277  version 1
VK_LAYER_VALVE_steam_fossilize_32 Steam Pipeline Caching Layer 1.3.207  version 1
VK_LAYER_VALVE_steam_fossilize_64 Steam Pipeline Caching Layer 1.3.207  version 1
VK_LAYER_VALVE_steam_overlay_32   Steam Overlay Layer          1.3.207  version 1
VK_LAYER_VALVE_steam_overlay_64   Steam Overlay Layer          1.3.207  version 1

Devices:
========
GPU0:
	apiVersion         = 1.3.277
	driverVersion      = 550.100.0.0
	vendorID           = 0x10de
	deviceID           = 0x2489
	deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
	deviceName         = NVIDIA GeForce RTX 3060 Ti
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 550.100
	conformanceVersion = 1.3.7.2
	deviceUUID         = cc3fa268-7693-8118-6220-0439c07a8f08
	driverUUID         = 63390c76-60ef-5529-a6ac-99167eb42a9c

[qhaas]

I recently was encountering similar problems (see my reply to issue 16 where I list details of my environment) for a headless (i.e. no wayland, no x11) vulkan application running in my organization's internal (openstack powered) cloud. Based on the snippets above, I assume some of you, like me, are not using a GUI either. Here is what I learned:

• --runtime=nvidia should be combined with --gpus all in order for the vulkan ICDs to be mounted from the host with recent versions of the nvidia-container-toolkit. Without it, you can see from docker inspect -f '{{ .HostConfig.Runtime }}' that runc is being used instead of the nvidia runtime. For some reason, this isn't needed for nvidia-smi to work, possibly due to hooks(?)
• For headless, we should use EGL instead of GLX implementation since there is no X11
• The stack deployed by the vulkan-tools package in Ubuntu 22.04 only recognizes the deprecated VK_ICD_FILENAMES environment variable when setting paths to an ICD
• The path to the needed ICD (/usr/share/glvnd/egl_vendor.d/10_nvidia.json) isn't automatically found by this version's vulkan loader
• XDG_RUNTIME_DIR and DISPLAY errors can be ignored because we are not using X11
• The non-default graphics capability is needed
• You do not need the cuda or the (abandoned?) vulkan nvidia container images. You can get this functionality out of the base Ubuntu 22.04 docker image (and likely others) by installing the (equivalent) vulkan-tools package and setting environment variables at container launch needed by nvidia-container-toolkit (NVIDIA_DRIVER_CAPABILITIES) and the vulkan loader (VK_ICD_FILENAMES).

Example Dockerfile using a headless, third-party sample program that will render a ppm file on the gpu using vulkan as a non-root, limited user (luser). The use of a non-root user is a preference; not required. If you are quick, you can see /home/luser/bin/renderheadless running on the gpu via nvidia-smi on the host:

$ cat Dockerfile
FROM ubuntu:22.04 AS vulkan-sample-dev

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y gcc g++ make cmake libvulkan-dev libglm-dev curl unzip && apt-get clean

RUN useradd luser
USER luser
WORKDIR /home/luser
RUN curl -L -o master.zip https://github.com/SaschaWillems/Vulkan/archive/refs/heads/master.zip && unzip master.zip && rm master.zip
RUN cmake -DUSE_HEADLESS=ON Vulkan-master && \
    make renderheadless

FROM ubuntu:22.04 AS vulkan-sample-run

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y vulkan-tools && apt-get clean

ENV VK_ICD_FILENAMES=/usr/share/glvnd/egl_vendor.d/10_nvidia.json 
ENV NVIDIA_DRIVER_CAPABILITIES=graphics

RUN useradd luser
COPY --chown=luser:luser --from=vulkan-sample-dev /home/luser/bin/renderheadless /home/luser/bin/renderheadless
COPY --chown=luser:luser --from=vulkan-sample-dev /home/luser/Vulkan-master/shaders/glsl/renderheadless/ /home/luser/Vulkan-master/shaders/glsl/renderheadless/
USER luser
WORKDIR /home/luser
CMD [ "/home/luser/bin/renderheadless" ]

$ docker build --target vulkan-sample-run -t localhost:vulkan-sample-run .
...
$ docker run --runtime=nvidia --gpus all --name vulkan localhost:vulkan-sample-run
Running headless rendering example
GPU: NVIDIA GeForce GTX 1650 with Max-Q Design
Framebuffer image saved to headless.ppm
Finished. Press enter to terminate...
...
$ docker cp vulkan:/home/luser/headless.ppm .
...
$ eog headless.ppm
...
$ docker run --runtime=nvidia --gpus all --rm localhost:vulkan-sample-run vulkaninfo --summary | grep deviceName
...
	deviceName         = NVIDIA GeForce GTX 1650 with Max-Q Design

[papaj-na-wrotkach]

@qhaas I tried building your code, but…

$ docker run --runtime=nvidia --gpus all --name vulkan localhost:vulkan-sample-run
renderheadless: /home/luser/Vulkan-master/examples/renderheadless/renderheadless.cpp:238: VulkanExample::VulkanExample(): Assertion `res == VK_SUCCESS' failed.

$ docker run --runtime=nvidia --gpus all --rm localhost:vulkan-sample-run vulkaninfo --summary | grep deviceName
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libEGL_nvidia.so.0
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at ./vulkaninfo/vulkaninfo.h:649:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

Update:
It works when I change NVIDIA_DRIVER_CAPABILITIES to compute,graphics or simply all