Discoverability issue during CDI spec generation when the program lives inside a snap container

[gabrielmougard]

I'm trying to generate a CDI specification in a program living inside a snap container (see canonical/lxd#13562)

When generating a CDI spec from inside a snap package, we need to tell the CDI discoverability process to start
looking after the host root file system which is defined inside the snap by /var/lib/snapd/hostfs (a.k.a, this is the same as / if we are not in a snap)

In consequence, here is how I created my cdilib object:

package cdi

import (
	"fmt"
	"os/exec"
	"path/filepath"

	"github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi"
	"github.com/canonical/lxd/shared"
	cdpiSpec "tags.cncf.io/container-device-interface/pkg/cdi"
	"tags.cncf.io/container-device-interface/specs-go"
)

const (
	// defaultNvidiaTegraMountSpecPath is default location of CSV files that define the modifications required to the OCI spec
	defaultNvidiaTegraMountSpecPath = "/etc/nvidia-container-runtime/host-files-for-container.d"
)

// DefaultNvidiaTegraCSVFiles returns the default CSV files for the Nvidia Tegra platform.
func DefaultNvidiaTegraCSVFiles() []string {
	files := []string{
		"devices.csv",
		"drivers.csv",
		"l4t.csv",
	}

	var paths []string
	for _, file := range files {
		paths = append(paths, filepath.Join(defaultNvidiaTegraMountSpecPath, file))
	}

	return paths
}

// generateNvidiaSpec generates a CDI spec for an Nvidia vendor.
func generateNvidiaSpec(cdiID ID) (*specs.Spec, error) {
	mode := nvcdi.ModeAuto
	if cdiID.DeviceType() == IGPU {
		mode = nvcdi.ModeCSV
	}

	deviceNamer, err := nvcdi.NewDeviceNamer(nvcdi.DeviceNameStrategyIndex)
	if err != nil {
		return nil, fmt.Errorf("Failed to create device namer: %v", err)
	}

	deviceNamers := []nvcdi.DeviceNamer{deviceNamer}
	nvidiaCTKPath, err := exec.LookPath("nvidia-ctk")
	if err != nil {
		return nil, fmt.Errorf("Failed to find the nvidia-ctk binary: %v", err)
	}

	rootPath := ""
	if shared.InSnap() {
		rootPath = "/var/lib/snapd/hostfs"
	}

	cdilib, err := nvcdi.New(
		nvcdi.WithLogger(&CDILogger{}),
		nvcdi.WithDriverRoot(rootPath),
		nvcdi.WithDevRoot(rootPath),
		nvcdi.WithNVIDIACTKPath(nvidiaCTKPath),
		nvcdi.WithDeviceNamers(deviceNamers...),
		nvcdi.WithMode(mode),
		nvcdi.WithCSVFiles(DefaultNvidiaTegraCSVFiles()),
	)
	if err != nil {
		return nil, fmt.Errorf("Failed to create CDI library: %v", err)
	}

	deviceSpecs, err := cdilib.GetAllDeviceSpecs()
	if err != nil {
		return nil, fmt.Errorf("Failed to create device CDI specs: %v", err)
	}

	commonEdits, err := cdilib.GetCommonEdits()
	if err != nil {
		return nil, fmt.Errorf("Failed to create edits common for entities: %v", err)
	}

	return &specs.Spec{
		Version:        cdpiSpec.CurrentVersion,
		Kind:           fmt.Sprintf("%s/%s", cdiID.Vendor(), cdiID.Product()),
		Devices:        deviceSpecs,
		ContainerEdits: *commonEdits.ContainerEdits,
	}, nil
}

Here is the resulting log for the generation:

DEBUG  [2024-06-20T15:41:27Z] Is WSL-based system? false: could not load DXCore library: libdxcore.so: cannot open shared object file: No such file or directory 
DEBUG  [2024-06-20T15:41:27Z] Is NVML-based system? true: found NVML library 
INFO   [2024-06-20T15:41:27Z] Auto-detected mode as "nvml"                 
DEBUG  [2024-06-20T15:41:27Z] Is Tegra-based system? true: /sys/devices/soc0/family has 'tegra' prefix 
DEBUG  [2024-06-20T15:41:27Z] returning cached mounts                      
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia0                        
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia0" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia0' 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia0 as [/var/lib/snapd/hostfs/dev/nvidia0] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/dri/card1' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia0 as /dev/nvidia0 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/dri/card1 as /dev/dri/card1 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/dri/card1                      
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/dri/controlD65" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/dri/card1" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/dri/controlD65                 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/dri/card1 as [/var/lib/snapd/hostfs/dev/dri/card1] 
WARNING[2024-06-20T15:41:27Z] Could not locate /dev/dri/controlD65: pattern /dev/dri/controlD65 not found 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/dri/renderD128                 
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/dri/renderD128" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/dri/renderD128' 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/dri/renderD128 as [/var/lib/snapd/hostfs/dev/dri/renderD128] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/dri/renderD128 as /dev/dri/renderD128 
DEBUG  [2024-06-20T15:41:27Z] returning cached mounts                      
WARNING[2024-06-20T15:41:27Z] Failed to evaluate symlink /var/lib/snapd/hostfs/dev/dri/by-path/pci-0005:09:00.0-card; ignoring 
WARNING[2024-06-20T15:41:27Z] Failed to evaluate symlink /var/lib/snapd/hostfs/dev/dri/by-path/pci-0005:09:00.0-render; ignoring 
DEBUG  [2024-06-20T15:41:27Z] Opening ld.conf at /var/lib/snapd/hostfs/etc/ld.so.cache 
DEBUG  [2024-06-20T15:41:27Z] Locating "libcuda.so.*.*" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/usr/lib64 /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib64 /var/lib/snapd/hostfs/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib/aarch64-linux-gnu/nvidia/current] 
DEBUG  [2024-06-20T15:41:27Z] Opening ld.conf at /var/lib/snapd/hostfs/etc/ld.so.cache 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Using driver version 535.161.08              
DEBUG  [2024-06-20T15:41:27Z] Opening ld.conf at /var/lib/snapd/hostfs/etc/ld.so.cache 
DEBUG  [2024-06-20T15:41:27Z] Locating "*.so.535.161.08" in [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating "libcuda.so.535.161.08" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/usr/lib64 /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib64 /var/lib/snapd/hostfs/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib/aarch64-linux-gnu/nvidia/current] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia-modeset                 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia-modeset as /dev/nvidia-modeset 
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia-modeset" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia-modeset' 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia-modeset as [/var/lib/snapd/hostfs/dev/nvidia-modeset] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia-uvm' 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia-uvm-tools               
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia-uvm-tools" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia-uvm-tools as /dev/nvidia-uvm-tools 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia-uvm-tools as [/var/lib/snapd/hostfs/dev/nvidia-uvm-tools] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia-uvm-tools' 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia-uvm                     
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia-uvm" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidiactl as /dev/nvidiactl 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia-uvm as [/var/lib/snapd/hostfs/dev/nvidia-uvm] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia-uvm as /dev/nvidia-uvm 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidiactl                      
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidiactl" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidiactl as [/var/lib/snapd/hostfs/dev/nvidiactl] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidiactl' 
DEBUG  [2024-06-20T15:41:27Z] Locating libnvidia-egl-gbm.so.*              
DEBUG  [2024-06-20T15:41:27Z] Locating "libnvidia-egl-gbm.so.*" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/usr/lib64 /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib64 /var/lib/snapd/hostfs/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib/aarch64-linux-gnu/nvidia/current] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1' 
DEBUG  [2024-06-20T15:41:27Z] Located libnvidia-egl-gbm.so.* as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0] 
DEBUG  [2024-06-20T15:41:27Z] Locating glvnd/egl_vendor.d/10_nvidia.json   
DEBUG  [2024-06-20T15:41:27Z] Locating "glvnd/egl_vendor.d/10_nvidia.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Located glvnd/egl_vendor.d/10_nvidia.json as [/var/lib/snapd/hostfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json' 
DEBUG  [2024-06-20T15:41:27Z] Locating vulkan/icd.d/nvidia_icd.json        
DEBUG  [2024-06-20T15:41:27Z] Locating "vulkan/icd.d/nvidia_layers.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating "vulkan/icd.d/nvidia_icd.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating vulkan/icd.d/nvidia_layers.json     
DEBUG  [2024-06-20T15:41:27Z] Located vulkan/icd.d/nvidia_icd.json as [/var/lib/snapd/hostfs/usr/share/vulkan/icd.d/nvidia_icd.json] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/vulkan/icd.d/nvidia_icd.json as /usr/share/vulkan/icd.d/nvidia_icd.json 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/vulkan/icd.d/nvidia_icd.json' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json as /usr/share/glvnd/egl_vendor.d/10_nvidia.json 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0 as /usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0 
DEBUG  [2024-06-20T15:41:27Z] Locating "vulkan/implicit_layer.d/nvidia_layers.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating vulkan/implicit_layer.d/nvidia_layers.json 
WARNING[2024-06-20T15:41:27Z] Could not locate vulkan/icd.d/nvidia_layers.json: pattern vulkan/icd.d/nvidia_layers.json not found 
DEBUG  [2024-06-20T15:41:27Z] Located vulkan/implicit_layer.d/nvidia_layers.json as [/var/lib/snapd/hostfs/usr/share/vulkan/implicit_layer.d/nvidia_layers.json] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/vulkan/implicit_layer.d/nvidia_layers.json' 
DEBUG  [2024-06-20T15:41:27Z] Locating egl/egl_external_platform.d/15_nvidia_gbm.json 
DEBUG  [2024-06-20T15:41:27Z] Located egl/egl_external_platform.d/15_nvidia_gbm.json as [/var/lib/snapd/hostfs/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json] 
DEBUG  [2024-06-20T15:41:27Z] Locating "egl/egl_external_platform.d/10_nvidia_wayland.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating "egl/egl_external_platform.d/15_nvidia_gbm.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/libglxserver_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json' 
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/libglxserver_nvidia.so.535.161.08: pattern nvidia/xorg/libglxserver_nvidia.so.535.161.08 not found 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json as /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json 
DEBUG  [2024-06-20T15:41:27Z] Locating egl/egl_external_platform.d/10_nvidia_wayland.json 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia/nvoptix.bin as [/var/lib/snapd/hostfs/usr/share/nvidia/nvoptix.bin] 
WARNING[2024-06-20T15:41:27Z] Could not locate egl/egl_external_platform.d/10_nvidia_wayland.json: pattern egl/egl_external_platform.d/10_nvidia_wayland.json not found 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/nvoptix.bin                  
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/nvoptix.bin" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/nvidia/nvoptix.bin' 
DEBUG  [2024-06-20T15:41:27Z] Locating "X11/xorg.conf.d/10-nvidia.conf" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/nvidia/nvoptix.bin as /usr/share/nvidia/nvoptix.bin 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/nvidia_drv.so" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/nvidia_drv.so           
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/nvidia_drv.so: pattern nvidia/xorg/nvidia_drv.so not found 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/libglxserver_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating X11/xorg.conf.d/10-nvidia.conf      
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located X11/xorg.conf.d/10-nvidia.conf as [/var/lib/snapd/hostfs/usr/share/X11/xorg.conf.d/10-nvidia.conf] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/X11/xorg.conf.d/10-nvidia.conf' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/X11/xorg.conf.d/10-nvidia.conf as /usr/share/X11/xorg.conf.d/10-nvidia.conf 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/vulkan/implicit_layer.d/nvidia_layers.json as /usr/share/vulkan/implicit_layer.d/nvidia_layers.json 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08" in [/var/lib/snapd/hostfs] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/nvidia-fabricmanager/socket" in [/var/lib/snapd/hostfs/run /var/lib/snapd/hostfs/var/run] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08" in [/var/lib/snapd/hostfs] 
WARNING[2024-06-20T15:41:27Z] Could not locate /tmp/nvidia-mps: pattern /tmp/nvidia-mps not found 
DEBUG  [2024-06-20T15:41:27Z] Locating "/nvidia-persistenced/socket" in [/var/lib/snapd/hostfs/run /var/lib/snapd/hostfs/var/run] 
DEBUG  [2024-06-20T15:41:27Z] Locating /nvidia-persistenced/socket         
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08" in [/var/lib/snapd/hostfs] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/run/nvidia-persistenced/socket as /run/nvidia-persistenced/socket 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 
WARNING[2024-06-20T15:41:27Z] Could not locate /nvidia-fabricmanager/socket: pattern /nvidia-fabricmanager/socket not found 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /nvidia-fabricmanager/socket        
DEBUG  [2024-06-20T15:41:27Z] Locating "/tmp/nvidia-mps" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /tmp/nvidia-mps                     
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/run/nvidia-persistenced/socket' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Located /nvidia-persistenced/socket as [/var/lib/snapd/hostfs/run/nvidia-persistenced/socket] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/535.161.08/gsp*.bin          
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/535.161.08/gsp*.bin" in [/var/lib/snapd/hostfs/lib/firmware/updates/5.15.0-1012-nvidia-tegra-igx /var/lib/snapd/hostfs/lib/firmware/updates /var/lib/snapd/hostfs/lib/firmware/5.15.0-1012-nvidia-tegra-igx /var/lib/snapd/hostfs/lib/firmware] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_tu10x.bin' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_ga10x.bin' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_ga10x.bin as /lib/firmware/nvidia/535.161.08/gsp_ga10x.bin 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-smi" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_tu10x.bin as /lib/firmware/nvidia/535.161.08/gsp_tu10x.bin 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-smi                          
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia/535.161.08/gsp*.bin as [/var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_ga10x.bin /var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_tu10x.bin] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-smi' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-smi as [/var/lib/snapd/hostfs/usr/bin/nvidia-smi] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-debugdump                    
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-smi as /usr/bin/nvidia-smi 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-debugdump" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-debugdump' 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-debugdump as [/var/lib/snapd/hostfs/usr/bin/nvidia-debugdump] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-debugdump as /usr/bin/nvidia-debugdump 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-persistenced                 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-persistenced" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-persistenced as [/var/lib/snapd/hostfs/usr/bin/nvidia-persistenced] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-persistenced' 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-persistenced as /usr/bin/nvidia-persistenced 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-cuda-mps-control" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-cuda-mps-control as [/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-control] 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-cuda-mps-server              
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-control' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-control 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-server' 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-cuda-mps-control             
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-cuda-mps-server" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-cuda-mps-server as [/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-server] 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/nvidia_drv.so" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/nvidia_drv.so           
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/nvidia_drv.so: pattern nvidia/xorg/nvidia_drv.so not found 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/libglxserver_nvidia.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-server as /usr/bin/nvidia-cuda-mps-server 
DEBUG  [2024-06-20T15:41:27Z] returning cached mounts                      
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/libglxserver_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/libglxserver_nvidia.so.535.161.08: pattern nvidia/xorg/libglxserver_nvidia.so.535.161.08 not found

What is quite odd is the fact that the discoverability logic is looking for path starting by /var/lib/snapd/hostfs/var/lib/snapd/hostfs and not /var/lib/snapd/hostfs...
Is it a bug or is it because the cdilib object has not been well configured?

This actually leads to a problem where nvidia/xorg/libglxserver_nvidia.so.535.161.08 is not generated in the output (missing mount entry + hook entry)
The logic look for the pattern inside /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu and /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu.
If it was looking in /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu, then it would have worked.

What do you think @elezar ? /cc @tomponline

[elezar]

This is a bug in the "discovery" code for the graphics libraries as implemented here:

nvidia-container-toolkit/internal/discover/graphics.go

Lines 268 to 281 in ddeeca3

	libRoot := filepath.Dir(libcudaPath)
	xorgLibs := NewMounts(
	logger,
	lookup.NewFileLocator(
	lookup.WithLogger(logger),
	lookup.WithRoot(driver.Root),
	lookup.WithSearchPaths(libRoot, "/usr/lib/x86_64-linux-gnu"),
	lookup.WithCount(1),
	),
	driver.Root,
	[]string{
	"nvidia/xorg/nvidia_drv.so",
	fmt.Sprintf("nvidia/xorg/libglxserver_nvidia.so.%s", version),
	},

:

	libRoot := filepath.Dir(libcudaPath)
	xorgLibs := NewMounts(
		logger,
		lookup.NewFileLocator(
			lookup.WithLogger(logger),
			lookup.WithRoot(driver.Root),
			lookup.WithSearchPaths(libRoot, "/usr/lib/x86_64-linux-gnu"),
			lookup.WithCount(1),
		),
		driver.Root,
		[]string{
			"nvidia/xorg/nvidia_drv.so",
			fmt.Sprintf("nvidia/xorg/libglxserver_nvidia.so.%s", version),
		},
	)

The issue in this case is that the libRoot as /var/lib/snapd/hostfs/lib/aarch64-linux-gnu and this is prepended with driver.Root (/var/lib/snapd/hostfs) to locate libglxserver_nvidia.so. Note that we allow for /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu to be searched, but not /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu.

#548 reworks this somewhat, but from your description it seems that there is still a bug that would surface in your case.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[tomponline]

Still an issue