nvidia-container-cli tries to open libnvidia-tls.so.575.64.03 while only libnvidia-tls.so.575.64.05 is available

[Laflamme]

Describe the bug

I did a couple hours of research and this seems to be a version mismatch that I cannot solve as an end user of Bluefin (if could actually solve this issue myself, any pointers would be greatly appreciated).

On the following deployment:

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia-open:stable
                   Digest: sha256:c086f08ad00343d3f319a8e5f424160aae827144226669bd42530e81391a0d34
                  Version: 42.20250727 (2025-07-27T19:45:24Z)

I get the following error when I try to deploy containers that use nvidia-container-cli:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: detection error: open failed: /usr/lib64/libnvidia-tls.so.575.64.03: no such file or directory: unknown

There is indeed no /usr/lib64/libnvidia-tls.so.575.64.03, but there is a /usr/lib64/libnvidia-tls.so.575.64.05.

Here is the version information for nvidia-ctk:

VERSION:
   1.17.8
commit: f202b80a9b9d0db00d9b1d73c0128c8962c55f4d

What did you expect to happen?

I would expect nvidia-container-cli to open libnvidia-tls.so.575.64.05.

Output of bootc status

● Booted image: ghcr.io/ublue-os/bluefin-dx-nvidia-open:stable
        Digest: sha256:c086f08ad00343d3f319a8e5f424160aae827144226669bd42530e81391a0d34 (amd64)
       Version: 42.20250727 (2025-07-27T19:45:24Z)

  Rollback image: ghcr.io/ublue-os/bluefin-dx-nvidia-open:stable
          Digest: sha256:733f0537510a32ab5a7180583e35b185a1d2fdfba2bfb19304fcdd92acaf0f5a (amd64)
         Version: 42.20250715 (2025-07-15T00:31:34Z)

Output of groups

my_user_name wheel docker incus-admin libvirt

Extra information or context

No response

[dosubot]

This error is caused by a version mismatch: nvidia-container-cli is looking for libnvidia-tls.so.575.64.03, but only .05 is present. Bluefin relies on upstream NVIDIA packaging and does not currently provide an automated fix or symlink for this kind of mismatch, so it can happen if the container runtime and host libraries get out of sync (for example, after a driver update) source.

First, try updating to the latest Bluefin image using ujust rebase-helper—this often resolves version mismatches if the latest image and drivers are in sync.

If the problem persists, you can manually create a symlink as a workaround (replace .03 with the missing version and .05 with the available one):

sudo ln -s /usr/lib64/libnvidia-tls.so.575.64.05 /usr/lib64/libnvidia-tls.so.575.64.03

This is a workaround and may not be officially supported—remove the symlink if you notice any instability.

For best practices and troubleshooting, see the official Bluefin NVIDIA container documentation:
https://docs.projectbluefin.io/hardware/nvidia/containers/

tldr tip:
To check which NVIDIA TLS libraries are present, run:

ls -l /usr/lib64/libnvidia-tls.so*

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[Laflamme]

@dosu Isn't /usr read-only?

[Laflamme]

@dosu Also, https://docs.projectbluefin.io/hardware/nvidia/containers/ is a broken link.

[dosubot]

That link is broken—there isn’t a dedicated Bluefin NVIDIA container documentation page at this time. The most relevant info is in the administration docs, which mention using the correct -nvidia image variant. For container runtime setup and troubleshooting, the best reference is the official NVIDIA Container Toolkit guide and this Bluefin discussion on NVIDIA containers.

On Bluefin, /usr is read-only, so manual symlink workarounds aren’t supported or persistent discussion. Version mismatches like the one you’re seeing are a known issue—keeping your system updated and using the correct image variant is the recommended approach. If you need more help, let me know!

You can always find the latest official docs at https://docs.projectbluefin.io.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[tulilirockz]

I'm personally not very familiar with nvidia-container-toolkit at all, but you can test if using sudo bootc usroverlay and following Dosu's recommendation works

[Laflamme]

Thank you very much tulilirockz; using sudo bootc usroverlay and creating all necessary symlinks does solve the issue temporarily.

I am new to Bluefin: would someone familiar with nvidia-container-toolkit be able to tell me if this kind of version mismatch is a regular issue, or if this is exceptional and just bad timing on my part?

If it is a regular issue, could the official deployment automatically generate symlinks to the previous version as a failsafe?

[Laflamme]

This issue is resolved in 42.20250803.2.

[Laflamme]

I'm having this issue again with today's release and I really would like to know how other users who rely on nvidia-container-toolkit deal with this.

Do you have a shell script to automate the symlink workaround (there are 10-15 symlinks to create afte a reboot)?

Do you just run docker inside a VM?

Is there something about upstream, unrelated to Bluefin's design, that I'm missing and should be investigating? I didn't encounter this issue using Pop!_OS, so is it something about the Fedora base?

@dosu please feel free to chime in!

[castrojo]

Yeah sorry, not a lot of nvidia experience on the team, but maybe we can get some visibility here.

[dosubot]

Most Bluefin users work around NVIDIA container toolkit version mismatches by temporarily making /usr writable with sudo bootc usroverlay and manually creating symlinks for the missing libraries. This is a known issue on Fedora-based systems, including Bluefin, because driver and container toolkit updates can get out of sync—unlike Pop!_OS, which uses its own NVIDIA packaging and tends to keep these components in sync, so you’re less likely to see this problem there.

There’s no official or community-provided script for automating symlink creation, but you can use something like this after enabling the overlay:

sudo bootc usroverlay
for lib in /usr/lib64/libnvidia-*.so.*.*.*; do
  base=$(basename "$lib")
  # Example: create symlinks for all .so.*.*.* to the latest version
  latest=$(ls /usr/lib64/$base | sort -V | tail -n1)
  for missing in $(seq 1 15); do
    ver=$(echo "$latest" | sed "s/[0-9]*$/$(printf '%02d' $missing)/")
    if [ ! -e "/usr/lib64/$ver" ]; then
      sudo ln -s "$latest" "/usr/lib64/$ver"
    fi
  done
done

You’ll want to adjust the script to match the actual versions and libraries you need—run ls /usr/lib64/libnvidia-* to see what’s present.

Some users do run Docker inside a VM as a workaround for persistent NVIDIA driver issues, but this is less common and usually a last resort for stability reasons example.

Best practice: keep your system and images updated with ujust rebase-helper, always use the correct Bluefin NVIDIA image variant, and monitor upstream changes. If you hit this often, it’s worth watching this discussion for updates.

Official docs: https://docs.projectbluefin.io

tldr tip:
To list all NVIDIA libraries and spot mismatches, run:

ls -l /usr/lib64/libnvidia-*

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[Laflamme]

Hey @dosu, based on Bluefin's current design, if its developers wanted to implement a solution to this problem, where could they start? The solution should be fully automated and minimally disruptive. It is safe to assume it should only apply to the Nvidia variant of bluefin-dx.

Keep in mind that "if the team has to make a thing, it's because there is no choice, and if it does have to make it, it's not happy about it."

[dosubot]

The most practical starting point for a fully automated, minimally disruptive fix is to add a post-boot systemd unit or script that runs only on the Nvidia variant of bluefin-dx. This unit would scan for missing expected NVIDIA library versions (like libnvidia-tls.so.*) and create symlinks to the latest available version if needed. This approach avoids modifying the immutable image at build time and keeps the workaround isolated to affected systems, minimizing risk and maintenance overhead. There’s no existing automation for this in Bluefin—current scripts only handle specific symlinks (like libnvidia-ml.so) and do not address broad version mismatches source.

The logic would need to run after the driver payloads land, check for missing .so files, and create symlinks as needed—ideally gated to the Nvidia variant to avoid side effects. This would be new work, not an extension of current automation source.

tldr tip:
To check for mismatches, run:

ls -l /usr/lib64/libnvidia-*

For more context, see this discussion and the official docs.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.