[Support]: TensorRT image missing CUDA dependencies for ONNX ("Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: No such file or directory")

[reidprichard]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.
• I have asked the AI at https://docs.frigate.video about my issue.

Describe the problem you are having

The ONNX detector won't start with the tensorRT image, producing the error "Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: No such file or directory".

Steps to reproduce

1. Set image to TensorRT
2. Add GPU to config
3. Set detector and model appropriately for ONNX in Frigate config.yaml
4. Start Frigate

Version

0.16.2-4d58206

In which browser(s) are you experiencing the issue with?

n/a

Frigate config file

detectors:
  onnx:
    type: onnx
  
model:
  path: plus://a688a8f1733544be72cf4a13dce6d582 # onnx yolov9t

docker-compose file or Docker CLI command

services:
  frigate:
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1 # number of GPUs
              capabilities: [gpu]
    container_name: frigate
    restart: unless-stopped
    stop_grace_period: 30s # allow enough time to shut down the various services
    shm_size: "512mb" # update for your cameras based on calculation above
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config/
      - ./data:/database
      - /mnt/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "1984:1984"
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      - "PLUS_API_KEY=xxxx"
      - USE_FP16=true

Relevant Frigate log output

frigate  | 2025-11-23 12:39:02.764734004  [2025-11-23 12:39:02] frigate.util.config            INFO    : Checking if frigate config needs migration...
frigate  | 2025-11-23 12:39:02.905878713  [2025-11-23 12:39:02] frigate.util.config            INFO    : frigate config does not need migration...
frigate  | 2025-11-23 12:39:03.158093783  [2025-11-23 12:39:03] frigate.app                    INFO    : Starting Frigate (0.16.2-4d58206)
frigate  | 2025-11-23 12:39:03.494486284  [2025-11-23 12:39:03] peewee_migrate.logs            INFO    : Starting migrations
frigate  | 2025-11-23 12:39:03.497617131  [2025-11-23 12:39:03] peewee_migrate.logs            INFO    : There is nothing to migrate
frigate  | 2025-11-23 12:39:03.536400742  [2025-11-23 12:39:03] frigate.app                    INFO    : Recording process started: 475
frigate  | 2025-11-23 12:39:03.550489104  [2025-11-23 12:39:03] frigate.app                    INFO    : Review process started: 478
frigate  | 2025-11-23 12:39:03.559171618  [2025-11-23 12:39:03] frigate.app                    INFO    : go2rtc process pid: 128
frigate  | 2025-11-23 12:39:03.595945450  [2025-11-23 12:39:03] detector.onnx                  INFO    : Starting detection process: 500
frigate  | 2025-11-23 12:39:03.603844072  2025-11-23 12:39:03.603720233 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 508, index: 0, mask: {2, 34, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.604329448  2025-11-23 12:39:03.604244532 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 514, index: 4, mask: {10, 42, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.604523560  2025-11-23 12:39:03.604402618 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 515, index: 5, mask: {12, 44, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.604535167  2025-11-23 12:39:03.604465251 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 516, index: 6, mask: {14, 46, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.604744844  2025-11-23 12:39:03.604675034 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 518, index: 8, mask: {18, 50, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.608016291  2025-11-23 12:39:03.605133517 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 521, index: 11, mask: {24, 56, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.608027358  2025-11-23 12:39:03.605328417 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 520, index: 10, mask: {22, 54, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.608032271  [2025-11-23 12:39:03] frigate.app                    INFO    : Embedding process started: 502
frigate  | 2025-11-23 12:39:03.608041044  2025-11-23 12:39:03.605728299 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 519, index: 9, mask: {20, 52, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.608082178  2025-11-23 12:39:03.607694939 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 527, index: 17, mask: {5, 37, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.612069817  2025-11-23 12:39:03.612003141 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 531, index: 18, mask: {7, 39, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.617116748  2025-11-23 12:39:03.614096254 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 535, index: 19, mask: {9, 41, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.622133113  [2025-11-23 12:39:03] frigate.detectors.plugins.onnx INFO    : ONNX: loaded onnxruntime module
frigate  | 2025-11-23 12:39:03.622144056  2025-11-23 12:39:03.621991243 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 537, index: 21, mask: {13, 45, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.626418725  [2025-11-23 12:39:03] frigate.detectors.plugins.onnx INFO    : ONNX: loading /config/model_cache/a688a8f1733544be72cf4a13dce6d582
frigate  | 2025-11-23 12:39:03.630936560  2025-11-23 12:39:03.627979603 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 538, index: 22, mask: {15, 47, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.639934054  2025-11-23 12:39:03.637008744 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 542, index: 26, mask: {23, 55, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.642927412  2025-11-23 12:39:03.640253248 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 543, index: 27, mask: {25, 57, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.646930505  2025-11-23 12:39:03.643209352 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 536, index: 20, mask: {11, 43, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.649926223  2025-11-23 12:39:03.647085796 [E:onnxruntime:Default, env.cc:234 ThreadMain] pthread_setaffinity_np failed for thread: 544, index: 28, mask: {27, 59, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.
frigate  | 2025-11-23 12:39:03.692995013  [2025-11-23 12:39:03] frigate.app                    INFO    : Output process started: 555
frigate  | 2025-11-23 12:39:03.705608530  2025-11-23 12:39:03.705476642 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
frigate  | 2025-11-23 12:39:03.705617347
frigate  | 2025-11-23 12:39:03.705636280  2025-11-23 12:39:03.705547641 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:993 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Require cuDNN 9.* and CUDA 12.*. Please install all dependencies as mentioned in the GPU requirements page (https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

Relevant go2rtc log output

n/a

Operating system

Debian

Install method

Docker Compose

Network connection

Wired

Camera make and model

n/a

Screenshots of the Frigate UI's System metrics pages

Note: these are with my detector reverted to the USB coral since the container shuts down when the ONNX fails to start.

Any other information that may be helpful

CUDA, cuDNN, CUBLAS, etc. are all installed on the host, and the NVIDIA Container Toolkit is installed and set up. The GPU works for decoding video, and I can launch ONNX on the GPU on the host by this trivial example (after installing onnxruntime-gpu in a venv):

import onnxruntime as ort
model_path = '/root/frigate/config/model_cache/a688a8f1733544be72cf4a13dce6d582'
session = ort.InferenceSession(model_path, providers=['CUDAExecutionProvider'])

The purportedly missing file, libcublasLt.so.12, does not seem to exist in the container:

docker exec -it frigate /bin/bash
find / -name libcublasLt.so.12 2>/dev/null

It does, however, exist on the host (I don't get how containers share resources with the host but I don't think this is relevant).

It's a fairly old GPU, a Tesla P100 (Pascal generation) so that could be playing into it. However, the error says that CUDA 12.x and cuDNN 9.x are required, and the P100 supports both. CUDA 12.9.1 and cuDNN 9.16.0.29 are installed on the host.

[inkeep-ai-assistant[bot]]

The error you're experiencing is related to missing CUDA libraries in the TensorRT Docker image. The -tensorrt image is designed to automatically detect and use TensorRT with ONNX when an NVIDIA GPU is available(1).

Key Issues

Your configuration is missing the --gpus=all flag in your Docker Compose setup(2). This flag is essential for proper GPU access by the container.

Solution

Update your Docker Compose configuration to include the --gpus=all flag. Here's the corrected configuration:

services:
  frigate:
    image: ghcr.io/blakeblackshear/frigate:stable-tensorrt
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    container_name: frigate
    restart: unless-stopped
    stop_grace_period: 30s
    shm_size: "512mb"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config/
      - ./data:/database
      - /mnt/frigate:/media/frigate
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "1984:1984"
      - "8971:8971"
      - "5000:5000"
      - "8554:8554"
      - "8555:8555/tcp"
      - "8555:8555/udp"
    environment:
      - "PLUS_API_KEY=xxxx"
      - USE_FP16=true
      - NVIDIA_VISIBLE_DEVICES=all

(2)

Additional Configuration Notes

For ONNX with NVIDIA GPU support using the -tensorrt image, TensorRT will automatically be detected and used as a detector when a supported ONNX model is configured(1).

Your detector configuration should work as-is once the GPU access is properly configured:

detectors:
  onnx:
    type: onnx
  
model:
  path: plus://a688a8f1733544be72cf4a13dce6d582

(1)

Compatibility Check

Your Tesla P100 (Pascal generation) should be compatible. The minimum requirement for the TensorRT detector is a GPU supporting Compute Capability of 5.0 or greater, and the minimum driver version on the host must be >=545(3). The P100 has Compute Capability 6.0, so it meets this requirement.

After making these changes, restart your container and the CUDA libraries should be properly accessible to ONNX Runtime.

📚 Sources:

1. Supported Hardware
2. [Detector Support]: Updated to 16.0 and did all the convertion from nvidia tensorrt to onnx and for some reason cuda is not working.
3. Recommended hardware

---

Have another question? Just tag @inkeep.

[reidprichard]

I'm fairly sure NVIDIA_VISIBLE_DEVICES=all (the only change I notice above) is irrelevant?

[hawkeye217]

Are you using Proxmox?

[NickM-27]

Not sure, it seems like it is using the correct image

[reidprichard]

I'm noticing something very odd. If I run:

docker compose down
docker system prune -a -f
docker pull ghcr.io/blakeblackshear/frigate:stable-tensorrt

It pulls in < 1s without the usual printout of the different hunks and their progress. On the other hand, docker pull ghcr.io/blakeblackshear/frigate:stable takes awhile and prints out the download process.

Is it possible something is screwed up with Docker itself regarding this image?

[NickM-27]

Something might be incorrectly cached, if the image is truly deleted then it would take longer than the main image to pull since it is larger

[reidprichard]

Okay, I finally cracked it. No idea why this was necessary, but (obviously after a cautionary system snapshot):

$ systemctl stop containerd
$ rm -rf /var/lib/containerd/*
$ systemctl start containers

Appreciate the help!

[NickM-27]

We usually only see this when an installation takes over some preset values. You can try overwriting it with LD_LIBRARY_PATH but this is quite well tested so we know it works correctly

[reidprichard]

Thanks. I may just be obtuse, but I'm not sure what you mean by 'preset values' - I thought the point of containerization was to be agnostic of environment? Nonetheless, this container is for Frigate and nothing else, so its environment is quite clean:

root@frigate:~/frigate$ env
SHELL=/bin/bash
COLORTERM=truecolor
MAN_POSIXLY_CORRECT=1
EDITOR=/usr/bin/nvim
PWD=/root/frigate
LOGNAME=root
XDG_SESSION_TYPE=tty
MOTD_SHOWN=pam
HOME=/root
LANG=C
GROFF_NO_SGR=1
XDG_SESSION_CLASS=user
TERM=xterm-256color
USER=root
MANPAGER=nvim +Man! --cmd "map d <C-D>" --cmd "map u <C-U>"
SHLVL=1
XDG_SESSION_ID=c21
LD_LIBRARY_PATH=/usr/local/cuda-12.9/targets/x86_64-linux/lib/
XDG_RUNTIME_DIR=/run/user/0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.local/bin/
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
SSH_TTY=/dev/pts/3
OLDPWD=/root
_=/usr/bin/env

(minus some TERMCAP/SSH vars I'm certain were irrelevant)

I tried the following:

export LD_LIBRARY_PATH=/usr/local/cuda-12.9/targets/x86_64-linux/lib/
docker compose down
docker compose up -d

And still nothing. Again, perhaps I am being dumb, but I don't understand how LD_LIBRARY_PATH would affect the Docker container. I'm looking through the tensorrt dockerfile, and it seems like the nvidia libs are installed via pip inside the container?

[NickM-27]

Frigate doesn't use CUDA 12.9 so that's not going to work. You need to use find in the container to see what the correct path is