Don't use nvidia-tensorrt. Instead specify version with tensorrt==8.5.3.1

[carsonhh]

The run-prereq.sh script used to build the image calls 'pip3 install "${PIP_ARGS[@]}" nvidia-tensorrt' on line 273

The nvidia-tensorrt package should not be used. It now just points to whatever the latest tensorrt version is
NVIDIA/TensorRT#2668

The result is that depending on when you build, you may be actually downloading different versions. The result is that the latest version requires Cuda 12 but the image has Cuda 11. This creates conflicts and is likely behind the singularity GPU issues coming up on the issues page.

Instead you should specifically specify tensorrt==8.5.3.1 to get the cuda 11 version.

[pichuan]

Thank you @carsonhh ! I'll test this and make a change accordingly.

[carsonhh]

Note you will also have to revert a couple of lines referred to in run-preset.sh to match the tensorrt version

This is the text in run-preset.sh, you should probably revert to the way it was in 1.5.0 for this section:

  # In v8.6.1, the libs got moved to tensorrt_libs:
  # https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#rel-8-6-1
  sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer.so.8" "${TENSORRT_PATH}_libs/libnvinfer.so.7"
  sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.8" "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.7"

[carsonhh]

Here is my diff for run-preset.sh edits:

274c274
<   pip3 install "${PIP_ARGS[@]}" tensorrt==8.5.3.1
---
>   pip3 install "${PIP_ARGS[@]}" nvidia-tensorrt
276c276
<   pip3 show tensorrt
---
>   pip3 show nvidia-tensorrt
278,285c278,282
<   ## In v8.6.1, the libs got moved to tensorrt_libs:
<   ## https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#rel-8-6-1
<   ##sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer.so.8" "${TENSORRT_PATH}_libs/libnvinfer.so.7"
<   ##sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.8" "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.7"
<   ##export LD_LIBRARY_PATH="${LD_LIBRARY_PATH-}:${TENSORRT_PATH}_libs"
<   sudo ln -sf "${TENSORRT_PATH}/libnvinfer.so.8" "${TENSORRT_PATH}/libnvinfer.so.7"
<   sudo ln -sf "${TENSORRT_PATH}/libnvinfer_plugin.so.8" "${TENSORRT_PATH}/libnvinfer_plugin.so.7"
<   export LD_LIBRARY_PATH="${LD_LIBRARY_PATH-}:${TENSORRT_PATH}"
---
>   # In v8.6.1, the libs got moved to tensorrt_libs:
>   # https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#rel-8-6-1
>   sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer.so.8" "${TENSORRT_PATH}_libs/libnvinfer.so.7"
>   sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.8" "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.7"
>   export LD_LIBRARY_PATH="${LD_LIBRARY_PATH-}:${TENSORRT_PATH}_libs"
292,295c289,290
<     ##sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer.so.7" /usr/local/nvidia/lib/libnvinfer.so.7
<     ##sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.7" /usr/local/nvidia/lib/libnvinfer_plugin.so.7
<     sudo ln -sf "${TENSORRT_PATH}/libnvinfer.so.7" /usr/local/nvidia/lib/libnvinfer.so.7
<     sudo ln -sf "${TENSORRT_PATH}/libnvinfer_plugin.so.7" /usr/local/nvidia/lib/libnvinfer_plugin.so.7
---
>     sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer.so.7" /usr/local/nvidia/lib/libnvinfer.so.7
>     sudo ln -sf "${TENSORRT_PATH}_libs/libnvinfer_plugin.so.7" /usr/local/nvidia/lib/libnvinfer_plugin.so.7

[pichuan]

Thanks @carsonhh .
I'll plan to fix this before our next release (1.7.0). We have a small fix recently (1.6.1) which won't have this updated yet.

For the 1.7.0 release, I'm also working on updating Python version to 3.10 (and use Ubuntu 22.04 as base image). I'll try your change with those changes and see if it works!

I have filed an internal issue to track, so I'll close this. Thanks again for your suggestion!

[pichuan]

Hi @carsonhh , I'm going through internal issues that I have not finished. It seems like this one has slipped my mind.
I'll plan to take a look again.

If you actually have an updated suggestion (now that we're at 1.8), let me know. I'll try with your last suggestion, and I'll report back. Just to make sure I won't forget this time, I'll reopen this issue on GitHub as well.

[pichuan]

Hi @carsonhh ,
I have tried the changes you suggested, and it worked!
After your change, I no longer see this warning message in the log: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT, which is nice!

This change is already submitted in our internal codebase, and it will show up in the next release. Thank you for your feedback to make DeepVariant codebase better!