RuntimeError: Tensorflow has not been built with TensorRT Support

[Raghu-dev-pixel]

Hello,
I was trying out tensorflow tensorRT on the Google Colab to get some hands-on experience with tensorRT , but the code suggested in the repository is not working.

https://github.com/tensorflow/tensorrt/blob/master/tftrt/benchmarking-python/image_classification/Colab-TFv2-TF-TRT-inference-from-Keras-saved-model.ipynb

https://github.com/tensorflow/tensorrt/blob/master/tftrt/benchmarking-python/image_classification/NGC-TFv2-TF-TRT-inference-from-Keras-saved-model.ipynb

The above two git repositories show how to use a tensorflow TensorRT on a Google Colab, but this was working with tensorflow version 2.0.0, however with later versions of tensorflow 2.1.. further it is not working and throwing a run time error message saying tensorflow has not been built with tensorRT support. There has been an issue raised in the developer forums regarding this issue.

https://forums.developer.nvidia.com/t/runtimeerror-tensorflow-has-not-been-built-with-tensorrt-support/238124/9

I tried to run the tensorflow tensorRT on Google Colab and it did work after some small changes on the tensor flow 2.17.0 . I wanted to post it as part of the tensorRT open-source and documentation, I asked for the same in the above developer forums and it is been more than 10 days since I did not get an update, and now there is a new version of tensorflow (2.18.0) and after which my code with 2.17.0 tensorflow is not working, but when I revert to tensorflow 2.17.0 on the notebook, tensorRT works. Ideally, there is some compatibility issue and I have also seen some people on stack overflow trying different approaches with tensorflow 2.1 + series but don't seem to have a successful solution, at least I got it working with tensorflow 2.17.0 .

//This was the code changes that I made when I ran Tensorflow-TRT on google colab with tensorflow version 2.17.

!sudo apt-get update
!sudo apt-get install -y libnvinfer8 libnvinfer-plugin8

I removed the below code !!

!pip install tensorflow-gpu==2.0.0
%%bash
wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb

dpkg -i nvidia-machine-learning-repo-*.deb
apt-get update

sudo apt-get install libnvinfer5
//=================================================================

Ideally it should have worked as in the repo "https://github.com/tensorflow/tensorrt/blob/master/tftrt/benchmarking-python/image_classification/NGC-TFv2-TF-TRT-inference-from-Keras-saved-model.ipynb" but it did not work and still got a runtime error, so I had to make the changes as I mentioned above.

I wanted to suggest these changes and make it as a documentation for the open-source if that was an option, but now we have a new version of tensorflow 2.18 on google colab which should have worked with

!sudo apt-get update
!sudo apt-get install -y libnvinfer10 libnvinfer-plugin10

But it is not working and getting a runtime error.

I believe since tensorRT is widely used, this solution might help a lot of people trying out tensorflow TRT.

Thanks in advance!!

Thanks and Regards,
Raghu

[Raghu-dev-pixel]

Hello,
I found out that in Tensorflow 2.18.0, the TensorRT is not supported.

https://newreleases.io/project/github/tensorflow/tensorflow/release/v2.18.0-rc2

So the latest tensorflow 2.17.0 comes with TensorRT support, however, the code mentioned in the above two git repositories does not work and requires some code changes as I have mentioned in my previous comment. I have this in my local git repository and would like to document it here if possible.

Thanks and Regards,
Raghu