bevformer-int8-eq

About

This repository contains an end-to-end example of deploying BEVFormer with explicit quantization with NVIDIA's ModelOpt Toolkit. At the end, we show TensorRT deployment results in terms of runtime and accuracy.

Requirements

• TensorRT 10.x
• onnx-graphsurgeon
• ModelOpt toolkit 0.41.0
• BEVFormer_tensorrt
• PyTorch 2.9.0

Prepare dataset

Follow the Data Preparation steps for NuScenes and CAN bus. This will prepare the full train / validation dataset.

Prepare docker image

Build docker image:

$ export TAG=tensorrt_bevformer:26.01
$ docker build -f docker/tensorrt.Dockerfile --no-cache --tag=$TAG .

How to Run

1. Export model to ONNX and compile plugins

A. Download model weights from here and save it in ./models:

$ wget -P ./models https://github.com/zhiqi-li/storage/releases/download/v1.0/bevformer_tiny_epoch_24.pth

B. Run docker container:

$ docker run -it --rm --gpus device=0 --network=host --shm-size 20g -v $(pwd):/mnt -v <path to data>:/workspace/BEVFormer_tensorrt/data $TAG

C. In docker container, patch the BEVFormer_tensorrt folder and compile plugins:

# 1. Apply patch to BEVFormer_tensorrt with changes necessary for TensorRT 10 support
$ cd /workspace/BEVFormer_tensorrt
$ git apply /mnt/bevformer_trt10.patch

# 2. Compile plugins
$ cd TensorRT/build
$ cmake .. -DCMAKE_TENSORRT_PATH=/usr && make -j$(nproc) && make install

The compiled plugin will be saved in TensorRT/lib/libtensorrt_ops.so, which will later be used by both ModelOpt and TensorRT.

D. Export ONNX model from torch:

$ cd /workspace/BEVFormer_tensorrt
$ python tools/pth2onnx.py configs/bevformer/plugin/bevformer_tiny_trt_p2.py /mnt/models/bevformer_tiny_epoch_24.pth --opset=13 --cuda --flag=cp2_op13
$ cp checkpoints/onnx/bevformer_tiny_epoch_24_cp2_op13.onnx /mnt/models/

2. Prepare the calibration data

A. Post-process ONNX model:

$ export PLUGIN_PATH=/workspace/BEVFormer_tensorrt/TensorRT/lib/libtensorrt_ops.so
$ python /mnt/tools/onnx_postprocess.py --onnx=/mnt/models/bevformer_tiny_epoch_24_cp2_op13.onnx --trt_plugins=$PLUGIN_PATH

This will generate an ONNX file of same name as the input ONNX file with the suffix *_post.onnx, which will be used to generate the calibration data. May need to use CUDA_MODULE_LOADING=LAZY if using CUDA 12.x. No such variable is needed with CUDA 11.8.

This script does the following post-processing actions:

1. Automatically detect custom TRT ops in the ONNX model.
2. Ensure that the custom ops are supported as a TRT plugin in ONNX-Runtime (trt.plugins domain).
3. Update all tensor types and shapes in the ONNX graph with onnx-graphsurgeon.

B. Prepare the calibration data:

$ cd /workspace/BEVFormer_tensorrt
$ PYTHONPATH=$(pwd) python /mnt/tools/calib_data_prep.py configs/bevformer/plugin/bevformer_tiny_trt_p2.py \
    --onnx_path=/mnt/models/bevformer_tiny_epoch_24_cp2_op13_post.onnx \
    --trt_plugins=$PLUGIN_PATH

The calibration data will be saved in data/nuscenes/calib_data.npz. The script uses 600 calibration samples by default. See instructions in the ModelOpt toolkit for more info on generating the calibration data.

Note: the *_post.onnx model is only needed to generate the calibration data, so the original ONNX model can be used in the quantization step (Step 3).

3. Quantize ONNX model with calibration data

$ python -m modelopt.onnx.quantization --onnx_path=/mnt/models/bevformer_tiny_epoch_24_cp2_op13.onnx \
      --trt_plugins=$PLUGIN_PATH \
      --op_types_to_exclude MatMul \
      --calibration_data_path=/workspace/BEVFormer_tensorrt/data/nuscenes/calib_data.npz

This generates an ONNX model with suffix .quant.onnx with Q/DQ nodes around relevant layers.

Notes:

• MatMul ops are not being quantized (--op_types_to_exclude MatMul). The reasoning for this is that MHA blocks, present in Transformer-based models, are currently recommended to run in FP16. Keep in mind that optimal Q/DQ node placement can vary for different models, so there may be cases where quantizing MatMul ops may be more advantageous. This is up to the user to decide.
• If you're running out of memory, you may need to add CUDA_MODULE_LOADING=LAZY to the beginning of that quantization command. This is only valid for CUDA 12.x. No such variable is needed with CUDA 11.8.
• If you wish to simply check the explicitly quantized model's runtime performance, without considering its accuracy, you may also skip using the --calibration_data_path flag (and thus Step 2 altogether).

4. Build TensorRT engine

$ trtexec --onnx=/mnt/models/bevformer_tiny_epoch_24_cp2_op13.quant.onnx \
	      --saveEngine=/mnt/models/bevformer_tiny_epoch_24_cp2_op13.quant.engine \
	      --staticPlugins=$PLUGIN_PATH \
	      --stronglyTyped

Note: In order to deploy the quantized ONNX model in another platform or with another TensorRT version, simply re-compile the plugin for the required settings and deploy the engine using the same explicitly-quantized ONNX model.

5. Evaluate accuracy of TensorRT engine

Run evaluation script:

$ cd /workspace/BEVFormer_tensorrt
$ python tools/bevformer/evaluate_trt.py \
         configs/bevformer/plugin/bevformer_tiny_trt_p2.py \
         /mnt/models/bevformer_tiny_epoch_24_cp2_op13.quant.engine \
         --trt_plugins=$PLUGIN_PATH

Results

System: NVIDIA A40 GPU, TensorRT 10.14.1.48

BEVFormer tiny with FP16 plugins with nv_half2 (bevformer_tiny_epoch_24_cp2_op13.onnx):

Precision	GPU Compute Time (median, ms)	Accuracy (NDS / mAP)
FP32	18.68	NDS: 0.355, mAP: 0.252
FP16	9.27	NDS: 0.355, mAP: 0.252
IQ_BEST (TensorRT PTQ - Implicit Quantization)	6.16	NDS: 0.353, mAP: 0.249
EQ_StronglyTyped (ModelOpt PTQ - Explicit Quantization)	5.76	NDS: 0.352, mAP: 0.251

BEVFormer tiny with FP16 plugins with nv_half (bevformer_tiny_epoch_24_cp_op13.onnx):

Precision	GPU Compute Time (median, ms)	Accuracy (NDS / mAP)
FP32	18.65	NDS: 0.355, mAP: 0.252
FP16	9.72	NDS: 0.355, mAP: 0.252
IQ_BEST (TensorRT PTQ - Implicit Quantization)	6.41	NDS: 0.352, mAP: 0.248
EQ_StronglyTyped (ModelOpt PTQ - Explicit Quantization)	6.22	NDS: 0.352, mAP: 0.251

Steps to reproduce

To reproduce the results, run:

1. ./deploy_trt.sh to build/save the TensorRT engine and obtain the runtime;
2. ./evaluate_trt.sh to evaluate the TensorRT engine's accuracy.