Inference Throughput vs. Batchsize

[quic-nmorillo]

I'm running a bert base onnx model with the tensorrt docker container. My model has dynamic batchsize.

docker pull nvcr.io/nvidia/tensorrt:21.09-py3
docker run --gpus 1 -it --rm -v $PWD:/workspace nvcr.io/nvidia/tensorrt:21.09-py3

Run trtexec
trtexec --onnx=./data/bert/bert-base-128.onnx --useCudaGraph --iterations=1000 --workspace=10000 --fp16 --optShapes=input_mask:1x128,segment_ids:1x128,input_ids:1x128 --verbose

trtexec --onnx=./data/bert/bert-base-128.onnx --useCudaGraph --iterations=1000 --workspace=10000 --fp16 --optShapes=input_mask:1x128,segment_ids:1x128,input_ids:1x128 --verbose

Here is my table for throughput and latency

	bert-base
Input shapes	Throughput (qps)	Avg E2E Latency (ms)
1x128	1089.81	1.75901
2x128	921.662	2.09433
4x128	734.285	2.64538
8x128	480.935	4.04696
32x128	149.114	13.1884

Question: How should I interpret the results? Why throughput decreases with increasing batchsize?

[kahmed10]

You need to multiply qps by the batch size. You should also look at the GPU compute time, which should be equivalent to qps if you do (1000/gpu_compute_time(ms)).

So in your example results above for batch size 32, you can multiply qps by 32, giving an actual qps result of 4,771.648.

[quic-nmorillo]

thanks for the explanation!

[quic-nmorillo]

thanks!

[biyuehuang]

You need to multiply qps by the batch size. You should also look at the GPU compute time, which should be equivalent to qps if you do (1000/gpu_compute_time(ms)).

So in your example results above for batch size 32, you can multiply qps by 32, giving an actual qps result of 4,771.648.

Hello,

I found 4218.33 qps != 3977 (30 *1000/ 7.54175 ms) (batch size / GPU Compute Time).
Do you mean 3977 is FPS (image per second)?

[09/06/2022-15:25:49] [I] === Performance summary ===
[09/06/2022-15:25:49] [I] Throughput: 4218.33 qps
[09/06/2022-15:25:49] [I] Latency: min = 12.5073 ms, max = 14.7645 ms, mean = 12.6645 ms, median = 12.6033 ms, pe rcentile(99%) = 13.9117 ms
[09/06/2022-15:25:49] [I] Enqueue Time: min = 0.0229492 ms, max = 0.0625 ms, mean = 0.0242712 ms, median = 0.0236 816 ms, percentile(99%) = 0.0396729 ms
[09/06/2022-15:25:49] [I] H2D Latency: min = 5.422 ms, max = 7.59802 ms, mean = 5.45845 ms, median = 5.4248 ms, p ercentile(99%) = 6.75647 ms
[09/06/2022-15:25:49] [I] GPU Compute Time: min = 6.97144 ms, max = 7.82031 ms, mean = 7.09014 ms, median = 7.059 45 ms, percentile(99%) = 7.54175 ms
[09/06/2022-15:25:49] [I] D2H Latency: min = 0.112793 ms, max = 0.212952 ms, mean = 0.115901 ms, median = 0.11474 6 ms, percentile(99%) = 0.170593 ms
[09/06/2022-15:25:49] [I] Total Host Walltime: 3.02252 s
[09/06/2022-15:25:49] [I] Total GPU Compute Time: 3.01331 s
[09/06/2022-15:25:49] [W] * GPU compute time is unstable, with coefficient of variance = 1.29424%.
[09/06/2022-15:25:49] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improv e the stability.
[09/06/2022-15:25:49] [I] Explanations of the performance metrics are printed in the verbose logs.
[09/06/2022-15:25:49] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v8403] # ./trtexec --avgRuns=100 --deploy=resnet10.prototxt --fp16 --iterations=10 --output=conv2d_bbox --useSpinWait --model=resnet10.caffemodel --batch=30

[kahmed10]

@biyuehuang Your run seems to already account for the batch size. QPS (queries per second) in this case is just a unit to describe inference since the output may not always be an image. So QPS = FPS in this example.

As for why the numbers are different when doing the computation, I don't know which latency number they use. Maybe they don't report the exact number in the statistics for some reason.

[duckduck-gaga]

You need to multiply qps by the batch size. You should also look at the GPU compute time, which should be equivalent to qps if you do (1000/gpu_compute_time(ms)).
So in your example results above for batch size 32, you can multiply qps by 32, giving an actual qps result of 4,771.648.

Hello,

I found 4218.33 qps != 3977 (30 *1000/ 7.54175 ms) (batch size / GPU Compute Time). Do you mean 3977 is FPS (image per second)?

[09/06/2022-15:25:49] [I] === Performance summary === [09/06/2022-15:25:49] [I] Throughput: 4218.33 qps [09/06/2022-15:25:49] [I] Latency: min = 12.5073 ms, max = 14.7645 ms, mean = 12.6645 ms, median = 12.6033 ms, pe rcentile(99%) = 13.9117 ms [09/06/2022-15:25:49] [I] Enqueue Time: min = 0.0229492 ms, max = 0.0625 ms, mean = 0.0242712 ms, median = 0.0236 816 ms, percentile(99%) = 0.0396729 ms [09/06/2022-15:25:49] [I] H2D Latency: min = 5.422 ms, max = 7.59802 ms, mean = 5.45845 ms, median = 5.4248 ms, p ercentile(99%) = 6.75647 ms [09/06/2022-15:25:49] [I] GPU Compute Time: min = 6.97144 ms, max = 7.82031 ms, mean = 7.09014 ms, median = 7.059 45 ms, percentile(99%) = 7.54175 ms [09/06/2022-15:25:49] [I] D2H Latency: min = 0.112793 ms, max = 0.212952 ms, mean = 0.115901 ms, median = 0.11474 6 ms, percentile(99%) = 0.170593 ms [09/06/2022-15:25:49] [I] Total Host Walltime: 3.02252 s [09/06/2022-15:25:49] [I] Total GPU Compute Time: 3.01331 s [09/06/2022-15:25:49] [W] * GPU compute time is unstable, with coefficient of variance = 1.29424%. [09/06/2022-15:25:49] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improv e the stability. [09/06/2022-15:25:49] [I] Explanations of the performance metrics are printed in the verbose logs. [09/06/2022-15:25:49] [I] &&&& PASSED TensorRT.trtexec [TensorRT v8403] # ./trtexec --avgRuns=100 --deploy=resnet10.prototxt --fp16 --iterations=10 --output=conv2d_bbox --useSpinWait --model=resnet10.caffemodel --batch=30

probably should use the mean value to calculate