|
| 1 | +--- |
| 2 | +title: NVIDIA - TensorRT RTX |
| 3 | +description: Instructions to execute ONNX Runtime on NVIDIA RTX GPUs with the Nvidia TensorRT RTX execution provider |
| 4 | +parent: Execution Providers |
| 5 | +nav_order: 17 |
| 6 | +redirect_from: /docs/reference/execution-providers/TensorRTRTX-ExecutionProvider |
| 7 | +--- |
| 8 | + |
| 9 | +# Nvidia TensorRT RTX Execution Provider |
| 10 | +{: .no_toc } |
| 11 | + |
| 12 | +Nvidia TensorRT RTX execution provider is the preferred execution provider for GPU acceleration on consumer hardware (RTX PCs). It is more straightforward to use than the datacenter focused legacy TensorRT Execution provider and more performant than CUDA EP. |
| 13 | +Just some of the things that make it a better fit on RTX PCs than our legacy TensorRT Execution Provider: |
| 14 | +* Much smaller footprint |
| 15 | +* Much faster model compile/load times. |
| 16 | +* Better usability in terms of use of cached models across multiple RTX GPUs. |
| 17 | + |
| 18 | +The Nvidia TensorRT RTX execution provider in the ONNX Runtime makes use of NVIDIA's [TensorRT](https://developer.nvidia.com/tensorrt) RTX Deep Learning inferencing engine (TODO: correct link to TRT RTX documentation once available) to accelerate ONNX models on RTX GPUs. Microsoft and NVIDIA worked closely to integrate the TensorRT RTX execution provider with ONNX Runtime. |
| 19 | + |
| 20 | +Currently TensorRT RTX supports RTX GPUs from Ampere or later architectures. Support for Turing GPUs is coming soon. |
| 21 | + |
| 22 | +## Contents |
| 23 | +{: .no_toc } |
| 24 | + |
| 25 | +* TOC placeholder |
| 26 | +{:toc} |
| 27 | + |
| 28 | +## Install |
| 29 | +Please select the Nvidia TensorRT RTX version of Onnx Runtime: https://onnxruntime.ai/docs/install. (TODO!) |
| 30 | + |
| 31 | +## Build from source |
| 32 | +See [Build instructions](../build/eps.md#TensorRT-RTX). |
| 33 | + |
| 34 | +## Requirements |
| 35 | + |
| 36 | +| ONNX Runtime | TensorRT-RTX | CUDA | |
| 37 | +| :----------- | :----------- | :------------- | |
| 38 | +| main | 1.0 | 12.0-12.9 | |
| 39 | +| 1.22 | 1.0 | 12.0-12.9 | |
| 40 | + |
| 41 | +## Usage |
| 42 | +### C/C++ |
| 43 | +```c++ |
| 44 | +const auto& api = Ort::GetApi(); |
| 45 | +Ort::SessionOptions session_options; |
| 46 | +api.SessionOptionsAppendExecutionProvider(session_options, "NvTensorRtRtx", nullptr, nullptr, 0); |
| 47 | +Ort::Session session(env, model_path, session_options); |
| 48 | +``` |
| 49 | +
|
| 50 | +The C API details are [here](../get-started/with-c.md). |
| 51 | +
|
| 52 | +### Python |
| 53 | +To use TensorRT RTX execution provider, you must explicitly register TensorRT RTX execution provider when instantiating the `InferenceSession`. |
| 54 | +
|
| 55 | +```python |
| 56 | +import onnxruntime as ort |
| 57 | +sess = ort.InferenceSession('model.onnx', providers=['NvTensorRtRtxExecutionProvider']) |
| 58 | +``` |
| 59 | + |
| 60 | +## Configurations |
| 61 | +TensorRT RTX settings can be configured via [TensorRT Execution Provider Session Option](./TensorRTRTX-ExecutionProvider.md#execution-provider-options). |
| 62 | + |
| 63 | +Here are examples and different [scenarios](./TensorRTRTX-ExecutionProvider.md#scenario) to set NV TensorRT RTX EP session options: |
| 64 | + |
| 65 | +#### Click below for Python API example: |
| 66 | + |
| 67 | +<details> |
| 68 | + |
| 69 | +```python |
| 70 | +import onnxruntime as ort |
| 71 | + |
| 72 | +model_path = '<path to model>' |
| 73 | + |
| 74 | +# note: for bool type options in python API, set them as False/True |
| 75 | +provider_options = { |
| 76 | + 'device_id': 0, |
| 77 | + 'nv_dump_subgraphs': False, |
| 78 | + 'nv_detailed_build_log': True, |
| 79 | + 'user_compute_stream': stream_handle |
| 80 | +} |
| 81 | + |
| 82 | +sess_opt = ort.SessionOptions() |
| 83 | +sess = ort.InferenceSession(model_path, sess_options=sess_opt, providers=[('NvTensorRTRTXExecutionProvider', provider_options)]) |
| 84 | +``` |
| 85 | + |
| 86 | +</details> |
| 87 | + |
| 88 | +#### Click below for C++ API example: |
| 89 | + |
| 90 | +<details> |
| 91 | + |
| 92 | +```c++ |
| 93 | +Ort::SessionOptions session_options; |
| 94 | + |
| 95 | +cudaStream_t cuda_stream; |
| 96 | +cudaStreamCreate(&cuda_stream); |
| 97 | + |
| 98 | +// Need to put the CUDA stream handle in a string |
| 99 | +char streamHandle[32]; |
| 100 | +sprintf_s(streamHandle, "%lld", (uint64_t)cuda_stream); |
| 101 | + |
| 102 | +const auto& api = Ort::GetApi(); |
| 103 | +std::vector<const char*> option_keys = { |
| 104 | + "device_id", |
| 105 | + "user_compute_stream", // this implicitly sets "has_user_compute_stream" |
| 106 | +}; |
| 107 | +std::vector<const char*> option_values = { |
| 108 | + "1", |
| 109 | + streamHandle |
| 110 | +}; |
| 111 | + |
| 112 | +Ort::ThrowOnError(api.SessionOptionsAppendExecutionProvider(session_options, "NvTensorRtRtx", option_keys.data(), option_values.data(), option_keys.size())); |
| 113 | + |
| 114 | +``` |
| 115 | +
|
| 116 | +</details> |
| 117 | +
|
| 118 | +### Scenario |
| 119 | +
|
| 120 | +| Scenario | NV TensorRT RTX EP Session Option | Type | |
| 121 | +| :------------------------------------------------- | :----------------------------------------------------------------------------------------- | :----- | |
| 122 | +| Specify GPU id for execution | [device_id](./TensorRTRTX-ExecutionProvider.md#device_id) | int | |
| 123 | +| Set custom compute stream for GPU operations | [user_compute_stream](./TensorRTRTX-ExecutionProvider.md#user_compute_stream) | string | |
| 124 | +| Set TensorRT RTX EP GPU memory usage limit | [nv_max_workspace_size](./TensorRTRTX-ExecutionProvider.md#nv_max_workspace_size) | int | |
| 125 | +| Dump optimized subgraphs for debugging | [nv_dump_subgraphs](./TensorRTRTX-ExecutionProvider.md#nv_dump_subgraphs) | bool | |
| 126 | +| Capture CUDA graph for reduced launch overhead | [nv_cuda_graph_enable](./TensorRTRTX-ExecutionProvider.md#nv_cuda_graph_enable) | bool | |
| 127 | +| Enable detailed logging of build steps | [nv_detailed_build_log](./TensorRTRTX-ExecutionProvider.md#nv_detailed_build_log) | bool | |
| 128 | +| Define min shapes | [nv_profile_min_shapes](./TensorRTRTX-ExecutionProvider.md#nv_profile_min_shapes) | string | |
| 129 | +| Define max shapes | [nv_profile_max_shapes](./TensorRTRTX-ExecutionProvider.md#nv_profile_max_shapes) | string | |
| 130 | +| Define optimal shapes | [nv_profile_opt_shapes](./TensorRTRTX-ExecutionProvider.md#nv_profile_opt_shapes) | string | |
| 131 | +
|
| 132 | +> Note: for bool type options, assign them with **True**/**False** in python, or **1**/**0** in C++. |
| 133 | +
|
| 134 | +### Execution Provider Options |
| 135 | +
|
| 136 | +TensorRT RTX configurations can be set by execution provider options. It's useful when each model and inference session have their own configurations. All configurations should be set explicitly, otherwise default value will be taken. |
| 137 | +
|
| 138 | +##### device_id |
| 139 | +
|
| 140 | +* Description: GPU device ID. |
| 141 | +* Default value: 0 |
| 142 | +
|
| 143 | +##### user_compute_stream |
| 144 | +
|
| 145 | +* Description: define the compute stream for the inference to run on. It implicitly sets the `has_user_compute_stream` option. The stream handle needs to be printed on a string as decimal number and passed down to the session options as shown in the example above. |
| 146 | +
|
| 147 | +* This can also be set using the python API. |
| 148 | + * i.e The cuda stream captured from pytorch can be passed into ORT-NV TensorRT RTX EP. Click below to check sample code: |
| 149 | +
|
| 150 | + <Details> |
| 151 | +
|
| 152 | + |
| 153 | + ```python |
| 154 | + import onnxruntime as ort |
| 155 | + import torch |
| 156 | + ... |
| 157 | + sess = ort.InferenceSession('model.onnx') |
| 158 | + if torch.cuda.is_available(): |
| 159 | + s = torch.cuda.Stream() |
| 160 | + provider_options = { |
| 161 | + 'device_id': 0, |
| 162 | + 'user_compute_stream': str(s.cuda_stream) |
| 163 | + } |
| 164 | +
|
| 165 | + sess = ort.InferenceSession( |
| 166 | + model_path, |
| 167 | + providers=[('NvTensorRtRtxExecutionProvider', provider_options)] |
| 168 | + ) |
| 169 | +
|
| 170 | + options = sess.get_provider_options() |
| 171 | + assert "NvTensorRtRtxExecutionProvider" in options |
| 172 | + assert options["NvTensorRtRtxExecutionProvider"].get("user_compute_stream", "") == str(s.cuda_stream) |
| 173 | + ... |
| 174 | + ``` |
| 175 | + |
| 176 | + </Details> |
| 177 | +
|
| 178 | +* To take advantage of user compute stream, it is recommended to use [I/O Binding](https://onnxruntime.ai/docs/performance/device-tensor.html) to bind inputs and outputs to tensors in device. |
| 179 | +
|
| 180 | +##### nv_max_workspace_size |
| 181 | +
|
| 182 | +* Description: maximum workspace size in bytes for TensorRT RTX engine. |
| 183 | +
|
| 184 | +* Default value: 0 (lets TensorRT pick the optimal). |
| 185 | +
|
| 186 | +##### nv_dump_subgraphs |
| 187 | +
|
| 188 | +* Description: dumps the subgraphs if the ONNX was split across multiple execution providers. |
| 189 | + * This can help debugging subgraphs, e.g. by using `trtexec --onnx subgraph_1.onnx` and check the outputs of the parser. |
| 190 | +
|
| 191 | +##### nv_detailed_build_log |
| 192 | +
|
| 193 | +* Description: enable detailed build step logging on NV TensorRT RTX EP with timing for each engine build. |
| 194 | +
|
| 195 | +##### nv_cuda_graph_enable |
| 196 | +
|
| 197 | +* Description: this will capture a [CUDA graph](https://developer.nvidia.com/blog/cuda-graphs/) which can drastically help for a network with many small layers as it reduces launch overhead on the CPU. |
| 198 | +
|
| 199 | +##### nv_profile_min_shapes |
| 200 | +
|
| 201 | +##### nv_profile_max_shapes |
| 202 | +
|
| 203 | +##### nv_profile_opt_shapes |
| 204 | +
|
| 205 | +* Description: build with explicit dynamic shapes using a profile with the min/max/opt shapes provided. |
| 206 | + * By default TensorRT RTX engines will support dynamic shapes, for perofmance improvements it is possible to specify one or multiple explicit ranges of shapes. |
| 207 | + * The format of the profile shapes is `input_tensor_1:dim_1xdim_2x...,input_tensor_2:dim_3xdim_4x...,...` |
| 208 | + * These three flags should all be provided in order to enable explicit profile shapes feature. |
| 209 | + * Note that multiple TensorRT RTX profiles can be enabled by passing multiple shapes for the same input tensor. |
| 210 | + * Check TensorRT doc [optimization profiles](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#opt_profiles) for more details. |
| 211 | +
|
| 212 | +## NV TensorRT RTX EP Caches |
| 213 | +There are two major TRT RTX EP caches: |
| 214 | +* Embedded engine model / EPContext model |
| 215 | +* Internal TensorRT RTX cache |
| 216 | +
|
| 217 | +The internal TensorRT RTX cache is automatically managed by the EP. The user only needs to manage EPContext caching. |
| 218 | +**Caching is important to help reduce session creation time drastically.** |
| 219 | +
|
| 220 | +TensorRT RTX separates compilation into an ahead of time (AOT) compiled engine and a just in time (JIT) compilation. The AOT compilation can be stored as EPcontext model, this model will be compatible across multiple GPU generations. |
| 221 | +Upon loading such an EPcontext model TensorRT RTX will just in time compile the engine to fit to the used GPU. This JIT process is accelerated by TensorRT RTX's internal cache. |
| 222 | +For an example usage see: |
| 223 | +https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/test/providers/nv_tensorrt_rtx/nv_basic_test.cc |
| 224 | +
|
| 225 | +### More about Embedded engine model / EPContext model |
| 226 | +* TODO: decide on a plan for using weight-stripped engines by default. Fix the EP implementation to enable that. Explain the motivation and provide example on how to use the right options in this document. |
| 227 | +* EPContext models also **enable packaging an externally compiled engine** using e.g. `trtexec`. A [python script](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/tensorrt/gen_trt_engine_wrapper_onnx_model.py) that is capable of packaging such a precompiled engine into an ONNX file is included in the python tools. (TODO: document how this works with weight-stripped engines). |
| 228 | +
|
| 229 | +## Performance Tuning |
| 230 | +For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](./../performance/tune-performance/index.md) |
| 231 | +
|
| 232 | +When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e nvtensorrttrx`. |
| 233 | +
|
| 234 | +
|
| 235 | +### TensorRT RTX Plugins Support |
| 236 | +TensorRT RTX doesn't support plugins. |
0 commit comments