You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/CUDA.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@
8
8
|`tensorrt`| ORT TensorRT EP (FP16, engine cache, opt-level 5) | NVIDIA GPU with TensorRT installed; 2–3× faster than `cuda`|
9
9
|`cuda-preprocess`| GPU-side preprocess + zero-copy device input to TRT | maximum throughput; `YOLOModel::predict_image` transparently uses a fused CUDA preprocess kernel |
10
10
11
-
`cuda-preprocess` implies `cuda` + `tensorrt`. When it's compiled in, no API change is required —`YOLOModel::predict_image` automatically routes through the GPU preprocess path on CUDA/TensorRT devices. Opt out per-model with [`InferenceConfig::with_cuda_preprocess(false)`](crate::InferenceConfig::with_cuda_preprocess).
11
+
`cuda-preprocess` implies `cuda` + `tensorrt`. When it's compiled in, no API change is required -`YOLOModel::predict_image` automatically routes through the GPU preprocess path on CUDA/TensorRT devices. Opt out per-model with [`InferenceConfig::with_cuda_preprocess(false)`](crate::InferenceConfig::with_cuda_preprocess).
12
12
13
13
## Requirements
14
14
@@ -32,7 +32,7 @@ ultralytics-inference = { version = "0.0.26", features = ["tensorrt"] }
32
32
ultralytics-inference = { version = "0.0.26", features = ["cuda-preprocess"] }
33
33
```
34
34
35
-
Then `cargo build --release`— no extra flags needed.
35
+
Then `cargo build --release`- no extra flags needed.
36
36
37
37
For the CLI / examples in this repo directly:
38
38
@@ -104,7 +104,7 @@ What to expect and how to avoid surprises:
104
104
to `.gitignore`, not to clean builds) to avoid paying the cost again.
105
105
-**Cache is keyed to the build context.** A new engine is built whenever the
106
106
model file, GPU/driver/TensorRT version, precision (`--half`), or **input
107
-
shape** changes. **Dynamic-shape models rebuild per new input size**— feed a
107
+
shape** changes. **Dynamic-shape models rebuild per new input size**- feed a
108
108
consistent size (the fast path uses the model's resolved `imgsz`) to keep it
109
109
to a single cached engine.
110
110
-**Warm up before timing.** The first `predict*` call also triggers an
This uses the TensorRT EP (FP16 + engine cache). The `cuda-preprocess` kernel
185
-
fast path is **not** used by the CLI — the CLI runs through the batch
185
+
fast path is **not** used by the CLI - the CLI runs through the batch
186
186
processor, which uses CPU preprocess. The GPU preprocess path is reached only
187
187
through `YOLOModel::predict_image` in library code.
188
188
@@ -195,5 +195,5 @@ through `YOLOModel::predict_image` in library code.
195
195
|`cudarc-* build script failed: \`nvcc --version\` failed` | Set `PATH` to include the toolkit's `bin/`, or set `CUDARC_CUDA_VERSION` (see above). |
196
196
|`libcudart.so.13: cannot open shared object file`| Toolkit not installed or not on `ld.so` path. Verify `ldconfig -p \| grep libcudart.so`. |
197
197
|`libnvinfer.so.10: cannot open shared object file`| TensorRT not installed. Required for `tensorrt` and `cuda-preprocess` features. |
198
-
| TRT engine build is slow on first run | Expected — engines are cached under `.trt_cache/`. Subsequent runs reuse them. |
198
+
| TRT engine build is slow on first run | Expected - engines are cached under `.trt_cache/`. Subsequent runs reuse them. |
199
199
| Build hits `Must specify one of the following features: [cuda-13020, ...]`| Your environment has neither `nvcc` on `PATH` nor `CUDARC_CUDA_VERSION` set. Pick one. |
0 commit comments