Skip to content

[Doc] Update CUDA option prefer_nhwc #23812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tianleiwu merged 2 commits into from
Mar 4, 2025
Merged

[Doc] Update CUDA option prefer_nhwc #23812

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
96fe603
[CUDA] update prefer_nhwc option
tianleiwu Feb 25, 2025
a9f70f2
style
tianleiwu Feb 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 5 additions & 11 deletions docs/execution-providers/CUDA-ExecutionProvider.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,17 +325,11 @@ Default value: 0

### prefer_nhwc

This option is not available in default builds ! One has to compile ONNX Runtime
with `onnxruntime_USE_CUDA_NHWC_OPS=ON`.
If this is enabled the EP prefers NHWC operators over NCHW. Needed transforms will be added to the model. As NVIDIA
tensor cores can only work on NHWC layout this can increase performance if the model consists of many supported
operators and does not need too many new transpose nodes. Wider operator support is planned in the future.

This flag is only supported from the V2 version of the provider options struct when used using the C API. The V2
provider options struct can be created
using [CreateCUDAProviderOptions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a0d29cbf555aa806c050748cf8d2dc172)
and updated
using [UpdateCUDAProviderOptions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a4710fc51f75a4b9a75bde20acbfa0783).
This option is available since ONNX Runtime 1.20, where the build has `onnxruntime_USE_CUDA_NHWC_OPS=ON` by default.

If this option is enabled, the execution provider prefers NHWC operators over NCHW. Necessary layout transformations will be applied to the model automatically. Since NVIDIA tensor cores operate more efficiently with NHWC layout, enabling this option can improve performance when the model consists of many supported operators and does not require excessive additional transpose operations. Wider operator support for NHWC is planned for future releases.

This flag is supported only in the V2 version of the provider options struct when using the C API. The V2 provider options struct can be created using [CreateCUDAProviderOptions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a0d29cbf555aa806c050748cf8d2dc172) and updated using [UpdateCUDAProviderOptions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a4710fc51f75a4b9a75bde20acbfa0783).

Default value: 0

Expand Down
Loading