Blackwell : Non-zero status code returned while running Slice node. #27237
Description
Environment
- ONNXRuntime version from vcpkg : 1.23.2 https://vcpkg.io/en/package/onnxruntime
- CUDA 13.0, Official NVIDIA docker image from
nvcr.io/nvidia/pytorch:25.10-py3 - Or CUDA 12.9.1, Offical NVIDIA docker image from
nvcr.io/nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04 - GPU : RTX 6000 Pro Blackwell
- Model : https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX
Description
The same docker image works well on RTX 4060 (SM89), but after deploying to RTX 6000 Pro Blackwell (SM120), we encounter the following error when performing model inference.
[E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Slice node. Name:'/encoder/bert/Slice' Status Message: CUDA error cudaErrorSymbolNotFound:named symbol not found
I tried different configuration when building the onnxruntime vcpkg, including the following attempts. All of them fail with the same error.
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CMAKE_SYSTEM_NAME Linux)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic) // I also tried static
set(VCPKG_CMAKE_CONFIGURE_OPTIONS
"-DCMAKE_CUDA_ARCHITECTURES=89"
)
set(VCPKG_CMAKE_CONFIGURE_OPTIONS
"-DCMAKE_CUDA_ARCHITECTURES=89;90"
)
set(VCPKG_CMAKE_CONFIGURE_OPTIONS
"-DCMAKE_CUDA_ARCHITECTURES=89;90;120"
)
set(VCPKG_CMAKE_CONFIGURE_OPTIONS
"-DCMAKE_CUDA_ARCHITECTURES=89;90;120-virtual"
)