[Build] Compile error with onnxruntime_providers_cuda.vcxproj

[MikeChaoSpin]

Describe the issue

Here is my env:

• OS: Windows 11
• Compiler: Visual Studio 2022 Community
• GPU: RTX 3070 Ti Laptop
• CUDA: v12.6
• cuDNN: 8.9.7.29_cuda12

I clone the onnxruntime-1.17.3 branch and run the following command on powershell:
.\build.bat --config Release --use_cuda --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" --parallel 4 --nvcc_threads 1 --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=86 --cmake_extra_defines onnxruntime_ENABLE_NVTX_PROFILE=ON --enable_cuda_line_info --compile_no_warning_as_error

After some waiting, the powershell show the message said the building was failed. And I scroll up and find this error message, I have no idea what is going wrong.

 tmpxft_00007408_00000000-7_moe_gemm_kernels_fp32_fp32.cudafe1.cpp
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\_deps\cutlass-src\include\cutlass/gemm/kernel
/sm90_gemm_tma_warpspecialized_pingpong.hpp(131): error C2061: 語法錯誤: 識別項 'SharedStorage' [C:\Users\kshs0\Documents\Libr
ary\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]

Urgency

No response

Target platform

x64

Build script

.\build.bat --config Release --use_cuda --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" --parallel 4 --nvcc_threads 1 --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=86 --cmake_extra_defines onnxruntime_ENABLE_NVTX_PROFILE=ON --enable_cuda_line_info --compile_no_warning_as_error

Error / output

tmpxft_00007408_00000000-7_moe_gemm_kernels_fp32_fp32.cudafe1.cpp
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kernel
/sm90_gemm_tma_warpspecialized_pingpong.hpp(131): error C2061: 語法錯誤: 識別項 'SharedStorage' [C:\Users\kshs0\Documents\Libr
ary\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(131): note: 範本具現化內容 (首先是最舊的) 是
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(60): note: 編譯類別範本部分特殊化 'cutlass::gemm::kernel::GemmUniversal<ProblemSha
pe_,CollectiveMainloop_,CollectiveEpilogue_,GridSwizzle_,enable_if<std::is_base_of_v<cutlass::gemm::KernelTmaWarpSpec
ializedPingpong,CollectiveMainloop_::DispatchPolicy::Schedule>,void>::type>' 時
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(119): note: 編譯類別 'cutlass::gemm::kernel::GemmUniversal<ProblemShape_,Co
llectiveMainloop_,CollectiveEpilogue_,GridSwizzle_,enable_if<std::is_base_of_v<cutlass::gemm::KernelTmaWarpSpecialize
dPingpong,CollectiveMainloop_::DispatchPolicy::Schedule>,void>::type>::SharedStorage' 時
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(128): note: 編譯類別 'cutlass::gemm::kernel::GemmUniversal<ProblemShape_,Co
llectiveMainloop_,CollectiveEpilogue_,GridSwizzle_,enable_if<std::is_base_of_v<cutlass::gemm::KernelTmaWarpSpecialize
dPingpong,CollectiveMainloop_::DispatchPolicy::Schedule>,void>::type>::SharedStorage::PipelineStorage' 時
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kernel
/sm90_gemm_tma_warpspecialized_pingpong.hpp(135): error C3646: 'math_wg_order': 未知的覆寫規範 [C:\Users\kshs0\Documents\Libra
ry\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\extras\visual_studio_integration\MSBuildExtensions\CUDA 12.6.t
argets(799,9): error MSB3721: 命令 ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin\nvcc.exe" --use-local-
env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.43.34808\bin\HostX64\x64" -x cu
-I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\utf8_range-src" -I"C:\Users\kshs0\Do
cuments\Library\onnxruntime-1.17.3\include\onnxruntime" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\include
onnxruntime\core\session" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\pytorch_cp
uinfo-src\include" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release" -I"C:\Users\kshs0\Docu
ments\Library\onnxruntime-1.17.3\onnxruntime" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Rele
ase_deps\abseil_cpp-src" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\safeint-sr
c" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\gsl-src\include" -I"C:\Users\kshs
0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\onnx-src" -I"C:\Users\kshs0\Documents\Library\onnxru
ntime-1.17.3\build\Windows\Release_deps\onnx-build" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windo
ws\Release_deps\protobuf-src\src" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\f
latbuffers-src\include" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src
include" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\examples" -I"C:
\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\eigen-src" -I"C:\Users\kshs0\Documents\Li
brary\onnxruntime-1.17.3\build\Windows\Release_deps\mp11-src\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit
\CUDA\v12.6\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\include" --keep-dir onnxrunt.33256
C6F\x64\Release -maxrregcount=0 --machine 64 --compile -cudart shared --expt-relaxed-constexpr --Werror default-stre
am-launch -Xcudafe --diag_suppress=bad_friend_decl -Xcudafe --diag_suppress=unsigned_compare_with_zero -Xcudafe --diag_
suppress=expr_has_no_effect -std=c++17 --generate-code=arch=compute_86,code=[compute_86,sm_86] --generate-line-info -Xc
udafe --diag_suppress=conversion_function_not_usable --threads 1 -Xcompiler="/guard:cf /Qspectre /Ob2 /EHsc -Ob2 -Zi /u
tf-8 /sdl /experimental:external /external:W0 /external:templates- /external:IC:/Users/kshs0/Documents/Library/onnxrunt
ime-1.17.3/cmake /external:IC:/Users/kshs0/Documents/Library/onnxruntime-1.17.3/build/Windows/Release /wd4251 /wd4201 /
wd4324 /wd5054 /w15038 /wd4251 /wd4201 /wd4324 /wd5054 /w15038 /wd4834 /wd4127" -DWIN32 -D_WINDOWS -DWINAPI_FAMILY=10
0 -DWINVER=0x0A00 -D_WIN32_WINNT=0x0A00 -DNTDDI_VERSION=0x0A000000 -DONNXRUNTIME_ENABLE_INTEL_METEOR_LAKE_MOBILE_PLATFO
RM_PERF_PATCH -DNDEBUG -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_USE_THREADS -DORT_NEURAL_SPEED -DDISABLE_CUSPARSE_DEPRECA
TED -DPLATFORM_WINDOWS -DNOGDI -DNOMINMAX -D_USE_MATH_DEFINES -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS -DUSE_CUDA=1 -D
USE_MEMORY_EFFICIENT_ATTENTION=1 -DONNX_NAMESPACE=onnx -DONNX_ML=1 -DONNX_USE_LITE_PROTO=1 -D__ONNX_NO_DOC_STRINGS -DWI
N32_LEAN_AND_MEAN -DENABLE_NVTX_PROFILE=1 -DORT_ENABLE_STREAM -DEIGEN_MPL2_ONLY -DEIGEN_HAS_CONSTEXPR -DEIGEN_HAS_VARIA
DIC_TEMPLATES -DEIGEN_HAS_CXX11_MATH -DEIGEN_HAS_CXX11_ATOMIC -DEIGEN_STRONG_INLINE=inline -D"CMAKE_INTDIR="Release""
-Donnxruntime_providers_cuda_EXPORTS -D_WINDLL -D_MBCS -DWIN32 -D_WINDOWS -DWINAPI_FAMILY=100 -DWINVER=0x0A00 -D_WIN32
WINNT=0x0A00 -DNTDDI_VERSION=0x0A000000 -DONNXRUNTIME_ENABLE_INTEL_METEOR_LAKE_MOBILE_PLATFORM_PERF_PATCH -DNDEBUG -DE
IGEN_HAS_C99_MATH -DCPUINFO_SUPPORTED -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_USE_THREADS -DORT_NEURAL_SPEED -DDISABLE_C
USPARSE_DEPRECATED -DPLATFORM_WINDOWS -DNOGDI -DNOMINMAX -D_USE_MATH_DEFINES -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS
-DUSE_CUDA=1 -DUSE_MEMORY_EFFICIENT_ATTENTION=1 -DONNX_NAMESPACE=onnx -DONNX_ML=1 -DONNX_USE_LITE_PROTO=1 -D__ONNX_NO_D
OC_STRINGS -DWIN32_LEAN_AND_MEAN -DENABLE_NVTX_PROFILE=1 -DORT_ENABLE_STREAM -DEIGEN_MPL2_ONLY -DEIGEN_HAS_CONSTEXPR -D
EIGEN_HAS_VARIADIC_TEMPLATES -DEIGEN_HAS_CXX11_MATH -DEIGEN_HAS_CXX11_ATOMIC -DEIGEN_STRONG_INLINE=inline -D"CMAKE_INTD
IR="Release"" -Donnxruntime_providers_cuda_EXPORTS -Xcompiler "/EHsc /W3 /nologo /O2 /FS /MD /GR" -Xcompiler "/Fdon
nxruntime_providers_cuda.dir\Release\vc143.pdb" -o onnxruntime_providers_cuda.dir\Release\moe_gemm_kernels_fp32_fp32.ob
j "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\onnxruntime\contrib_ops\cuda\moe\ft_moe\moe_gemm_kernels_fp32_fp
32.cu"" 已結束，代碼為 2。 [C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cud
a.vcxproj]
moe_gemm_kernels_fp16_fp16.cu
tmpxft_00006b90_00000000-7_moe_gemm_kernels_fp16_fp16.cudafe1.cpp
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kernel
/sm90_gemm_tma_warpspecialized_pingpong.hpp(131): error C2061: 語法錯誤: 識別項 'SharedStorage' [C:\Users\kshs0\Documents\Libr
ary\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(131): note: 範本具現化內容 (首先是最舊的) 是
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(60): note: 編譯類別範本部分特殊化 'cutlass::gemm::kernel::GemmUniversal<ProblemSha
pe,CollectiveMainloop_,CollectiveEpilogue_,GridSwizzle_,enable_if<std::is_base_of_v<cutlass::gemm::KernelTmaWarpSpec
ializedPingpong,CollectiveMainloop_::DispatchPolicy::Schedule>,void>::type>' 時
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(119): note: 編譯類別 'cutlass::gemm::kernel::GemmUniversal<ProblemShape_,Co
llectiveMainloop_,CollectiveEpilogue_,GridSwizzle_,enable_if<std::is_base_of_v<cutlass::gemm::KernelTmaWarpSpecialize
dPingpong,CollectiveMainloop_::DispatchPolicy::Schedule>,void>::type>::SharedStorage' 時
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kern
el/sm90_gemm_tma_warpspecialized_pingpong.hpp(128): note: 編譯類別 'cutlass::gemm::kernel::GemmUniversal<ProblemShape_,Co
llectiveMainloop_,CollectiveEpilogue_,GridSwizzle_,enable_if<std::is_base_of_v<cutlass::gemm::KernelTmaWarpSpecialize
dPingpong,CollectiveMainloop_::DispatchPolicy::Schedule>,void>::type>::SharedStorage::PipelineStorage' 時
C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\include\cutlass/gemm/kernel
/sm90_gemm_tma_warpspecialized_pingpong.hpp(135): error C3646: 'math_wg_order': 未知的覆寫規範 [C:\Users\kshs0\Documents\Libra
ry\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\extras\visual_studio_integration\MSBuildExtensions\CUDA 12.6.t
argets(799,9): error MSB3721: 命令 ""C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin\nvcc.exe" --use-local-
env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.43.34808\bin\HostX64\x64" -x cu
-I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\utf8_range-src" -I"C:\Users\kshs0\Do
cuments\Library\onnxruntime-1.17.3\include\onnxruntime" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\include
onnxruntime\core\session" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\pytorch_cp
uinfo-src\include" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release" -I"C:\Users\kshs0\Docu
ments\Library\onnxruntime-1.17.3\onnxruntime" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Rele
ase_deps\abseil_cpp-src" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\safeint-sr
c" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\gsl-src\include" -I"C:\Users\kshs
0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\onnx-src" -I"C:\Users\kshs0\Documents\Library\onnxru
ntime-1.17.3\build\Windows\Release_deps\onnx-build" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windo
ws\Release_deps\protobuf-src\src" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\f
latbuffers-src\include" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src
include" -I"C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\cutlass-src\examples" -I"C:
\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release_deps\eigen-src" -I"C:\Users\kshs0\Documents\Li
brary\onnxruntime-1.17.3\build\Windows\Release_deps\mp11-src\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit
\CUDA\v12.6\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\include" --keep-dir onnxrunt.33256
C6F\x64\Release -maxrregcount=0 --machine 64 --compile -cudart shared --expt-relaxed-constexpr --Werror default-stre
am-launch -Xcudafe --diag_suppress=bad_friend_decl -Xcudafe --diag_suppress=unsigned_compare_with_zero -Xcudafe --diag_
suppress=expr_has_no_effect -std=c++17 --generate-code=arch=compute_86,code=[compute_86,sm_86] --generate-line-info -Xc
udafe --diag_suppress=conversion_function_not_usable --threads 1 -Xcompiler="/guard:cf /Qspectre /Ob2 /EHsc -Ob2 -Zi /u
tf-8 /sdl /experimental:external /external:W0 /external:templates- /external:IC:/Users/kshs0/Documents/Library/onnxrunt
ime-1.17.3/cmake /external:IC:/Users/kshs0/Documents/Library/onnxruntime-1.17.3/build/Windows/Release /wd4251 /wd4201 /
wd4324 /wd5054 /w15038 /wd4251 /wd4201 /wd4324 /wd5054 /w15038 /wd4834 /wd4127" -DWIN32 -D_WINDOWS -DWINAPI_FAMILY=10
0 -DWINVER=0x0A00 -D_WIN32_WINNT=0x0A00 -DNTDDI_VERSION=0x0A000000 -DONNXRUNTIME_ENABLE_INTEL_METEOR_LAKE_MOBILE_PLATFO
RM_PERF_PATCH -DNDEBUG -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_USE_THREADS -DORT_NEURAL_SPEED -DDISABLE_CUSPARSE_DEPRECA
TED -DPLATFORM_WINDOWS -DNOGDI -DNOMINMAX -D_USE_MATH_DEFINES -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS -DUSE_CUDA=1 -D
USE_MEMORY_EFFICIENT_ATTENTION=1 -DONNX_NAMESPACE=onnx -DONNX_ML=1 -DONNX_USE_LITE_PROTO=1 -D__ONNX_NO_DOC_STRINGS -DWI
N32_LEAN_AND_MEAN -DENABLE_NVTX_PROFILE=1 -DORT_ENABLE_STREAM -DEIGEN_MPL2_ONLY -DEIGEN_HAS_CONSTEXPR -DEIGEN_HAS_VARIA
DIC_TEMPLATES -DEIGEN_HAS_CXX11_MATH -DEIGEN_HAS_CXX11_ATOMIC -DEIGEN_STRONG_INLINE=inline -D"CMAKE_INTDIR="Release""
-Donnxruntime_providers_cuda_EXPORTS -D_WINDLL -D_MBCS -DWIN32 -D_WINDOWS -DWINAPI_FAMILY=100 -DWINVER=0x0A00 -D_WIN32
_WINNT=0x0A00 -DNTDDI_VERSION=0x0A000000 -DONNXRUNTIME_ENABLE_INTEL_METEOR_LAKE_MOBILE_PLATFORM_PERF_PATCH -DNDEBUG -DE
IGEN_HAS_C99_MATH -DCPUINFO_SUPPORTED -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_USE_THREADS -DORT_NEURAL_SPEED -DDISABLE_C
USPARSE_DEPRECATED -DPLATFORM_WINDOWS -DNOGDI -DNOMINMAX -D_USE_MATH_DEFINES -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS
-DUSE_CUDA=1 -DUSE_MEMORY_EFFICIENT_ATTENTION=1 -DONNX_NAMESPACE=onnx -DONNX_ML=1 -DONNX_USE_LITE_PROTO=1 -D__ONNX_NO_D
OC_STRINGS -DWIN32_LEAN_AND_MEAN -DENABLE_NVTX_PROFILE=1 -DORT_ENABLE_STREAM -DEIGEN_MPL2_ONLY -DEIGEN_HAS_CONSTEXPR -D
EIGEN_HAS_VARIADIC_TEMPLATES -DEIGEN_HAS_CXX11_MATH -DEIGEN_HAS_CXX11_ATOMIC -DEIGEN_STRONG_INLINE=inline -D"CMAKE_INTD
IR="Release"" -Donnxruntime_providers_cuda_EXPORTS -Xcompiler "/EHsc /W3 /nologo /O2 /FS /MD /GR" -Xcompiler "/Fdon
nxruntime_providers_cuda.dir\Release\vc143.pdb" -o onnxruntime_providers_cuda.dir\Release\moe_gemm_kernels_fp16_fp16.ob
j "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\onnxruntime\contrib_ops\cuda\moe\ft_moe\moe_gemm_kernels_fp16_fp
16.cu"" 已結束，代碼為 2。 [C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cud
a.vcxproj]
test_execution_provider.vcxproj -> C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\Release
test_execution_provider.dll
Traceback (most recent call last):
File "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\tools\ci_build\build.py", line 2893, in
sys.exit(main())
^^^^^^
File "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\tools\ci_build\build.py", line 2785, in main
build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
File "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\tools\ci_build\build.py", line 1663, in build_targets
run_subprocess(cmd_args, env=env)
File "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\tools\ci_build\build.py", line 839, in run_subprocess
return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\tools\python\util\run.py", line 49, in run
completed_process = subprocess.run(
^^^^^^^^^^^^^^^
File "C:\msys64\ucrt64\lib\python3.12\subprocess.py", line 573, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\Program Files\CMake\bin\cmake.EXE', '--build', 'C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\\build\Windows\Release', '--config', 'Release', '--', '/maxcpucount:4', '/nodeReuse:False', '/p:CL_MPCount=4']' returned non-zero exit status 1.

Visual Studio Version

Community 2022

GCC / Compiler Version

MSVC 14.43.34808

[xadupre]

Compiling cuda files can takes quite some memory. Is it possible to try reducing the parallelism? You can also reduce the number of CUDA achitecture you want to support when compiling onnxruntime (with extra parameter --cmake_extra_defines '"CMAKE_CUDA_ARCHITECTURES='${CMAKE_CUDA_ARCHITECTURES}'"').

[MikeChaoSpin]

Hi, @xadupre

I reduced the parallelism to 2, and remove the --cmake_extra_defines in my command.
Here is my revised command
.\build.bat --config Release --use_cuda --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" --cudnn_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" --parallel 2 --nvcc_threads 1

However, the build process still failed, After reviewing the messages on terminal, I think the most important message is this:

C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\_deps\cutlass-src\include\cute/arch/cluster_s
m90.hpp(101): error #940-D: missing return statement at end of non-void function "cute::cluster_grid_dims" [C:\Users\ks
hs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
    }
    ^

  Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\_deps\cutlass-src\include\cute/arch/cluster_s
m90.hpp(120): error #940-D: missing return statement at end of non-void function "cute::cluster_id_in_grid" [C:\Users\k
shs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
    }
    ^

C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\_deps\cutlass-src\include\cute/arch/cluster_s
m90.hpp(101): error #940-D: missing return statement at end of non-void function "cute::cluster_grid_dims" [C:\Users\ks
hs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
    }
    ^

  Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

C:\Users\kshs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\_deps\cutlass-src\include\cute/arch/cluster_s
m90.hpp(120): error #940-D: missing return statement at end of non-void function "cute::cluster_id_in_grid" [C:\Users\k
shs0\Documents\Library\onnxruntime-1.17.3\build\Windows\Release\onnxruntime_providers_cuda.vcxproj]
    }
    ^

  2 errors detected in the compilation of "C:/Users/kshs0/Documents/Library/onnxruntime-1.17.3/onnxruntime/contrib_ops/
  cuda/moe/ft_moe/moe_gemm_kernels_fp16_fp16.cu".

I'm not sure it is due to lack of memory. During building process, I saw the memery usage is less than 8GB in total, I have 16GB of RAM on my laptop. After some trial, I still have no idea what is the problem...

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[snnn]

This issue has been automatically closed because it has been marked as 'stale' for more than 30 days without activity. If you believe this is still an issue, please feel free to reopen it.