Action failed with error: Command "docker run --rm --gpus all --volume /mnt/vss/_work/onnxruntime/onnxruntime:/onnxruntime_src --volume /mnt/vss/_work/_temp:/onnxruntime_src/build --volume /home/cloudtest/.cache:/home/onnxruntimedev/.cache --volume /data/onnx:/data/onnx:ro --volume /data/models:/data/models:ro --volume /home/cloudtest/.onnx:/home/onnxruntimedev/.onnx -w /onnxruntime_src -e ALLOW_RELEASED_ONNX_OPSET_ONLY=0 -e NIGHTLY_BUILD=1 -e RUNNER_TEMP=/onnxruntime_src/build ghcr.io/microsoft/onnxruntime/onnxruntimetensorrt86gpubuild:sha256-8272d24e2c607527b015532966bb58b9c45418fac04442c4a902efe6605d5d7b /bin/bash -c set -ex; PATH=/opt/python/cp310-cp310/bin:$PATH && python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt && python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --test" failed with exit code 1

stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/usr/lib/jvm/msopenjdk-17/bin:/opt/rh/gcc-toolset-12/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. [notice] A new release of pip is available: 24.3.1 -> 25.1.1 [notice] To update, run: pip install --upgrade pip + python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --test 2025-06-30 00:48:22,323 build [DEBUG] - Command line arguments: --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --test 2025-06-30 00:48:22,328 build [INFO] - Build started 2025-06-30 00:48:22,328 build [DEBUG] - create symlink /data/models -> build/Release/models 2025-06-30 00:48:22,328 build [INFO] - Running tests for Release configuration 2025-06-30 00:48:22,328 build [INFO] - /usr/bin/ctest --build-config Release --verbose --timeout 10800 Errors while running CTest Output from these tests are in: /onnxruntime_src/build/Release/Release/Testing/Temporary/LastTest.log Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. Traceback (most recent call last): File "/onnxruntime_src/tools/ci_build/build.py", line 2616, in <module> sys.exit(main()) File "/onnxruntime_src/tools/ci_build/build.py", line 2526, in main run_onnxruntime_tests(args, source_dir, ctest_path, build_dir, configs) File "/onnxruntime_src/tools/ci_build/build.py", line 1687, in run_onnxruntime_tests run_subprocess(ctest_cmd, cwd=cwd, dll_path=dll_path) File "/onnxruntime_src/tools/ci_build/build.py", line 147, in run_subprocess return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env) File "/onnxruntime_src/tools/python/util/run.py", line 50, in run completed_process = subprocess.run( File "/opt/python/cp310-cp310/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/bin/ctest', '--build-config', 'Release', '--verbose', '--timeout', '10800']' returned non-zero exit status 8.

stdout: Collecting numpy==2.1.2 (from -r tools/ci_build/github/linux/python/requirements.txt (line 1)) Downloading numpy-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB) Requirement already satisfied: mypy in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 2)) (1.16.1) Requirement already satisfied: pytest in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 3)) (8.4.1) Requirement already satisfied: setuptools>=68.2.2 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 4)) (75.6.0) Requirement already satisfied: wheel in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 5)) (0.45.1) Requirement already satisfied: onnx==1.17.0 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 6)) (1.17.0) Requirement already satisfied: protobuf==4.21.12 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 7)) (4.21.12) Collecting sympy==1.12 (from -r tools/ci_build/github/linux/python/requirements.txt (line 8)) Downloading sympy-1.12-py3-none-any.whl.metadata (12 kB) Requirement already satisfied: flatbuffers in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 9)) (25.2.10) Requirement already satisfied: psutil in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from -r tools/ci_build/github/linux/python/requirements.txt (line 10)) (7.0.0) Collecting onnxscript==0.2.3 (from -r tools/ci_build/github/linux/python/requirements.txt (line 11)) Downloading onnxscript-0.2.3-py3-none-any.whl.metadata (15 kB) Collecting jinja2 (from -r tools/ci_build/github/linux/python/requirements.txt (line 12)) Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB) Collecting markupsafe (from -r tools/ci_build/github/linux/python/requirements.txt (line 13)) Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB) Requirement already satisfied: mpmath>=0.19 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from sympy==1.12->-r tools/ci_build/github/linux/python/requirements.txt (line 8)) (1.3.0) Requirement already satisfied: typing_extensions>=4.10 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from onnxscript==0.2.3->-r tools/ci_build/github/linux/python/requirements.txt (line 11)) (4.14.0) Collecting ml_dtypes (from onnxscript==0.2.3->-r tools/ci_build/github/linux/python/requirements.txt (line 11)) Downloading ml_dtypes-0.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB) Requirement already satisfied: packaging in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from onnxscript==0.2.3->-r tools/ci_build/github/linux/python/requirements.txt (line 11)) (24.2) Requirement already satisfied: mypy_extensions>=1.0.0 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 2)) (1.1.0) Requirement already satisfied: pathspec>=0.9.0 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 2)) (0.12.1) Requirement already satisfied: tomli>=1.1.0 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from mypy->-r tools/ci_build/github/linux/python/requirements.txt (line 2)) (2.2.1) Requirement already satisfied: exceptiongroup>=1 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from pytest->-r tools/ci_build/github/linux/python/requirements.txt (line 3)) (1.3.0) Requirement already satisfied: iniconfig>=1 in /opt/_internal/cpython-3.10.16/lib/python3.10/site-packages (from

Command failed with exit code 1

Wheel output directory /mnt/vss/_work/_temp/Release/dist does not exist.

stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/usr/lib/jvm/msopenjdk-17/bin:/opt/rh/gcc-toolset-12/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. [notice] A new release of pip is available: 24.3.1 -> 25.1.1 [notice] To update, run: pip install --upgrade pip + python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build 2025-06-30 00:08:52,436 build [DEBUG] - Command line arguments: --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --build 2025-06-30 00:08:52,440 build [INFO] - Build started 2025-06-30 00:08:52,440 build [INFO] - Building targets for Release configuration 2025-06-30 00:08:52,440 build [INFO] - /usr/bin/cmake --build build/Release/Release --config Release -- -j16 2025-06-30 00:31:26,324 build [INFO] - /opt/python/cp310-cp310/bin/python3 /onnxruntime_src/setup.py bdist_wheel --nightly_build --wheel_name_suffix=gpu --cuda_version=12.2 /opt/python/cp310-cp310/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. !! ******************************************************************************** Please avoid running ``setup.py`` directly. Instead, use pypa/build, pypa/installer or other standards-based tools. See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details. ******************************************************************************** !! self.initialize_options() DEBUG:auditwheel.musllinux:musl libc not detected DEBUG:auditwheel.libc:Falling back to GNU libc INFO:auditwheel.main_repair:Repairing onnxruntime_gpu-1.23.0.dev20250630-cp310-cp310-linux_x86_64.whl DEBUG:auditwheel.wheel_abi:processing: onnxruntime/capi/libonnxruntime.so.1.23.0 DEBUG:auditwheel.musllinux:musl libc not detected DEBUG:auditwheel.libc:Falling back to GNU libc DEBUG:auditwheel.lddtree:parse_ld_so_conf(//etc/ld.so.conf) DEBUG:auditwheel.lddtree: glob: //etc/ld.so.conf.d/*.conf DEBUG:auditwheel.lddtree: parse_ld_so_conf(//etc/ld.so.conf.d/00-manylinux.conf) DEBUG:auditwheel.lddtree: parse_ld_so_conf(//etc/ld.so.conf.d/gds-12-2.conf) DEBUG:auditwheel.lddtree: parse_ld_so_conf(//etc/ld.so.conf.d/nvidia.conf) DEBUG:auditwheel.lddtree: parse_ld_so_conf(//etc/ld.so.conf.d/988_cuda-12.conf) DEBUG:auditwheel.lddtree: parse_ld_so_conf(//etc/ld.so.conf.d/000_cuda.conf) DEBUG:auditwheel.lddtree:linker ldpaths: {'conf': ['/usr/local/lib', '/usr/local/cuda-12.2/targets/x86_64-linux/lib', '/usr/local/cuda-12/tar

Wheel output directory /mnt/vss/_work/_temp/Release/dist does not exist.

stderr: + PATH=/opt/python/cp310-cp310/bin:/usr/local/dotnet:/usr/lib/jvm/msopenjdk-17/bin:/opt/rh/gcc-toolset-12/root/usr/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + python3 -m pip install --user -r tools/ci_build/github/linux/python/requirements.txt WARNING: The script isympy is installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts f2py and numpy-config are installed in '/home/onnxruntimedev/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. [notice] A new release of pip is available: 24.3.1 -> 25.1.1 [notice] To update, run: pip install --upgrade pip + python3 tools/ci_build/build.py --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --update 2025-06-30 00:03:03,874 build [DEBUG] - Command line arguments: --build_dir build/Release --config Release --cmake_generator Ninja --skip_submodule_sync --build_shared_lib --parallel --use_vcpkg --use_vcpkg_ms_internal_asset_cache --enable_onnx_tests --use_cuda --use_tensorrt --use_binskim_compliant_compile_flags --build_wheel --cuda_version=12.2 --cuda_home=/usr/local/cuda-12.2 --cudnn_home=/usr/local/cuda-12.2 --use_tensorrt --tensorrt_home /usr --build_java --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=80 onnxruntime_BUILD_UNIT_TESTS=ON onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON --update 2025-06-30 00:03:03,878 build [INFO] - Build started 2025-06-30 00:03:03,878 build [INFO] - Generating CMake build tree 2025-06-30 00:03:03,896 build [INFO] - /usr/bin/cmake /onnxruntime_src/cmake -Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF -Donnxruntime_RUN_ONNX_TESTS=ON -Donnxruntime_GENERATE_TEST_REPORTS=ON -DPython_EXECUTABLE=/opt/python/cp310-cp310/bin/python3 -Donnxruntime_USE_VCPKG=ON -Donnxruntime_USE_MIMALLOC=OFF -Donnxruntime_ENABLE_PYTHON=ON -Donnxruntime_BUILD_CSHARP=OFF -Donnxruntime_BUILD_JAVA=ON -Donnxruntime_BUILD_NODEJS=OFF -Donnxruntime_BUILD_OBJC=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF -Donnxruntime_USE_DNNL=OFF -Donnxruntime_USE_NNAPI_BUILTIN=OFF -Donnxruntime_USE_VSINPU=OFF -Donnxruntime_USE_RKNPU=OFF -Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF -Donnxruntime_USE_VITISAI=OFF -Donnxruntime_USE_TENSORRT=ON -Donnxruntime_USE_NV=OFF -Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON -Donnxruntime_USE_TENSORRT_INTERFACE=OFF -Donnxruntime_USE_CUDA_INTERFACE=OFF -Donnxruntime_USE_NV_INTERFACE=OFF -Donnxruntime_USE_OPENVINO_INTERFACE=OFF -Donnxruntime_USE_VITISAI_INTERFACE=OFF -Donnxruntime_USE_QNN_INTERFACE=OFF -Donnxruntime_USE_MIGRAPHX=OFF -Donnxruntime_DISABLE_CONTRIB_OPS=OFF -Donnxruntime_DISABLE_ML_OPS=OFF -Donnxruntime_DISABLE_RTTI=OFF -Donnxruntime_DISABLE_EXCEPTIONS=OFF -Donnxruntime_MINIMAL_BUILD=OFF -Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF -Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF -Donnxruntime_REDUCED_OPS_BUILD=OFF -Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF -Donnxruntime_ENABLE_LTO=OFF -Donnxruntime_USE_ACL=OFF -Donnxruntime_USE_ARMNN=OFF -Donnxruntime_ARMNN_RELU_USE_CPU=ON -Donnxruntime_ARMNN_BN_USE_CPU=ON -Donnxruntime_USE_JSEP=OFF -Donnxruntime_USE_WEBGPU=OFF -Donnxruntime_USE_EXTERNAL_DAWN=OFF -Donnxruntime_ENABLE_NVTX_PROFILE=OFF -Donnxruntime_ENABLE_TRAINING=OFF -Donnxruntime_ENABLE_TRAINING_OPS=OFF -Donnxruntime_ENABLE_TRAINING_APIS=OFF -Donnxruntime_ENABLE_CPU_FP16_OPS=OFF -Donnxruntime_USE_NCCL=OFF -Donnxruntime_BUILD_BENCHMARKS=OFF

stderr: WARNING! Your credentials are stored unencrypted in '/home/cloudtest/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/