From 13f4e3b47a8a36432d6b72bbbec56980af96ef3b Mon Sep 17 00:00:00 2001 From: Jean Schmidt Date: Sun, 23 Aug 2026 07:11:34 -0700 Subject: [PATCH] Exclude CUDA 13.4 from the docker release/validation matrix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Impact:** CI only — the nightly/test "Build Official Docker Images" validation matrix in pytorch/pytorch (no published images change) **Risk:** low ## What Adds a `CUDA_ARCHES_NO_DOCKER = ["13.4"]` exclusion list and filters it out of the docker matrix in `generate_docker_release_matrix.py`, so the docker validation matrix stops asking for CUDA 13.4 images. Wheel, libtorch and domain-library matrices are untouched. ## Why The nightly `Build Official Docker Images` workflow has been red every night since 2026-08-11 — the `validate` stage tries to `docker pull` `cuda13.4-cudnn9-{runtime,devel}` images that the `build` stage in the same run never produces, failing with `manifest unknown` (e.g. [run 32626369633, cuda13.4-cudnn9-runtime](https://github.com/pytorch/pytorch/actions/runs/32626369633/job/97164375717)). The root cause is drift between two independently-maintained CUDA lists in two repos. The build matrix comes from `CUDA_ARCHES_FULL_VERSION` in pytorch/pytorch, which deliberately withholds 13.4 (its release Dockerfile installs the devel toolkit from the standard NVIDIA apt repo, but `cuda-toolkit-13-4` ships only in NVIDIA's *preview* channel so far). The validate matrix comes from `CUDA_ARCHES_DICT` in test-infra, which grew a 13.4 entry (test-infra #8477) so torchvision/torchaudio would build cu134 — and that same constant, unknowingly, also feeds pytorch/pytorch's docker validation via the `@main`-pinned reusable workflow. Adding 13.4 for domain libraries silently turned on validation of docker images that were never meant to be built. The fix keeps pytorch/pytorch's deliberate exclusion intact and re-aligns the validation matrix on the test-infra side, where the drift was introduced. No real coverage is lost — no cuda13.4 image exists to validate. The exclusion is applied across all channels, which also closes the same latent failure on the `test` channel before it gets hit. ## Notes Full root-cause writeup, timeline, and options considered are in the analysis doc this change is based on. When revisiting: once NVIDIA promotes CUDA 13.4 into the standard apt repo, the images can be built (a two-line change to `CUDA_ARCHES_FULL_VERSION` in pytorch/pytorch) and `CUDA_ARCHES_NO_DOCKER` can drop back to `[]`. Known gap this doesn't address: nothing tests `generate_docker_release_matrix.py`, and the validate job reports a bare `manifest unknown` rather than "image never built" — which is why this drift went unexamined for ~two weeks. Both are worth a follow-up but are out of scope here. Authored with assistance from an AI coding assistant (Claude). Signed-off-by: Jean Schmidt --- tools/scripts/generate_binary_build_matrix.py | 8 ++++++++ tools/scripts/generate_docker_release_matrix.py | 7 ++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/tools/scripts/generate_binary_build_matrix.py b/tools/scripts/generate_binary_build_matrix.py index aa852a260c..fdbd187283 100755 --- a/tools/scripts/generate_binary_build_matrix.py +++ b/tools/scripts/generate_binary_build_matrix.py @@ -84,6 +84,14 @@ # and release builds and their validation are unchanged. CUDA_ARCHES_NO_GETTING_STARTED = ["13.4"] +# CUDA versions with no official Docker image, mirroring the omissions from +# CUDA_ARCHES_FULL_VERSION in pytorch/pytorch, which drives the docker build +# matrix. Only affects the docker matrix; wheel and libtorch builds and their +# validation are unchanged. 13.4 is here because cuda-toolkit-13-4 ships only +# in NVIDIA's preview channel (packages.nvidia.com), not in the CUDA apt repo +# that pytorch/pytorch's release Dockerfile installs the devel toolkit from. +CUDA_ARCHES_NO_DOCKER = ["13.4"] + PACKAGE_TYPES = ["wheel", "libtorch"] CXX11_ABI = "cxx11-abi" RELEASE = "release" diff --git a/tools/scripts/generate_docker_release_matrix.py b/tools/scripts/generate_docker_release_matrix.py index c8605a11ce..b0b3ab9bc7 100644 --- a/tools/scripts/generate_docker_release_matrix.py +++ b/tools/scripts/generate_docker_release_matrix.py @@ -44,7 +44,12 @@ def generate_docker_matrix( else: docker_image_version = f"{prefix}-nightly:{generate_binary_build_matrix.CURRENT_NIGHTLY_VERSION}.dev{datetime.today().strftime('%Y%m%d')}" - for cuda in generate_binary_build_matrix.CUDA_ARCHES_DICT[channel]: + cuda_arches = [ + cuda + for cuda in generate_binary_build_matrix.CUDA_ARCHES_DICT[channel] + if cuda not in generate_binary_build_matrix.CUDA_ARCHES_NO_DOCKER + ] + for cuda in cuda_arches: version = generate_binary_build_matrix.CUDA_CUDNN_VERSIONS[cuda] for image in DOCKER_IMAGE_TYPES: ret.append(