Skip to content

Commit 13f4e3b

Browse files
committed
Exclude CUDA 13.4 from the docker release/validation matrix
**Impact:** CI only — the nightly/test "Build Official Docker Images" validation matrix in pytorch/pytorch (no published images change) **Risk:** low ## What Adds a `CUDA_ARCHES_NO_DOCKER = ["13.4"]` exclusion list and filters it out of the docker matrix in `generate_docker_release_matrix.py`, so the docker validation matrix stops asking for CUDA 13.4 images. Wheel, libtorch and domain-library matrices are untouched. ## Why The nightly `Build Official Docker Images` workflow has been red every night since 2026-08-11 — the `validate` stage tries to `docker pull` `cuda13.4-cudnn9-{runtime,devel}` images that the `build` stage in the same run never produces, failing with `manifest unknown` (e.g. [run 32626369633, cuda13.4-cudnn9-runtime](https://github.com/pytorch/pytorch/actions/runs/32626369633/job/97164375717)). The root cause is drift between two independently-maintained CUDA lists in two repos. The build matrix comes from `CUDA_ARCHES_FULL_VERSION` in pytorch/pytorch, which deliberately withholds 13.4 (its release Dockerfile installs the devel toolkit from the standard NVIDIA apt repo, but `cuda-toolkit-13-4` ships only in NVIDIA's *preview* channel so far). The validate matrix comes from `CUDA_ARCHES_DICT` in test-infra, which grew a 13.4 entry (test-infra #8477) so torchvision/torchaudio would build cu134 — and that same constant, unknowingly, also feeds pytorch/pytorch's docker validation via the `@main`-pinned reusable workflow. Adding 13.4 for domain libraries silently turned on validation of docker images that were never meant to be built. The fix keeps pytorch/pytorch's deliberate exclusion intact and re-aligns the validation matrix on the test-infra side, where the drift was introduced. No real coverage is lost — no cuda13.4 image exists to validate. The exclusion is applied across all channels, which also closes the same latent failure on the `test` channel before it gets hit. ## Notes Full root-cause writeup, timeline, and options considered are in the analysis doc this change is based on. When revisiting: once NVIDIA promotes CUDA 13.4 into the standard apt repo, the images can be built (a two-line change to `CUDA_ARCHES_FULL_VERSION` in pytorch/pytorch) and `CUDA_ARCHES_NO_DOCKER` can drop back to `[]`. Known gap this doesn't address: nothing tests `generate_docker_release_matrix.py`, and the validate job reports a bare `manifest unknown` rather than "image never built" — which is why this drift went unexamined for ~two weeks. Both are worth a follow-up but are out of scope here. Authored with assistance from an AI coding assistant (Claude). Signed-off-by: Jean Schmidt <contato@jschmidt.me>
1 parent 61a7757 commit 13f4e3b

2 files changed

Lines changed: 14 additions & 1 deletion

File tree

tools/scripts/generate_binary_build_matrix.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,14 @@
8484
# and release builds and their validation are unchanged.
8585
CUDA_ARCHES_NO_GETTING_STARTED = ["13.4"]
8686

87+
# CUDA versions with no official Docker image, mirroring the omissions from
88+
# CUDA_ARCHES_FULL_VERSION in pytorch/pytorch, which drives the docker build
89+
# matrix. Only affects the docker matrix; wheel and libtorch builds and their
90+
# validation are unchanged. 13.4 is here because cuda-toolkit-13-4 ships only
91+
# in NVIDIA's preview channel (packages.nvidia.com), not in the CUDA apt repo
92+
# that pytorch/pytorch's release Dockerfile installs the devel toolkit from.
93+
CUDA_ARCHES_NO_DOCKER = ["13.4"]
94+
8795
PACKAGE_TYPES = ["wheel", "libtorch"]
8896
CXX11_ABI = "cxx11-abi"
8997
RELEASE = "release"

tools/scripts/generate_docker_release_matrix.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,12 @@ def generate_docker_matrix(
4444
else:
4545
docker_image_version = f"{prefix}-nightly:{generate_binary_build_matrix.CURRENT_NIGHTLY_VERSION}.dev{datetime.today().strftime('%Y%m%d')}"
4646

47-
for cuda in generate_binary_build_matrix.CUDA_ARCHES_DICT[channel]:
47+
cuda_arches = [
48+
cuda
49+
for cuda in generate_binary_build_matrix.CUDA_ARCHES_DICT[channel]
50+
if cuda not in generate_binary_build_matrix.CUDA_ARCHES_NO_DOCKER
51+
]
52+
for cuda in cuda_arches:
4853
version = generate_binary_build_matrix.CUDA_CUDNN_VERSIONS[cuda]
4954
for image in DOCKER_IMAGE_TYPES:
5055
ret.append(

0 commit comments

Comments
 (0)