Exclude CUDA 13.4 from the docker release/validation matrix - #8605
Open
jeanschmidt wants to merge 1 commit into
Open
Exclude CUDA 13.4 from the docker release/validation matrix#8605jeanschmidt wants to merge 1 commit into
jeanschmidt wants to merge 1 commit into
Conversation
**Impact:** CI only — the nightly/test "Build Official Docker Images" validation matrix in pytorch/pytorch (no published images change)
**Risk:** low
## What
Adds a `CUDA_ARCHES_NO_DOCKER = ["13.4"]` exclusion list and filters it out of the docker matrix in `generate_docker_release_matrix.py`, so the docker validation matrix stops asking for CUDA 13.4 images. Wheel, libtorch and domain-library matrices are untouched.
## Why
The nightly `Build Official Docker Images` workflow has been red every night since 2026-08-11 — the `validate` stage tries to `docker pull` `cuda13.4-cudnn9-{runtime,devel}` images that the `build` stage in the same run never produces, failing with `manifest unknown` (e.g. [run 32626369633, cuda13.4-cudnn9-runtime](https://github.com/pytorch/pytorch/actions/runs/32626369633/job/97164375717)).
The root cause is drift between two independently-maintained CUDA lists in two repos. The build matrix comes from `CUDA_ARCHES_FULL_VERSION` in pytorch/pytorch, which deliberately withholds 13.4 (its release Dockerfile installs the devel toolkit from the standard NVIDIA apt repo, but `cuda-toolkit-13-4` ships only in NVIDIA's *preview* channel so far). The validate matrix comes from `CUDA_ARCHES_DICT` in test-infra, which grew a 13.4 entry (test-infra #8477) so torchvision/torchaudio would build cu134 — and that same constant, unknowingly, also feeds pytorch/pytorch's docker validation via the `@main`-pinned reusable workflow. Adding 13.4 for domain libraries silently turned on validation of docker images that were never meant to be built.
The fix keeps pytorch/pytorch's deliberate exclusion intact and re-aligns the validation matrix on the test-infra side, where the drift was introduced. No real coverage is lost — no cuda13.4 image exists to validate. The exclusion is applied across all channels, which also closes the same latent failure on the `test` channel before it gets hit.
## Notes
Full root-cause writeup, timeline, and options considered are in the analysis doc this change is based on. When revisiting: once NVIDIA promotes CUDA 13.4 into the standard apt repo, the images can be built (a two-line change to `CUDA_ARCHES_FULL_VERSION` in pytorch/pytorch) and `CUDA_ARCHES_NO_DOCKER` can drop back to `[]`.
Known gap this doesn't address: nothing tests `generate_docker_release_matrix.py`, and the validate job reports a bare `manifest unknown` rather than "image never built" — which is why this drift went unexamined for ~two weeks. Both are worth a follow-up but are out of scope here.
Authored with assistance from an AI coding assistant (Claude).
Signed-off-by: Jean Schmidt <contato@jschmidt.me>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
meta-cla
Bot
added
the
CLA Signed
Contributor
Author
|
reds are unrelated :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Impact: CI only — the nightly/test "Build Official Docker Images"
Risk: low
What
Adds a
CUDA_ARCHES_NO_DOCKER = ["13.4"]exclusion list and filters it out of the docker matrix ingenerate_docker_release_matrix.py, so the docker validation matrix stops asking for CUDA 13.4 images. Wheel, libtorch and domain-library matrices are untouched.Why
The nightly
Build Official Docker Imagesworkflow has been red every night since 2026-08-11 — thevalidatestage tries todocker pullcuda13.4-cudnn9-{runtime,devel}images that thebuildstage in the same run never produces, failing withmanifest unknown(e.g. run 32626369633, cuda13.4-cudnn9-runtime).The root cause is drift between two independently-maintained CUDA lists in two repos. The build matrix comes from
CUDA_ARCHES_FULL_VERSIONin pytorch/pytorch, which deliberately withholds 13.4 (its release Dockerfile installs the devel toolkit from the standard NVIDIA apt repo, butcuda-toolkit-13-4ships only in NVIDIA's preview channel so far). The validate matrix comes fromCUDA_ARCHES_DICTin test-infra, which grew a 13.4 entry (test-infra #8477) so torchvision/torchaudio would build cu134 — and that same constant, unknowingly, also feeds pytorch/pytorch's docker validation via the@main-pinned reusable workflow. Adding 13.4 for domain libraries silently turned on validation of docker images that were never meant to be built.The fix keeps pytorch/pytorch's deliberate exclusion intact and re-aligns the validation matrix on the test-infra side, where the drift was introduced. No real coverage is lost — no cuda13.4 image exists to validate. The exclusion is applied across all channels, which also closes the same latent failure on the
testchannel before it gets hit.Notes
Full root-cause writeup, timeline, and options considered are in the analysis doc this change is based on. When revisiting: once NVIDIA promotes CUDA 13.4 into the standard apt repo, the images can be built (a two-line change to
CUDA_ARCHES_FULL_VERSIONin pytorch/pytorch) andCUDA_ARCHES_NO_DOCKERcan drop back to[].Known gap this doesn't address: nothing tests
generate_docker_release_matrix.py, and the validate job reports a baremanifest unknownrather than "image never built" — which is why this drift went unexamined for ~two weeks. Both are worth a follow-up but are out of scope here.