Skip to content

Commit 90a18ed

Browse files
authored
Re-add nvidia-cudnn-cu12 9.1.0.70 to the mirrored cudnn index (#8532)
Fixes #193491 `nvidia-cudnn-cu12==9.1.0.70` disappeared from the cudnn index around 2026-08-12, which breaks every `pip install torch --index-url https://download.pytorch.org/whl/cu124` on Linux x86_64 with `ResolutionImpossible`. Every torch 2.4.0-2.6.0 build pins that exact version. **Cause:** NVIDIA stopped publishing 9.1.0.70 on `pypi.nvidia.com/nvidia-cudnn-cu12/` (the listing now jumps `9.0.0.312` -> `9.1.1.17`). `upload_package_using_simple_index` mirrors that index verbatim, so the next run silently dropped the entry even though the wheel itself is still served fine from S3/R2. **Fix:** a `RETAINED_WHEELS` table plus `append_retained_wheels()`, following the existing `append_preview_numpy_wheels()` idiom, re-injects links for wheels we still host but upstream no longer lists. Skipped if upstream ever restores the version, so there are no duplicate links. The injected href is a **root-relative path** (`/whl/cu124/nvidia_cudnn_cu12-9.1.0.70-...whl`), matching the form `manage_v2.py` already uses for its libtorch and source-code indexes. It therefore resolves against whichever host serves the index -- the wheel is present and identical on both `download.pytorch.org` and `download-r2.pytorch.org` -- and, being root-relative rather than directory-relative, stays correct wherever the index is copied. Injection consequently runs *after* `replace_relative_links_with_absolute`, which would otherwise resolve the path against `pypi.nvidia.com`. **Injection is limited to the channel roots** (`whl/nightly`, `whl/test`, `whl`). `manage_v2.py` copies a root package index down into every accepted subdirectory (`cu[0-9]+`, `rocm*`, `cpu`, `xpu`) for packages in its `PACKAGE_LINKS_ALLOW_LIST`, which includes `nvidia-cudnn-cu12`, so the CUDA targets inherit the link automatically. That keeps `cu124` and the other conserved targets conserved -- nothing re-mirrors them from upstream, so no other version NVIDIA removes can be dropped from them either. `PACKAGES_PER_PROJECT` and `manage_v2.py` are both unchanged. `cu118` is unaffected: torch 2.6.0+cu118 pins `nvidia-cudnn-cu11==9.1.0.70`, and `nvidia_cudnn_cu11` is deliberately absent from `PACKAGE_LINKS_ALLOW_LIST` -- `whl/cu118/nvidia-cudnn-cu11/index.html` is a hand-curated 4-link index that still lists the version and is untouched here. ## Verification - sha256 `165764f4...` is PyPI's. Confirmed the hosted file is byte-identical without downloading 664 MB: the S3 ETag on `whl/cu124` and `whl/nightly/cu124` is `97a2573180f8d6968a8068086339ff75`, exactly PyPI's published md5, and content-length matches at `664752741` on both `download.pytorch.org` and `download-r2.pytorch.org`. - Ran the full `upload_package_using_simple_index` path with a stubbed S3: the link lands in the three channel-root indexes with the href intact as `/whl/cu124/...`, is absent from `whl/nightly/cu126`, and no href anywhere was mangled into a `pypi.nvidia.com/.../whl/` URL. - Fed the result through pip's own `parse_links` collector, both as the root index and as the same bytes copied into a `cu124/` subdirectory (what `manage_v2.py` does): 151 links in each, both resolving to the same absolute wheel URL against the serving host, sha256 recognised as the link hash. - The patched root index still satisfies `manage_v2.index_has_external_links()` (the 150 mirrored `pypi.nvidia.com` links), so the copy-to-subdirectory step is not skipped. - Confirmed via AST extraction that `nvidia_cudnn_cu12` is in `PACKAGE_LINKS_ALLOW_LIST` and that `cu124` matches `ACCEPTED_SUBDIR_PATTERNS`. This also matches observed state: every `cu*` subdir index currently has an ETag byte-identical to its channel root. Not exercised: the S3/R2 `put` calls themselves (no boto3 or usable pip on the machine this was written on), so a `dryrun: enabled` run is worth doing first. ## Rollout `mode=update-packages, package=torch` covers nightly + test; prod needs a second run with `include-prod=enabled` (the `promote-env`-gated `update-prod` job). The cu\* subdirectories pick the link up on the next `manage_v2.py` run.
1 parent a339e80 commit 90a18ed

1 file changed

Lines changed: 72 additions & 1 deletion

File tree

s3_management/update_dependencies.py

Lines changed: 72 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import os
22
import re
33
import time
4-
from typing import Dict, List
4+
from typing import Dict, List, NamedTuple
55
from urllib.parse import urljoin
66

77
import boto3 # type: ignore[import-untyped]
@@ -919,6 +919,40 @@
919919
}
920920

921921

922+
class RetainedWheel(NamedTuple):
923+
filename: str
924+
# Root-relative path to the wheel, so it resolves against whichever host
925+
# serves the index (download.pytorch.org or download-r2.pytorch.org) and
926+
# stays valid when the index is copied into subdirectories.
927+
path: str
928+
sha256: str
929+
930+
931+
# Wheels that we still host but the upstream simple index has stopped
932+
# publishing, so mirroring alone drops them and pinned installs stop resolving.
933+
# Keyed by normalised package name.
934+
#
935+
# Only the channel root is injected: manage_v2.py copies a root package index
936+
# down into every accepted subdirectory for packages in its
937+
# PACKAGE_LINKS_ALLOW_LIST, so the CUDA targets inherit these links without
938+
# their conserved indexes being re-mirrored from upstream.
939+
RETAINED_WHEEL_PREFIXES = {"whl/nightly", "whl/test", "whl"}
940+
RETAINED_WHEELS: Dict[str, List[RetainedWheel]] = {
941+
"nvidia_cudnn_cu12": [
942+
# Pinned by every torch 2.4.0-2.6.0 build; removed from pypi.nvidia.com
943+
# around 2026-08-12. See pytorch/pytorch#193491.
944+
RetainedWheel(
945+
filename="nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl",
946+
path=(
947+
"/whl/cu124/"
948+
"nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl"
949+
),
950+
sha256="165764f44ef8c61fcdfdfdbe769d687e06374059fbb388b6c89ecb0e28793a6f",
951+
),
952+
],
953+
}
954+
955+
922956
def is_nvidia_package(pkg_name: str) -> bool:
923957
"""Check if a package is from NVIDIA and is therefore CUDA-target only"""
924958
return pkg_name.startswith("nvidia-") or pkg_name.startswith("cuda-")
@@ -1028,6 +1062,10 @@ def upload_index_html(
10281062
html, base_url, resolve_relative_paths=is_amd_package(pkg_name)
10291063
)
10301064

1065+
# Re-add wheels we still host that upstream has stopped publishing. Runs
1066+
# after the rewrite above so the root-relative paths are left intact.
1067+
modified_html = append_retained_wheels(modified_html, pkg_name, prefix)
1068+
10311069
index_key = f"{prefix}/{pkg_name}/index.html"
10321070

10331071
if dry_run:
@@ -1136,6 +1174,39 @@ def append_preview_numpy_wheels(html: str, pkg_name: str, prefix: str) -> str:
11361174
return f"{html}\n{block}\n"
11371175

11381176

1177+
def append_retained_wheels(html: str, pkg_name: str, prefix: str) -> str:
1178+
"""Merge :data:`RETAINED_WHEELS` links for *pkg_name* into *html*.
1179+
1180+
Limited to the channel roots in :data:`RETAINED_WHEEL_PREFIXES`. Wheels the
1181+
upstream index still lists are skipped, so this becomes a no-op if a version
1182+
is ever restored upstream.
1183+
1184+
Must run after :func:`replace_relative_links_with_absolute`, which would
1185+
otherwise resolve these root-relative paths against the upstream index.
1186+
"""
1187+
entries = RETAINED_WHEELS.get(normalize_pkg_name(pkg_name))
1188+
if not entries or prefix not in RETAINED_WHEEL_PREFIXES:
1189+
return html
1190+
1191+
additions = [
1192+
f' <a href="{entry.path}#sha256={entry.sha256}">{entry.filename}</a><br/>'
1193+
for entry in entries
1194+
if entry.filename not in html
1195+
]
1196+
1197+
if not additions:
1198+
return html
1199+
1200+
print(
1201+
f"INFO: Merging {len(additions)} retained wheel link(s) "
1202+
f"for {pkg_name} under {prefix}"
1203+
)
1204+
block = "\n".join(additions)
1205+
if "</body>" in html:
1206+
return html.replace("</body>", f"{block}\n </body>", 1)
1207+
return f"{html}\n{block}\n"
1208+
1209+
11391210
def upload_package_using_simple_index(
11401211
pkg_name: str,
11411212
prefix: str,

0 commit comments

Comments
 (0)