Skip to content

Normalize PACKAGE_LINKS_ALLOW_LIST to the spelling lookups actually use - #8513

Merged
atalman merged 1 commit into
pytorch:mainfrom
atalman:atalman/fix-package-links-normalization
Aug 12, 2026
Merged

Normalize PACKAGE_LINKS_ALLOW_LIST to the spelling lookups actually use#8513
atalman merged 1 commit into
pytorch:mainfrom
atalman:atalman/fix-package-links-normalization

Conversation

@atalman

@atalman atalman commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

whl/test/cu134/typing-extensions/ does not exist, so every CUDA 13.4 smoke test on v2.14.0-rc2 fails:

Looking in indexes: https://download.pytorch.org/whl/test/cu134
ERROR: Could not find a version that satisfies the requirement
       typing-extensions>=4.10.0 (from torch) (from versions: none)

All 8 Python versions of manywheel-*-cuda13_4-test fail this way (run). from versions: none rather than a version complaint, because the smoke test installs from that single index with no PyPI fallback and the package directory simply is not there.

Root cause

typing-extensions is in PACKAGE_LINKS_ALLOW_LIST, but the membership test can never succeed.

Every lookup compares an underscore-spelled name:

  • package names come from wheel filenames via obj_to_package_nametyping_extensions-4.16.0-...whl -> typing_extensions
  • get_packages_to_copy_from_parent maps parent directory names through the same conversion before comparing:
# manage_v2.py:801-803
pkg_name_with_underscores = parts[0].replace("-", "_")
parent_packages.add(pkg_name_with_underscores)
...
if pkg_name.lower() in PACKAGE_LINKS_ALLOW_LIST:

But the set stored whatever spelling was typed, which for most entries is hyphenated. So "typing_extensions" in {..., "typing-extensions", ...} is False, and the copy step skips it — in every subdirectory, in every channel, always.

This had been hit before and worked around by listing both spellings, with a comment saying as much:

"cuda-bindings",
"cuda_bindings",
"cuda-toolkit",
# Underscore spelling is the one that matches; see "spmd_types" above.

Applied to four entries, missed on the rest.

Scope

77 of the 102 entries could never be copied. Only single-word names (filelock, sympy, networkx, fsspec, ...) and the four with an explicit underscore twin worked — which is exactly what is published today: cu134 has filelock/sympy/networkx/fsspec and is missing typing-extensions.

cuda-toolkit is on the broken list too, which explains the other cu134 gap reported earlier.

Fix

Normalize once at definition:

PACKAGE_LINKS_ALLOW_LIST = {
    x.lower().replace("-", "_")
    for x in [...]
}

The four underscore twins and the comment documenting the workaround become redundant, so they are dropped. 102 entries -> 97.

Verified by replaying the real code paths (parent_dir.replace("-","_") and wheel_name.split("-",1)[0]) against the parsed list:

packages copyable BEFORE: 20
packages copyable AFTER:  97
newly working: 77          (cuda_toolkit, nvidia_*, rocm_sdk_*, intel_*, onemkl_*, typing_extensions, ...)
regressions:   none
entries that can never match (contain '-'): 0

Please dry-run before applying

This makes the next index run copy ~77 additional package indexes into every arch subdirectory. That is the intended behaviour and each one is already allow-listed, but it is a large one-time change to published indexes, so it is worth running the update in dry-run first and eyeballing the plan.

Does not retroactively repair existing subdirectories

get_packages_to_copy_from_parent only copies packages "in parent but not in subdir", so a subdir is never refreshed once a package exists there. Two consequences:

  • whl/test/cu134/typing-extensions/ will be created by this, since it is absent — so this does fix the rc2 breakage on the next run.
  • Already-populated copies stay frozen. whl/test/cu126/typing-extensions/ still lists 5 versions topping out at 4.15.0 while the parent has 64 up to 4.16.0. Harmless while the floor is >=4.10.0, but it is a separate latent issue and not addressed here.

ruff format and ruff check clean. No test added: s3_management/ has no test harness and no CI runs pytest over it — happy to add one if you would like that set up.

Every lookup against this set compares an underscore-spelled name. Package
names come from wheel filenames via obj_to_package_name, and
get_packages_to_copy_from_parent maps parent directory names through
.replace("-", "_") before comparing. But the set stored whatever spelling was
typed, which for most entries is hyphenated -- so "typing_extensions" was
tested against a set holding "typing-extensions" and never matched.

Effect: 77 of the 102 entries could never be copied into an arch subdirectory.
Only single-word names and the four that happened to carry an explicit
underscore twin worked. That is why whl/test/cu134/typing-extensions/ does not
exist and every cu134 smoke test on v2.14.0-rc2 fails to resolve
typing-extensions>=4.10.0 from an index that has no PyPI fallback. cuda-toolkit
was missing from cu134 for the same reason.

Normalize once at definition. The four underscore twins and the comment
explaining that workaround are now redundant, so drop them: 102 entries become
97, and the set they produce grows from 20 reachable names to 97, with no
name that previously worked dropping out.
@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 12, 2026
@atalman
atalman merged commit c22f7fa into pytorch:main Aug 12, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants