Commit 6958756
authored
s3_management: keep PyPI-sourced NVIDIA indexes from being overwritten (#8483)
Follow-up to #8473, which switched `cuda-bindings`, `cuda-pathfinder`,
`cuda-python` and `nvidia-ml-py` to be sourced from PyPI
(`pypi.org/simple/`) instead of `pypi.nvidia.com`.
`update_dependencies.py` writes those indexes **per accelerator target**
— `main()` loops `PACKAGES_PER_PROJECT` and calls
`upload_package_using_simple_index(pkg_name, f"{prefix}/{target}")`,
producing `whl/nightly/cu130/cuda-bindings/index.html` etc. with
absolute `files.pythonhosted.org` links. `cuda-bindings` is declared for
cu126, cu128, cu129, cu130, cu132 and cu134.
`manage_v2.py` separately regenerates per-package indexes from the
wheels it finds in S3. `PACKAGE_LINKS_ALLOW_LIST` is what stops it from
overwriting an externally-sourced index with an S3-relative one (see
`index_has_external_links()`, #8115). All four packages need to be in
that list now, or the PyPI-sourced indexes get clobbered on the next
`manage_v2` run.
Adds `cuda-bindings`, `cuda-pathfinder`, `cuda-python` and
`nvidia-ml-py`.
### Why both spellings
Membership is tested against the name from `obj_to_package_name()` — the
wheel filename up to the first `-` — which uses **underscores**
(`cuda_bindings`). A hyphen-only entry can never match, which is why the
list already carries both `spmd-types` and `spmd_types`. `cuda-bindings`
was already present hyphen-only and therefore inert.
Verified against the live nightly indexes (link counts, root vs
`cu128`):
| entry | spelling | root | cu128 | |
| --- | --- | --- | --- | --- |
| `numpy` | single word | 4188 | 4188 | identical — active |
| `spmd-types` | both listed | 11 | 11 | identical — active |
| `typing-extensions` | hyphen only | 116 | 6 | regenerated — inert |
| `cuda-bindings` | hyphen only | 159 | 39 | regenerated — inert |
The other ~40 hyphen-only entries in the list are dormant for the same
reason. Not touched here — activating them en masse is its own change.
### Upstream availability
`cuda-bindings` is not on the NVIDIA CDN, so PyPI is the correct source:
| index | `pypi.nvidia.com` | `pypi.org/simple` |
| --- | --- | --- |
| `cuda-bindings` | 404 | 200 (393 files) |
| `cuda-python` | 404 | 200 (335) |
| `cuda-pathfinder` | 404 | 200 (27) |
| `nvidia-ml-py` | 404 | 200 (67) |
| `nvidia-cublas-cu12` | 200 | 200 |
Every CUDA major is on the PyPI CDN, so a single upstream index serving
all targets is fine — cu126/cu128 pull the 12.9 files and cu130+ the
13.0 files from the same index.
### Test plan
`cuda-python`, `cuda-pathfinder` and `nvidia-ml-py` each publish one
`py3-none-any` wheel and currently resolve identically at root and in
all 14 live accelerator subdirectories (`cpu`, `cu118`–`cu134`,
`rocm6.3`–`rocm7.1`, `xpu`), so no index should change shape.
`cuda-bindings` is the one that moves: today `manage_v2` regenerates it
per subdirectory from S3 wheels (cu126/cu128/cu129 → 12.9.x,
cu130/cu132/cu134 → 13.0.3); after this it keeps the PyPI-sourced index
that `update_dependencies.py` writes.
Confirmed the set literal evaluates with all four underscore names
present. `lintrunner` PYFMT and `ruff` clean; remaining `flake8` E501
hits are pre-existing (7 on `main`, unchanged).
cc @tinglvv1 parent da70907 commit 6958756
1 file changed
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
491 | 491 | | |
492 | 492 | | |
493 | 493 | | |
| 494 | + | |
494 | 495 | | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
495 | 503 | | |
496 | 504 | | |
497 | 505 | | |
| |||
0 commit comments