Add ROCm pip index mirroring for rocm7.14 dependencies by jithunnair-amd · Pull Request #8436 · pytorch/test-infra

jithunnair-amd · 2026-08-04T23:45:22Z

Summary

Mirror ROCm SDK packages from AMD's multi-arch pip index (https://repo.amd.com/rocm/whl-multi-arch/) into download.pytorch.org dependency indexes for the rocm7.14 target.

Add torch_rocm workflow option to update-s3-dependencies.yml
Register 28 ROCm packages under torch_rocm / rocm7.14 in PACKAGES_PER_PROJECT (rocm, rocm-sdk-core, rocm-sdk-libraries, and all 25 rocm-sdk-device-* packages currently on the AMD index)
Route AMD packages via is_amd_package() and get_package_source_url()
Extend get_packages_for_target() so ROCm packages are included only for rocm* targets and create-target works for ROCm

Test plan

Verified AMD URL routing:

python3 -c "from s3_management.update_dependencies import get_package_source_url, is_amd_package; assert is_amd_package('rocm-sdk-core'); assert get_package_source_url('rocm-sdk-core') == 'https://repo.amd.com/rocm/whl-multi-arch/rocm-sdk-core/'"

Dry-run torch_rocm update (all 28 packages fetched from AMD index):

python3 s3_management/update_dependencies.py --package torch_rocm --dry-run

After merge: trigger Update S3 HTML dependencies workflow with package=torch_rocm, dryrun=enabled, then disabled
Confirm https://download.pytorch.org/whl/nightly/rocm7.14/rocm-sdk-core/index.html serves absolute links to repo.amd.com

Authored with assistance from Cursor

Made with Cursor

Mirror ROCm SDK packages from AMD's multi-arch pip index into download.pytorch.org dependency indexes for the rocm7.14 target. Adds a torch_rocm package group (28 packages: rocm, rocm-sdk-core, rocm-sdk-libraries, and all rocm-sdk-device-* targets on the AMD index), routes AMD packages through repo.amd.com, and extends get_packages_for_target so create-target works for ROCm targets. Test Plan: - Verified AMD URL routing: python3 -c "from s3_management.update_dependencies import get_package_source_url, is_amd_package; assert is_amd_package('rocm-sdk-core'); assert get_package_source_url('rocm-sdk-core') == 'https://repo.amd.com/rocm/whl-multi-arch/rocm-sdk-core/'" - Dry-run torch_rocm update: python3 s3_management/update_dependencies.py --package torch_rocm --dry-run Authored with assistance from Cursor Co-authored-by: Cursor <cursoragent@cursor.com>

vercel · 2026-08-04T23:45:25Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
torchci	Ignored	Preview	Aug 5, 2026 12:03am

Collapse multi-line print to a single line per pyfmt. Authored with assistance from Cursor Co-authored-by: Cursor <cursoragent@cursor.com>

## What Adds the 28 ROCm dependency packages introduced by #8436 to `PACKAGE_LINKS_ALLOW_LIST` in `s3_management/manage_v2.py`. #8436 taught `update_dependencies.py` to mirror the rocm7.14 dependency wheels from `repo.amd.com/rocm/whl-multi-arch`, adding 28 entries to `PACKAGES_PER_PROJECT` under project `torch_rocm`. Packages in `PACKAGE_LINKS_ALLOW_LIST` get their `index.html` **copied down from the parent prefix** rather than regenerated from wheel listings — e.g. `whl/nightly/rocm-sdk-core/index.html` → `whl/nightly/rocm7.14/rocm-sdk-core/index.html` — which is what lets a subdirectory index point at externally sourced packages. The `nvidia-*` and Intel/XPU dependency packages are already handled this way; the ROCm ones were missed. ## The packages ``` rocm rocm-sdk-core rocm-sdk-libraries rocm-sdk-device-gfx908 gfx90a gfx942 gfx950 rocm-sdk-device-gfx1010 gfx1011 gfx1012 rocm-sdk-device-gfx1030 gfx1031 gfx1032 gfx1033 gfx1034 gfx1035 gfx1036 rocm-sdk-device-gfx1100 gfx1101 gfx1102 gfx1103 rocm-sdk-device-gfx1150 gfx1151 gfx1152 gfx1153 rocm-sdk-device-gfx1200 gfx1201 gfx1250 ``` Rather than transcribe these from the diff by hand, I generated them from the merged source: every `PACKAGES_PER_PROJECT` key with a config whose `project` is `torch_rocm`. So the two lists cannot drift from a typo here. ## Test plan Checked by parsing both files with `ast` after the change: ``` torch_rocm packages in update_dependencies.py : 28 of those now in PACKAGE_LINKS_ALLOW_LIST : 28 missing : none total allow-list entries : 95 (67 -> 95) duplicates : none ``` - `python -m py_compile s3_management/manage_v2.py` passes - `black --check` reports the file unchanged - `flake8` output is byte-identical to `main`'s (the existing E501/B023 findings are pre-existing and far below the edited region) Data-only change; no logic touched. ## Note `PACKAGE_ALLOW_LIST` deliberately does **not** get these entries — it does not carry the `nvidia-*` or Intel dependency packages either, so this matches the established pattern for mirrored third-party deps. Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>

## Problem Follow-up to #8436. The ROCm indices it generates point at URLs that do not exist. From [whl/nightly/rocm7.14/rocm-sdk-device-gfx1010/](https://download.pytorch.org/whl/nightly/rocm7.14/rocm-sdk-device-gfx1010/): ```html <a href="https://repo.amd.com/rocm/whl-multi-arch/rocm-sdk-device-gfx1010/rocm_sdk_device_gfx1010-7.13.0-py3-none-linux_x86_64.whl"> ``` AMD serves a per-package *index* at `whl-multi-arch/<package>/`, but stores the *wheels* flat in `whl-multi-arch/`: | URL | result | |---|---| | `.../whl-multi-arch/rocm-sdk-device-gfx1010/rocm_sdk_device_gfx1010-7.13.0-...whl` (published) | **403** | | `.../whl-multi-arch/rocm_sdk_device_gfx1010-7.13.0-...whl` (actual location) | 206 | Every link in every ROCm package index is currently broken. ## Cause AMD's index hrefs carry the parent hop: ```html <a href="../rocm_sdk_device_gfx1010-7.13.0-py3-none-linux_x86_64.whl"> ``` `replace_relative_links_with_absolute()` normalised them with: ```python url = url.lstrip("./") url = url.lstrip("/") ``` `str.lstrip("./")` strips leading `.` and `/` **characters**, not a prefix, so `../x.whl` collapses to `x.whl`, which is then concatenated onto the package URL — silently producing the 403 path. ## Fix Resolve the href with `urllib.parse.urljoin`, which honours `../`. **Scoped to ROCm** via the existing `is_amd_package()`: NVIDIA (`pypi.nvidia.com/<pkg>/`, plain-filename hrefs) and PyPI (absolute `files.pythonhosted.org` hrefs) do not use `../` and keep the existing code path, so their generated indices are unchanged. ## Test plan Ran the real AMD index HTML for 6 ROCm packages through the patched function and HTTP range-requested every resulting URL: ``` rocm 2/2 reachable rocm-sdk-core 4/4 reachable rocm-sdk-libraries 4/4 reachable rocm-sdk-device-gfx1010 4/4 reachable rocm-sdk-device-gfx942 2/2 reachable rocm-sdk-device-gfx90a 3/3 reachable ------------------------------------------- TOTAL after fix 19/19 reachable currently published 0/4 reachable ``` Non-ROCm paths asserted byte-identical to today's output: | index | href in | href out | |---|---|---| | NVIDIA | `x.whl#sha256=a` | `https://pypi.nvidia.com/nvidia-cublas-cu12/x.whl#sha256=a` | | PyPI | `https://files.pythonhosted.org/...` | unchanged | `black --check` clean, `flake8` clean, `py_compile` passes. ## After merge The published indices are regenerated by the `update-s3-dependencies` workflow, so the broken links will be corrected on its next run — no manual S3/R2 cleanup needed. Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>

pytorch-bot Bot added ciflow/rocm module: rocm labels Aug 4, 2026

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 4, 2026

jithunnair-amd had a problem deploying to pytorchbot-env August 4, 2026 23:58 — with GitHub Actions Failure

jithunnair-amd requested a review from atalman August 5, 2026 00:01

Fix PYFMT lint in update_dependencies.py

ea7ffe4

Collapse multi-line print to a single line per pyfmt. Authored with assistance from Cursor Co-authored-by: Cursor <cursoragent@cursor.com>

atalman approved these changes Aug 5, 2026

View reviewed changes

atalman merged commit 0be39b5 into main Aug 5, 2026
10 checks passed

atalman deleted the rocm-pip-index-deps branch August 5, 2026 16:12

atalman mentioned this pull request Aug 5, 2026

Add ROCm packages to PACKAGE_LINKS_ALLOW_LIST #8445

Merged

atalman mentioned this pull request Aug 5, 2026

Fix ROCm index URLs dropping the parent-directory hop #8446

Merged

Navigation Menu

Uh oh!

Add ROCm pip index mirroring for rocm7.14 dependencies - #8436

Add ROCm pip index mirroring for rocm7.14 dependencies#8436
atalman merged 2 commits into
mainfrom
rocm-pip-index-deps

jithunnair-amd commented Aug 4, 2026

Uh oh!

vercel Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jithunnair-amd commented Aug 4, 2026

Summary

Test plan

Uh oh!

vercel Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Aug 4, 2026 •

edited

Loading