Commit 39aeb9a
committed
Add pip-index.py for PEP 503 simple-index style mirrors
Adapted from ustclug/ustcmirror-images pytorch/sync.py (originally
pytorch.py). Crawls PEP 503 / PEP 691 simple HTML indexes recursively
and rewrites href attributes so saved index pages point back to this
mirror. The same script is used for multiple jobs by parameterising
endpoint discovery and href rewriting via environment variables.
Additions on top of the upstream script:
- Multi-host href rewrite (REWRITE_HOSTS):
handles upstreams that emit absolute URLs across multiple origins
(e.g. download.pytorch.org + download-r2.pytorch.org).
- EXTRA_REWRITES:
redirect hrefs that point at sibling hosts (e.g. files.pythonhosted.org)
to a sibling local mirror prefix (e.g. /pypi/web), so transitive deps
stay on the same mirror without forcing users to set --extra-index-url.
- DEVPI_MODE:
query a devpi server's channel JSON API and crawl only that channel's
own projects via .../<channel>/+simple/<project>/ instead of walking
the full inherited PyPI namespace.
- aiohttp instead of httpx, so it can run inside the existing shared
tunathu/tunasync-scripts:latest image with no extra dependencies.1 parent 93b8cff commit 39aeb9a
1 file changed
Lines changed: 447 additions & 0 deletions
0 commit comments