Skip to content

Commit 39aeb9a

Browse files
committed
Add pip-index.py for PEP 503 simple-index style mirrors
Adapted from ustclug/ustcmirror-images pytorch/sync.py (originally pytorch.py). Crawls PEP 503 / PEP 691 simple HTML indexes recursively and rewrites href attributes so saved index pages point back to this mirror. The same script is used for multiple jobs by parameterising endpoint discovery and href rewriting via environment variables. Additions on top of the upstream script: - Multi-host href rewrite (REWRITE_HOSTS): handles upstreams that emit absolute URLs across multiple origins (e.g. download.pytorch.org + download-r2.pytorch.org). - EXTRA_REWRITES: redirect hrefs that point at sibling hosts (e.g. files.pythonhosted.org) to a sibling local mirror prefix (e.g. /pypi/web), so transitive deps stay on the same mirror without forcing users to set --extra-index-url. - DEVPI_MODE: query a devpi server's channel JSON API and crawl only that channel's own projects via .../<channel>/+simple/<project>/ instead of walking the full inherited PyPI namespace. - aiohttp instead of httpx, so it can run inside the existing shared tunathu/tunasync-scripts:latest image with no extra dependencies.
1 parent 93b8cff commit 39aeb9a

1 file changed

Lines changed: 447 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)