Shadowmire for air-gapped and disk‑space constrained environment

[taoky]

In my point of view, at least 90% of contents on PyPI are extremely niche -- 99% people won't need them. Also they exhibit possible threats (supply-chain attack) for air-gapped environments, where traditionally sync everything from PyPI.

Features to support (some would require separate scripts):

• Get full package list from a "curated package" list (considering all deps)
• Generate "curated package" list from uv lock file (and further, PEP 751 https://peps.python.org/pep-0751/)
• Generate "curated package" list from package popularity
• Allow users to use package list for shadowmire, instead of just syncing no package or every package (controlled by --sync-packages flag now)
• Allow users to customize index with "curated package" only (currently index contains all packages info)

[z0z0r4]

Good idea! But how can we automatically find out which packages are popular? Any API?

I'm a CS student from a relatively poor university, and I'm trying to build a basic, at least usable mirror from nothing.

Considering that PyPI is almost the most common requirement, but at the same time, mirroring PyPI is incredibly expensive, I'm also considering how to mirror only a select few packages (although maintaining a fixed-size cache based on file access frequency should also be sufficient).

OT: 从科大 Pypi Help Page 过来的，注意到它有写到动态调整缓存，但很可惜在这里没有找到相关功能

[taoky]

Good idea! But how can we automatically find out which packages are popular? Any API?

BigQuery 可以获取 PyPI 上的包下载情况；TUNA 的 log 是公开的，因此也可以从 Nginx log 分析。