perf(uv): extract wheel metadata per-wheel to avoid fetching sibling platform wheels#1356
perf(uv): extract wheel metadata per-wheel to avoid fetching sibling platform wheels#1356jbedard wants to merge 1 commit into
Conversation
✨ Aspect Workflows Tasks📅 Mon Jul 20 21:49:55 UTC 2026 ✅ 40 successful tasks
⏱ Last updated Mon Jul 20 21:54:07 UTC 2026 · 📊 GitHub API quota 959/15,000 (6% used, resets in 13m) |
py_binary startup benchmark
sys.path quality
Bazel analysis benchmark
Auxiliary metrics
|
tamird
left a comment
tamird
left a comment
There was a problem hiding this comment.
The per-bdist provider is the cleaner lazy boundary: the existing selector can choose one wheel repo, so universal wheels avoid #1348's extra install repo/selector. The benchmark agrees (101 loaded packages / 301 configured targets unchanged, with analysis about 5.6% faster).
I found three correctness regressions and one important coverage gap:
- both e2e jobs currently fail on the new provider requirement before the patch-failure action runs;
- replacing
http_filedrops its authenticated-download and default canonical-ID behavior; - supported hashless wheels are now reported reproducible without recording the discovered integrity;
- the tests no longer exercise the repository rule or prove that an inactive wheel stays unfetched.
This supersedes the lazy-metadata half of #1348, but it does not deduplicate actual_install actions across lock universes; that remains separate work. The existing downstream lazy/grouping fixtures and their //file labels cannot be applied unchanged on top of this design.
| # active configuration — its own repo's RECORD-derived metadata (or empty, | ||
| # for a source-built wheel). A sibling platform wheel's surface can't leak | ||
| # in: it lives in a different repo that is never fetched or consulted here. | ||
| meta = ctx.attr.src[PyWheelMetadataInfo] |
There was a problem hiding this comment.
This unconditionally requires PyWheelMetadataInfo, but the unchanged e2e/cases/patch-failure/BUILD.bazel:35-51 passes the raw :fake_wheel genrule to both installs. Both Bazel 8 and 9 jobs now stop here with:
<target //patch-failure:fake_wheel> (rule 'genrule') doesn't contain declared provider 'PyWheelMetadataInfo'
FAIL: expected the patch-failure diagnostic on stderr
So patch behavior is no longer tested. Wrap the generated wheel once in source_built_wheel (its layout is intentionally unknown) and point both installs at that wrapper, or retain the raw-wheel fallback if direct callers are still supported.
| url = rctx.attr.url, | ||
| output = basename, | ||
| sha256 = rctx.attr.sha256, | ||
| ) |
There was a problem hiding this comment.
Replacing http_file with a bare rctx.download drops two existing downloader guarantees. Bazel's http_file passes auth = get_auth(ctx, source_urls), so wheel URLs requiring $NETRC / ~/.netrc now fail with 401/403. It also passes the URL-derived default canonical_id; without that, a changed URL with a stale hash can reuse the old repository-cache entry locally and only fail on a clean machine.
Please preserve both behaviors (including the default-canonical-ID environment contract) when moving the download into whl_dist, and add authenticated-index coverage.
| console_scripts = _attr("console_scripts", meta.console_scripts), | ||
| )) | ||
|
|
||
| return rctx.repo_metadata(reproducible = True) |
There was a problem hiding this comment.
This is not always reproducible. Hashless wheels are explicitly supported (uv/private/extension/lockfile.bzl:33-41 and the uv-dep-hashes fixture), and defs.bzl:665 passes an empty sha256 here. The replaced http_file captures download_info.integrity and returns attrs_for_reproducibility when no hash was supplied; this rule discards the download result and always claims reproducibility.
Capture the returned integrity and report it through repo_metadata(attrs_for_reproducibility = ...) for the hashless case (or reject unsupported unpinned input). Otherwise a mutable find-links URL can silently change or leave stale bytes/layout cached without a discovered checksum.
| sha256 = rctx.attr.sha256, | ||
| ) | ||
|
|
||
| meta = extract_install_metadata(rctx, rctx.path(basename), _metadata_directory(basename)) |
There was a problem hiding this comment.
The new repository rule and the central invariant here are not exercised by the tests. The updated analysis tests construct whl_dist build targets from local empty files, while the snapshot updates remove all RECORD-derived topology payload for cffi, cryptography, azure, grpcio, and numpy without replacing it with per-wheel snapshots. The performance workload uses a universal wheel or fetches everything before timing, so it also cannot show lazy sibling fetching.
Please add an e2e case with two platform wheels where the inactive URL is deliberately unavailable and the active target still builds, and retain a representative extracted-metadata/provider snapshot (including namespace/native topology). That proves both “only the selected repo is fetched” and that the moved RECORD extraction still produces the advertised layout.
…platform wheels Each public whl_install repo (@…__pkg__ver) ran _extract_wheel_metadata over every selectable platform wheel at repo-rule time, peeking RECORD and entry_points.txt to derive top-level names, namespace/native layout, and console scripts. Reading a wheel forces fetching it, and the metadata was declared via a whl_files label_list, so resolving one package pulled down its Linux, macOS, Windows, and musl wheels even though only the host's is ever used. Move the peek into a new per-wheel `whl_dist` repo that downloads one wheel and carries its RECORD-derived layout as a PyWheelMetadataInfo provider. whl_install now reads the layout off whichever wheel the select chain resolves to, instead of a basename-keyed dict covering every platform. This restores Bazel's lazy select fetching: only the wheel a configuration actually selects is downloaded and inspected. Building numpy (27 platform wheels) on macOS/arm64 now fetches exactly one wheel instead of all 27. - add whl_install/metadata.bzl: RECORD/entry_points parsing + a composed extract_install_metadata() (moved out of repository.bzl) - add whl_install/dist_repository.bzl: the whl_dist repo rule - add whl_dist build rule + PyWheelMetadataInfo; fold SourceBuiltWheelInfo into it; whl_install reads the provider and drops its 8 string_list_dict attrs - strip extraction, the whl_files/metadata_directory attrs, and the label-mapping helper from whl_install's repo rule - wire the extension/lockfile to whl_dist (@whl__…//:whl) - regenerate uv_hub + e2e snapshots
2 participants
Each public whl_install repo (@…__pkg__ver) ran _extract_wheel_metadata over every selectable platform wheel at repo-rule time, peeking RECORD and entry_points.txt to derive top-level names, namespace/native layout, and console scripts. Reading a wheel forces fetching it, and the metadata was declared via a whl_files label_list, so resolving one package pulled down its Linux, macOS, Windows, and musl wheels even though only the host's is ever used.
Move the peek into a new per-wheel
whl_distrepo that downloads one wheel and carries its RECORD-derived layout as a PyWheelMetadataInfo provider. whl_install now reads the layout off whichever wheel the select chain resolves to, instead of a basename-keyed dict covering every platform. This restores Bazel's lazy select fetching: only the wheel a configuration actually selects is downloaded and inspected.Building numpy (27 platform wheels) on macOS/arm64 now fetches exactly one wheel instead of all 27.
Changes are visible to end-users: no
Test plan