Replace astral-tokio-tar with tar-codec - #19979
Conversation
Merging this PR will improve performance by ×2.1
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | unpack_sdist_many_files |
731 ms | 326.7 ms | ×2.2 |
| ⚡ | Simulation | unpack_sdist_many_files |
333.4 ms | 176.6 ms | +88.8% |
Tip
Curious why this is faster? Use the CodSpeed MCP and ask your agent.
Comparing ww/tar-codec (2dcf539) with main (3a9573c)
Footnotes
-
6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
Is the intent to do this behind preview first, or? |
I didn't have a strong intent either way 🙂. I guess we probably should? |
|
I would personally be ok to do this outside of preview, as long as we have sufficient testing, since it's not clear that we want to change any specific behavior in response to user feedback, etc. |
|
I think the main issue is that it will be much more restrictive. I guarantee we will break people. |
|
That makes sense, I forgot that this would actually be stricter. I’m a little worried that we won’t get enough feedback to know when we can stabilize it. |
|
I think another benefit of having a preview code path is that when we stabilize it we can trivially retain the old code path to allow opt-in to legacy when we change the default, which would give us the signal we need for long-term removal. Alas that lengthens the period of time in which we have two implementations for differentials :) |
|
Yeah, I think I lean towards a preview feature here. I'll get this into a working shape and then massage the old tokio-tar paths back into place. |
e5b51ab to
aa3e109
Compare
uv test inventory changesThis PR changes the tests when compared with the
|
A `.whl.tar.zst` member is recorded before traversal containment is enforced, so a skipped `../` member can survive `RECORD` healing and cause uninstall to remove an unrelated environment executable. Collect only members that were successfully unpacked. This overlaps the tar-wheel extraction work in [#19979](#19979).
Signed-off-by: William Woodruff <william@yossarian.net> Tighten extraction policy Signed-off-by: William Woodruff <william@yossarian.net> Bump tar-codec Signed-off-by: William Woodruff <william@yossarian.net> Cheeky clippy fix Signed-off-by: William Woodruff <william@yossarian.net> Tweak our decoding policy Signed-off-by: William Woodruff <william@yossarian.net> Fix some snapshots Signed-off-by: William Woodruff <william@yossarian.net> Add an offset filter This offset ranges by platform. Signed-off-by: William Woodruff <william@yossarian.net> Bump tar-codec Signed-off-by: William Woodruff <william@yossarian.net> Migrate the write path Signed-off-by: William Woodruff <william@yossarian.net> Remove a stale test Signed-off-by: William Woodruff <william@yossarian.net> Re-add both tar backends Signed-off-by: William Woodruff <william@yossarian.net> Bump snapshot Signed-off-by: William Woodruff <william@yossarian.net> Update some snapshots Signed-off-by: William Woodruff <william@yossarian.net> Fix snapshots Signed-off-by: William Woodruff <william@yossarian.net> Fix benchmarking Signed-off-by: William Woodruff <william@yossarian.net> Undo benchmark naches Signed-off-by: William Woodruff <william@yossarian.net> Bump tar-codec, use the new finalization API Signed-off-by: William Woodruff <william@yossarian.net> Briefer description Signed-off-by: William Woodruff <william@yossarian.net> Fix snapshot Signed-off-by: William Woodruff <william@yossarian.net> Bump snapshots Signed-off-by: William Woodruff <william@yossarian.net> Fix snapshot Signed-off-by: William Woodruff <william@yossarian.net> Clippy Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
| // as raw binary. | ||
| let decode_policy = DecodePolicy::default().pax_policy( | ||
| PaxDecodePolicy::default() | ||
| .allow_unknown_pax_vendor_records(true) |
There was a problem hiding this comment.
Summary
WIP, I've done the read path but not the (full) write path yet.This adds a preview feature that replaces astral-tokio-tar with tar-codec on both our read and write paths.
Notes:
uv buildif the cache dir is enclosed #19991 first to seal of the kind of failure thatbuild::build_with_hardlinkis currently testing indirectlyTest Plan
Ensure existing tests don't break, plus new tests.