refactor(pm): use libdeflater for gzip decompression

[elrrrrrrr]

• 🔒 Concurrency: Add OnceMap for download deduplication
• 🧹 Code cleanup in downloader.rs
• 🤖 Simplify Linux clone implementation with spawn_blocking
• ⚡ Simplify tokio runtime configuration

[gemini-code-assist]

Summary of Changes

Hello @elrrrrrrr, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request undertakes a significant refactoring of the package downloader module to enhance its overall performance and resource efficiency. The changes focus on optimizing the decompression and extraction of gzipped tarballs by leveraging more performant libraries and parallelizing file system operations. This leads to a more robust and faster download process, crucial for efficient package management.

Highlights

• Performance Improvement: Switched gzip decompression from async-compression to libdeflater for significantly faster decompression, and refactored tarball extraction to use synchronous tar crate with rayon for parallel file writing, offloading CPU-intensive tasks to tokio::task::spawn_blocking.
• Resource Management: Introduced a global buffer pool for decompression to reduce memory allocation overhead and improve efficiency during repeated extractions.
• Concurrency Control: Implemented a OnceMap to ensure that each unique URL is downloaded and extracted only once, even when multiple concurrent requests are made for the same resource.
• Platform Compatibility: Added Windows-specific path sanitization during extraction to handle invalid characters and prevent file system errors on Windows.
• Observability: Integrated performance metrics tracking for download, decompression, and write operations to provide better insights into the downloader's performance characteristics.

Changelog

• crates/pm/Cargo.toml
  • Removed async-compression and tokio-tar dependencies.
  • Added libdeflater, bytes, and rayon dependencies.
  • Updated tokio-util dependency to remove the io feature.
• crates/pm/src/util/downloader.rs
  • Replaced async-compression and tokio-tar with libdeflater and tar for decompression and archiving.
  • Introduced OnceMap for download concurrency control, replacing the DashMap based locking mechanism.
  • Implemented a global buffer pool for Vec<u8> to optimize memory usage during decompression.
  • Separated the download (network) and extraction (CPU/IO) phases into distinct download_bytes and extract_and_write functions.
  • Moved CPU-bound decompression and tar parsing to tokio::task::spawn_blocking to prevent blocking the async runtime.
  • Utilized rayon for parallel file writing during the extraction process.
  • Added sanitize_path_for_windows for cross-platform path compatibility, specifically addressing Windows path restrictions.
  • Integrated performance statistics tracking for download, decompress, and write operations.
  • Removed the size field from the ExtractedEntry struct, as content length can be derived.
  • Updated unit tests to reflect the new download_bytes and extract_and_write API.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors the downloader to use libdeflate for faster gzip decompression, which is a great performance improvement. The code is well-structured, separating network, decompression, and file writing logic. The introduction of a buffer pool is also a nice touch.

However, I've found a few issues that need to be addressed:

• A critical bug in the OnceMap implementation that could lead to packages not being extracted.
• Several places where errors are silently ignored, which could make debugging difficult.
• A test that seems to be broken after the refactoring.

I've left detailed comments on these points below.

[elrrrrrrr]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant refactoring of the package downloader to improve performance by replacing async-compression and tokio-tar with the much faster libdeflate. The new implementation shifts from a streaming model to an in-memory batch processing approach, which is a valid trade-off for performance in this context. The code is well-structured, utilizing a OnceMap for concurrent downloads, a buffer pool to reduce allocations, and rayon for parallel file I/O.

My review focuses on the robustness and clarity of the new implementation. I've identified a potential issue where the decompression logic might fail for files with very high compression ratios and suggested a more robust retry mechanism. I also pointed out a small piece of redundant code that can be removed to improve clarity. Overall, this is a high-quality performance enhancement.

[xusd320]

这叫files 不合适

[xusd320]

串行阻塞了。考虑多线程复制

[elrrrrrrr]

嗯嗯，现在就是同步同步 fs 放在 spawn_blocking 里去执行，性能会比之前 async 异步 api 好一些。
还试过使用 rayon 来并发写入，这个场景性能提升不明显没有采用