Support compressed package downloads #6069
Open
+70
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Handled automatically by reqwest.
Adjust the BuiltinPackageLoader to accept gzip or zstd compressed responses and decompress them during package download.
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds support for compressed package downloads using gzip and zstd compression, improving download efficiency for WebAssembly packages.
Changes:
- Added gzip and zstd compression support for package downloads in both CLI and WASIX
- Implemented custom decompression logic in WASIX's BuiltinPackageLoader
- Added required dependencies (flate2 and zstd) to handle decompression
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/wasix/src/runtime/package_loader/builtin_loader.rs | Added decode_response_body function to handle gzip/zstd decompression and Accept-Encoding header |
| lib/wasix/Cargo.toml | Added flate2 and zstd dependencies for decompression support |
| lib/cli/src/commands/package/download.rs | Added Accept-Encoding header to request compressed responses from registry |
| lib/cli/Cargo.toml | Enabled zstd feature in reqwest for automatic decompression |
| Cargo.toml | Added zstd to workspace dependencies |
| Cargo.lock | Updated dependency lock file with zstd and related packages |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| .get(download_url) | ||
| .header(http::header::ACCEPT, "application/webc"); | ||
| .header(http::header::ACCEPT, "application/webc") | ||
| // NOTE: reqwest handles gzip/zstd decoding the respone body |
There was a problem hiding this comment.
Spelling errors in comment: "respone" should be "response", and there's an extra space between "decoding" and "the".
Suggested change
| // NOTE: reqwest handles gzip/zstd decoding the respone body | |
| // NOTE: reqwest handles gzip/zstd decoding the response body |
Comment on lines
+329
to
+374
| fn decode_response_body(headers: &HeaderMap, body: Vec<u8>) -> Result<Vec<u8>, anyhow::Error> { | ||
| let encodings = match headers.get(http::header::CONTENT_ENCODING) { | ||
| Some(header) => header | ||
| .to_str() | ||
| .context("invalid content-encoding header")? | ||
| .split(',') | ||
| .map(|encoding| encoding.trim().to_ascii_lowercase()) | ||
| .filter(|encoding| !encoding.is_empty()) | ||
| .collect::<Vec<_>>(), | ||
| None => Vec::new(), | ||
| }; | ||
|
|
||
| if encodings.is_empty() || (encodings.len() == 1 && encodings[0] == "identity") { | ||
| return Ok(body); | ||
| } | ||
|
|
||
| let mut reader: Box<dyn Read> = Box::new(std::io::Cursor::new(body)); | ||
| for encoding in encodings.iter().rev() { | ||
| match encoding.as_str() { | ||
| "gzip" => { | ||
| reader = Box::new(flate2::read::GzDecoder::new(reader)); | ||
| } | ||
| "zstd" => { | ||
| #[cfg(not(target_arch = "wasm32"))] | ||
| { | ||
| reader = Box::new( | ||
| zstd::stream::read::Decoder::new(reader) | ||
| .context("failed to initialize zstd decoder")?, | ||
| ); | ||
| } | ||
| #[cfg(target_arch = "wasm32")] | ||
| { | ||
| bail!("zstd content-encoding is not supported on wasm32"); | ||
| } | ||
| } | ||
| "identity" => {} | ||
| other => bail!("unsupported content-encoding: {other}"), | ||
| } | ||
| } | ||
|
|
||
| let mut decoded = Vec::new(); | ||
| reader | ||
| .read_to_end(&mut decoded) | ||
| .context("failed to decode response body")?; | ||
| Ok(decoded) | ||
| } |
There was a problem hiding this comment.
The new decode_response_body function lacks test coverage. Consider adding tests to verify:
- Decoding of gzip-encoded responses
- Decoding of zstd-encoded responses (on non-wasm32 platforms)
- Handling of multiple encodings (e.g., "gzip, zstd")
- Handling of the "identity" encoding
- Error handling for unsupported encodings
- Error handling for invalid content-encoding headers
marxin
reviewed
Jan 15, 2026
Contributor
marxin
left a comment
marxin
left a comment
There was a problem hiding this comment.
Can we write a test that will test the behavior?
| // NOTE: gzip and zstd decoding is available on native platforms. | ||
| // In browser platforms, the fetch implementation should automatically | ||
| // handle decoding of gzip/zstd responses transparently. | ||
| headers.insert(http::header::ACCEPT_ENCODING, "gzip, zstd".parse().unwrap()); |
Contributor
There was a problem hiding this comment.
Should we use the qvalue weights to prefer ZSTD if possible?
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Encoding#q
| None => Vec::new(), | ||
| }; | ||
|
|
||
| if encodings.is_empty() || (encodings.len() == 1 && encodings[0] == "identity") { |
Contributor
There was a problem hiding this comment.
exactly_one from Itertools?
Contributor
Author
There was a problem hiding this comment.
at_most_one might work well.
marxin
reviewed
Jan 15, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for compressed package downloads, both in the CLI and
in WASIX.
Uses the builtin reqwest support in the CLI, and custom decoding logic in
the WASIX BuiltinPackageLoader.
Resolves: #6060