Feature/builtin compression#19337
Merged
memsharded merged 31 commits intoconan-io:develop2from Jan 14, 2026
Merged
Conversation
This change adds zstd support to conan in the following ways: 1. The person or build running `conan upload` can set a config value core.upload:compression_format = zstd to upload binaries using zstd instead of gzip. 2. The zstd compression is done entirely in Python using a combination of tarfile and python-zstandard. Then the file is uploaded as normal. 3. When downloading packages, if a .tar.zst file is encountered, the extraction code uses tarfile and python-zstandard to extract. I chose python-zstandard as the library because that is what urllib3 uses.
Because zstd decompression is expected to just work if the server has a .tar.zst file, I am including zstandard in requirements.txt. https://python-zstandard.readthedocs.io/en/latest/projectinfo.html#state-of-project recommends that we "Pin the package version to prevent unwanted breakage when this change occurs!", although I doubt that much will change before an eventual 1.0.
CI is unable to find 0.21.0
1. Change requirements.txt to allow either zstandard 0.20 or 0.21. That prevents a downgrade for people who already have 0.21 installed, while also allowing CI to find 0.20. 2. Move compressformat parameter earlier in compress_files() function. It made a bit more sense to have it earlier; as long as consumers are correctly using positional kwargs, it shouldn't break anyone.
1. Fix bad merge causing uploader.py change to still refer to `self._app.cache.new_config`, when now we are supposed to use `self._global_conf`. 2. Change two output calls in uploader.py to only output the package file basename to be consistent with other existing log lines. 3. Use double quotes instead of single quotes to be more consistent with existing code.
1. Downgrade bufsize to 32KB because that performs well for compression and decompression. The values don't need to be the same, but it happened to be the best value in both compression and decompression tests. 2. Use a context manager for stream_reader as I do for stream_writer. 3. Add some comments about the bufsize value.
Still need to do some testing though.
Newer Python has this warning: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior
czoido
reviewed
Dec 3, 2025
czoido
reviewed
Dec 3, 2025
czoido
reviewed
Dec 3, 2025
Contributor
|
Maybe we should try to raise a more meaningful error for the cases that you uploaded a package with zstd for example and try to use if in a Python < 3.14. Right now it would be something like: Also there's the case of a Client < 2.24 trying to use that package, but for this one I think we can only make a clear documentation about that this case could happen? |
memsharded
commented
Dec 5, 2025
conan/internal/api/uploader.py
Outdated
| # No compressed file exists, need to compress | ||
| compressformat = self._global_conf.get("core.upload:compression_format", | ||
| default="gz", choices=COMPRESSIONS) | ||
| if compressformat == "zst" and sys.version_info.minor < 14: |
Member
Author
There was a problem hiding this comment.
Add risk warnings because of experimental
This was referenced Dec 9, 2025
czoido
reviewed
Jan 9, 2026
conan/internal/api/uploader.py
Outdated
Comment on lines
+93
to
+96
| ConanOutput().warning(f"The {compressformat} compression is highly experimental, " | ||
| f"use it at your own risk and expect issues. Feedback welcome, " | ||
| f"please report it as Github tickets", | ||
| warn_tag="risk") |
Contributor
There was a problem hiding this comment.
Maybe we can be more specific about the issues you may have? Like:
The {compressformat} compression is experimental.
Consumers installing packages uploaded with this format must use Python >= 3.14.
Consumers using older Python versions will not be able to install these packages.
Feedback is welcome, please report any issues as GitHub tickets.
And also drop the use it at your own risk part? The warning is already marked as risk level.
czoido
approved these changes
Jan 9, 2026
Contributor
|
I noticed today that urllib3 falls back on using backports.zstd as of urllib3/urllib3#3697 to allow pre-3.14 to still support zstd compression. Is that something that Conan can do too? |
Member
Author
It is not planned at the moment, it would add extra variability and complexity, and dependency on wheels binaries for zstd, that might not be available in all the different platforms that Conan is running on, with the extra added maintenance, support, testing, docs, etc. Which will add on top of the feature itself. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog: Feature: (Experimental) Support built-in
xzandzstdcompression for Conan artifacts.Docs: conan-io/docs#4355
Close #14706
Close #18314
Close #648
Also for cache save/restore