We now have support for compatibility-version that allows us to more easily make changes that change the checksum of the result artifact without breaking reproducible builds. This allows as to look at more performant compression options to speed up image layer creation, even if they produce slightly different artifacts.
Benchmarks in https://tonistiigi.github.io/compression-bench/
Gzip: klauspost-pgzip clear winner for gzip compression
Zstd: no significant performance difference between implementations (at least in CI)
Decompress: pigz cli wins, but gains are minimal (we already use this if pigz is available)
So obvious suggestion would be to switch to klauspost-pgzip. Problem though is that we export layers in parallel and don't have much control over limits or overallocation in this process. kp-pgzip gets its speed increase by doing much more memory allocations and then processing these in parallel from memory. From bench you can also see that there isn't much difference between using pgzip sequentially or in parallel for multiple layers. This means that when we process multiple layers in parallel, we allocate much more memory with no real performance benefit and in big builds could possible eventually run out of memory.
I'm not sure how serious this problem is in practice. Maybe it would be possible to extend kp-pgzip to control memory allocation across multiple writers.
The bench also shows that all the implementations seem to be deterministic, that is important for our reproducible builds. kp-pgzip looks deterministic in any concurrency level until you use the same fixed blocksize.
@fiam @AkihiroSuda
We now have support for
compatibility-versionthat allows us to more easily make changes that change the checksum of the result artifact without breaking reproducible builds. This allows as to look at more performant compression options to speed up image layer creation, even if they produce slightly different artifacts.Benchmarks in https://tonistiigi.github.io/compression-bench/
Gzip: klauspost-pgzip clear winner for gzip compression
Zstd: no significant performance difference between implementations (at least in CI)
Decompress:
pigzcli wins, but gains are minimal (we already use this ifpigzis available)So obvious suggestion would be to switch to
klauspost-pgzip. Problem though is that we export layers in parallel and don't have much control over limits or overallocation in this process. kp-pgzip gets its speed increase by doing much more memory allocations and then processing these in parallel from memory. From bench you can also see that there isn't much difference between usingpgzipsequentially or in parallel for multiple layers. This means that when we process multiple layers in parallel, we allocate much more memory with no real performance benefit and in big builds could possible eventually run out of memory.I'm not sure how serious this problem is in practice. Maybe it would be possible to extend kp-pgzip to control memory allocation across multiple writers.
The bench also shows that all the implementations seem to be deterministic, that is important for our reproducible builds. kp-pgzip looks deterministic in any concurrency level until you use the same fixed blocksize.
@fiam @AkihiroSuda