Zlib: brotli compression is much slower than gzip

[csvan]

Version

16.10.0

Platform

OSX, Linux (CentOS 8)

Subsystem

zlib

What steps will reproduce the bug?

Full test suite: https://github.com/timotejroiko/zlib-benchmark (not authored by my team, but is comprehensive and reaches the same conclusion).

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

Brotli compression with zlib should be on par with gzip

What do you see instead?

Brotli compression is much slower than gzip

Additional information

We have replicated this issue on live production systems where we use zlib to perform over-the-air compression of static files. The (proprietary) code checks the accept-encoding header on client requests, and compresses accordingly (using default zlib settings).

The system originally prioritised brotli over gzip since the former is more efficient, however, we noticed a massive spike in CPU usage when doing so. After switching priority to gzip, CPU usage was at a minimum - for the same files and general load conditions.

We do not know if this is expected, but the discrepancy in performance is so large that it effectively disqualifies brotli for a large number of use cases involving high workloads.

[targos]

The results don't seem unexpected to me. The brotli repository points to the following site for benchmarks, where the difference between brotli and gzip is similar: http://mattmahoney.net/dc/text.html

[philz]

Both Gzip and Brotli have different "compression" settings. I believe Brotli defaults to maximum compression, whereas gzip doesn't. In my experiments, when I tuned Brotli to more middle-of-the-road compression, the timings where separate, but brotli still won on the overall space-time curve.

[patrickdk77]

I had to drop using brotli, in my tests node gzip takes 0.664s to compress 108megs of data, and the gzip cli takes 0.884s
when using brotli with nodejs it takes 29.4s (using node v18 or v22) and the brotli cli v1.0.9 takes 1.87s

This is with a 108meg json text

All the tests above where done with 9 as the compression setting