Add compress_for_servers build option for Web (and add opts.get_arguments())

[adamscott]

Note

This PR is the first of (maybe) two that intends to apply some optimizations underlined by @popcar2 in his great blog post on How to Minify Godot's Build Size (93MB -> 6.4MB exe).

compress_for_servers build option

This PR primarily adds the new compress_for_servers build option. See Compression in HTTP (MDN) for more information (tl;dr, it permits the server to send a compressed file instead of the raw file, for the receiver to decompress it seamlessly, saving bandwidth). It is configured as a ListVariable, which seem to be the first time it's used in the project. It accepts those values:

• gzip (alias gz): gzips web built files
• brotli (alias br): brotli compress built files
• none (default): no compression of built files
• all: will apply all compression formats to built files

ListVariable is interesting because it permits the use of multiple values as entry. compress_for_servers=all is the dynamic equivalent (handled by ListVariable) of compress_for_servers=gzip,brotli.

Note

A new PR (or this one) needs to follow-up in order to add those server compressed files into the official build templates.

opts.get_arguments()

Incidentally, in order to fix some issues with SCons.Variables.ListVariable against our current opts.Update() use in SConstruct, this PR also introduces opts.get_arguments() as a new Variables class method. It returns opts values as if they were entered by hand in the command line. Otherwise, the new ListVariable gets updated twice and the result isn't idempotent, and the ListVariable validator don't recognize it's own values. This change was made to prevent this.

This PR also cleans up (a lot) create_template_zip() in platform/web/emscripten_helpers.py. It simplifies adding files to in_files and out_files, no need anymore to track manually each file that are added to add their zipfile counterpart.

[adamscott]

Brotli is ungodly slow, as it is single-threaded. I'm wondering if it would be preferrable to cook up some magic in order to run instead rust-brotli. The problem here is that there's no binary to download, as it doesn't exist (yet).

[adamscott]

Brotli is ungodly slow, as it is single-threaded. I'm wondering if it would be preferrable to cook up some magic in order to run instead rust-brotli. The problem here is that there's no binary to download, as it doesn't exist (yet).

I just tried to compile and run rust-brotli from the command line, and it works. But it's not multi threaded. :\ (the library supports multi threading, but the command line API doesn't.)

[rburing]

I just tried to compile and run rust-brotli from the command line, and it works. But it's not multi threaded. :\ (the library supports multi threading, but the command line API doesn't.)

Doesn't it have a -j argument?

[adamscott]

I just tried to compile and run rust-brotli from the command line, and it works. But it's not multi threaded. :\ (the library supports multi threading, but the command line API doesn't.)

Doesn't it have a -j argument?

Dang it. Didn't know. I'll try it out.

[adamscott]

So, the results of my benchmarks.

40MiB wasm file

Binary	-j	Time
google/brotli	n/a (-j1)	0m58s
dropbox/rust-brotli	-j1 (default)	7m14s
dropbox/rust-brotli	-j12	0m24s

695MiB wasm file

Binary	-j	Time
google/brotli	n/a (-j1)	24m06s
dropbox/rust-brotli	-j1 (default)	> 72m30s (stopped it)
dropbox/rust-brotli	-j12	27m39s

Result

The results are inconclusive, so I will not support dropbox/rust-brotli.

[Calinou]

If you can find a way to efficiently decompress Zstandard on the web platform, supporting it could be worthwhile. All modern browsers except Safari support it: https://caniuse.com/zstd

In fact, it might not be worth supporting Brotli anymore given its slow compression speeds, as gzip will remain available for Safari until it adopts Zstandard.

Edit: zstd --ultra -22 doesn't compress as well as brotli --best from my testing though: #105388 (review)