Forward the client's Accept-Encoding to the proxy backend, and handle a brotli response (#265) - #509
Conversation
Previously proxyconn.cpp always rewrote the client's Accept-Encoding
header to 'gzip' before forwarding it to a proxy backend, clobbering
brotli ('br') and any other encoding the client actually requested.
Only forward the client's original header, still defaulting to gzip
when the client sent none, matching the existing backend<->OLS
decompression path (HttpSession::setupGzipFilter handles only
gzip/deflate from upstream).
…espeedtech#265) processHeaderLine2() only looked for gzip and deflate, so a 'br' response from a proxy backend left the request's compression flags untouched and the rest of the response path treated the body as uncompressed. Factor the value matching out into HttpCgiTool::parseContentEncoding() so the same classification can be reused elsewhere, and set UPSTREAM_BR when the backend used brotli.
…edtech#265) setupGzipFilter() never checked whether the body it was about to compress was already encoded. With UPSTREAM_BR now set for a brotli backend response, gz still equalled GZIP_REQUIRED for the usual 'gzip, deflate, br' client, so OLS gzipped brotli bytes and addGzipEncodingHeader() merged the header into 'Content-Encoding: br,gzip'. Bail out early for a brotli body instead and record it via HSF_RESP_BODY_BRCOMPRESSED. That also makes get_resp_buffer_compress_method() report LSI_BR_COMPRESS, which is what the cache module keys its own compression decisions off: cacheHeader() then skips its gzip pass and stores the entry as CEH_BR, and a hit serves 'Content-Encoding: br' from the stored bytes rather than replacing the header with gzip. contentEncodingFixup() had the mirror-image problem: for a client that accepts neither gzip nor br it ran modgzip's zlib decompress filter over whatever encoding was present. Restrict that to gzip and deflate, the only two it can actually decode.
…tech#265) httpcgitooltest.cpp is already in the unit test build but was entirely commented out. Add a case for HttpCgiTool::parseContentEncoding() covering gzip, deflate, none and br, plus the encodings the response filters cannot decode, since a wrong answer there is what leads to a body being compressed twice.
|
@litespeedtech your cache concern is handled now, so you don't have to work it back out of the diff. The reason the cache re-gzipped a brotli body was that
I had to open this as a new PR — GitHub kept refusing to reopen #504 ("Could not open the pull request") even though the branch merges cleanly. Same branch, three commits on top of the original one. |
Closes #265.
OLS overwrites the client's
Accept-Encodingwithgzipbefore proxying, so a backend that can serve brotli never gets the chance to. The first commit drops that rewrite and forwards the header as the client sent it, still falling back togzipwhen the client sent none at all.@litespeedtech pointed out that on its own this breaks cached pages. That's right, and it turned out to break more than the cache — nothing on the proxy response path knew what
brwas.HttpCgiTool::processHeaderLine2()matched onlygzipanddeflatein an upstreamContent-Encoding, so a brotli response left the request's compression flags untouched.HttpSession::setupGzipFilter()then sawgz == GZIP_REQUIREDfor the usualgzip, deflate, brclient and gzipped the brotli bytes, withaddGzipEncodingHeader()merging the header intoContent-Encoding: br,gzip. With the cache on it got worse:cacheHeader()decides whether to compress fromget_resp_buffer_compress_method(), which returnedLSI_NO_COMPRESS, so the module gzipped the brotli body and stored the entry asCEH_GZIP. A hit then setContent-Encoding: gzipover the storedbr, and the client gunzipped its way to garbage.The rest of the commits make that path brotli-aware.
parseContentEncoding()factors the value matching out ofprocessHeaderLine2()and addsbr, settingUPSTREAM_BR.setupGzipFilter()returns early for a brotli body instead of stacking gzip on it, and marks the buffer withHSF_RESP_BODY_BRCOMPRESSED.That flag is what fixes the cache, without touching cache.cpp.
get_resp_buffer_compress_method()now reportsLSI_BR_COMPRESS, socacheHeader()'sneedGzipis false,markReady()stores the entry asCEH_BR, andhandlerProcess()already has the branch that servesContent-Encoding: brfrom the stored bytes.toggleGzipState()already returns early for anything that isn't none or gzip, andbuildCacheKey()already varies onAccept-EncodingthroughgetVaryFlag(), so no new vary dimension is needed either.One more thing fell out of it.
contentEncodingFixup()ran modgzip's decompress filter over whateverContent-Encodingwas present whenever the client accepted neither gzip nor br. That filter is zlib only, so it would inflate brotli — or zstd — into garbage. It's now restricted to gzip and deflate. There is no brotli decoder wired into the response filter chain, so if a backend returns brotli to a client that never asked for it, OLS passes it through untouched rather than mangling it. That's what it already does for any other encoding it can't undo, and it's only reachable if the backend ignores theAccept-Encodingit was sent.I still don't have a working build of this tree here (empty lsquic submodule, brotli and boringssl need the fetch scripts), so I can't claim a compiled or runtime test. What I did check:
parseContentEncoding()compiled and run standalone against the cases in the new unit test, and the two editedHttpSessionbodies re-compiled as a standalone translation unit with stubbed dependencies to catch syntax and arity mistakes. The test case lives intest/http/httpcgitooltest.cpp, which was already in the test build but had every case commented out.