Prove compression stream finalization

Author: eldryoth

CHANGELOG.md

@@ -84,8 +84,9 @@ behavior when the change improves security or project direction.
   values, and require trusted immutable MMDB file paths during loading.
 - Pin publishing actions, security-tool installs, and container base images to
   immutable reviewed versions and digests.
-- Bound Brotli, gzip, and Zstandard output before allocation growth, avoid a
-  second body copy when draining codec buffers, and discard failed encoders.
+- Bound Brotli, gzip, and Zstandard logical output before accepting excess
+  encoded bytes, avoid a second body copy when draining codec buffers, and
+  discard failed encoders.
 - Fail malformed `Accept-Encoding` negotiation closed, honor explicit coding
   rejection over wildcard acceptance, and treat qualified `private` cache
   directives as compression-blocking security policy.

crates/fluxheim-compression/src/compression_tests.rs

@@ -5,7 +5,7 @@ use super::{
 };
 
 #[cfg(any(feature = "brotli", feature = "gzip", feature = "zstd"))]
-use std::io;
+use std::io::{self, Read as _};
 
 #[cfg(any(feature = "brotli", feature = "gzip", feature = "zstd"))]
 use bytes::Bytes;
@@ -97,6 +97,40 @@ fn gzip_encoder_releases_retained_output_between_chunks() {
     assert_eq!(gzip.get_ref().buffer_len(), 0);
 }
 
+#[cfg(feature = "gzip")]
+#[test]
+fn gzip_encoder_finalizes_decodable_multi_chunk_stream() {
+    let compressed = encode_multi_chunk_stream(ResponseCompressionEncoder::gzip(4, 64 * 1024));
+    let mut decoded = Vec::new();
+    flate2::read::GzDecoder::new(compressed.as_slice())
+        .read_to_end(&mut decoded)
+        .unwrap();
+
+    assert_eq!(decoded, multi_chunk_plaintext());
+}
+
+#[cfg(feature = "brotli")]
+#[test]
+fn brotli_encoder_finalizes_decodable_multi_chunk_stream() {
+    let compressed = encode_multi_chunk_stream(ResponseCompressionEncoder::brotli(5, 64 * 1024));
+    let mut decoded = Vec::new();
+    brotli::Decompressor::new(compressed.as_slice(), 4096)
+        .read_to_end(&mut decoded)
+        .unwrap();
+
+    assert_eq!(decoded, multi_chunk_plaintext());
+}
+
+#[cfg(feature = "zstd")]
+#[test]
+fn zstd_encoder_finalizes_decodable_multi_chunk_stream() {
+    let compressed =
+        encode_multi_chunk_stream(ResponseCompressionEncoder::zstd(3, 64 * 1024).unwrap());
+    let decoded = zstd::stream::decode_all(compressed.as_slice()).unwrap();
+
+    assert_eq!(decoded, multi_chunk_plaintext());
+}
+
 #[cfg(feature = "gzip")]
 #[test]
 fn gzip_limit_failure_discards_encoder() {
@@ -139,3 +173,25 @@ fn assert_limit_failure_discards_encoder(encoder: &mut ResponseCompressionEncode
     assert_eq!(error.io_kind(), Some(io::ErrorKind::InvalidData));
     assert!(encoder.encode_chunk(Some(&input), true).unwrap().is_empty());
 }
+
+#[cfg(any(feature = "brotli", feature = "gzip", feature = "zstd"))]
+fn encode_multi_chunk_stream(mut encoder: ResponseCompressionEncoder) -> Vec<u8> {
+    let mut compressed = Vec::new();
+    compressed.extend_from_slice(
+        &encoder
+            .encode_chunk(Some(b"first response segment; "), false)
+            .unwrap(),
+    );
+    compressed.extend_from_slice(
+        &encoder
+            .encode_chunk(Some(b"second response segment; "), false)
+            .unwrap(),
+    );
+    compressed.extend_from_slice(&encoder.encode_chunk(None, true).unwrap());
+    compressed
+}
+
+#[cfg(any(feature = "brotli", feature = "gzip", feature = "zstd"))]
+fn multi_chunk_plaintext() -> &'static [u8] {
+    b"first response segment; second response segment; "
+}

docs/compression.md

@@ -88,8 +88,8 @@ The first codec implementation is intentionally bounded:
 - encoded output must stay within `compression.max_output_bytes`;
 - `compression.max_input_bytes` is capped at 64 MiB by config validation;
 - `compression.max_output_bytes` is capped at 128 MiB by config validation;
-- every codec writes through a bounded sink that rejects output before buffer
-  growth would cross `compression.max_output_bytes`;
+- every codec writes through a bounded sink that rejects a write before the
+  logical encoded length would cross `compression.max_output_bytes`;
 - emitted chunks transfer their owned allocation into `Bytes` without copying,
   and a codec is discarded permanently after an output or allocation failure;
 - gzip levels are restricted to `0..=9`;
@@ -101,6 +101,13 @@ The first codec implementation is intentionally bounded:
 A later implementation may add bounded compression worker pools, per-vhost
 concurrency, and precompressed static asset variants.
 
+`max_output_bytes` is an encoded-byte limit, not an exact per-request RSS
+ceiling. The Rust allocator may reserve more `Vec` capacity than the current
+logical length, and each codec has internal CPU and working-memory costs that
+are separate from its output sink. Use route/vhost concurrency controls to
+bound aggregate request pressure until a dedicated compression worker pool is
+available.
+
 ## Cache Integration
 
 Future shared-cache compression variants must be cache-isolated by:

docs/config-reference.md

@@ -2006,10 +2006,12 @@ response rather than enabling an encoding.
 
 `min_bytes` and `max_input_bytes` bound the original response size.
 `max_output_bytes` bounds the encoded response size with a codec sink that
-rejects writes before allocating beyond the limit. Emitted output allocations
-are transferred without a second body copy, and an encoder is discarded after
-any output-limit or allocation failure. The configured input maximum cannot
-exceed 64 MiB and the output maximum cannot exceed 128 MiB.
+rejects a write before its logical output length exceeds the limit. It is not
+an exact RSS ceiling: the allocator may reserve additional `Vec` capacity and
+codec-internal working memory is separate. Emitted output allocations are
+transferred without a second body copy, and an encoder is discarded after any
+output-limit or allocation failure. The configured input maximum cannot exceed
+64 MiB and the output maximum cannot exceed 128 MiB.
 `gzip_level` must be between `0` and `9`, `zstd_level` between `1` and `19`,
 and `brotli_quality` between `0` and `11`.
 

release-notes/RELEASE_NOTES_1.7.7.md

@@ -94,9 +94,10 @@ deterministic unsupported-call rejection.
   descriptor identity and reject config files modified during bounded reads.
 - Restore the all-feature config security suite, including tracing/privacy and
   dual-FIPS-backend feature combinations.
-- Bound Brotli, gzip, and Zstandard output before allocation growth, transfer
-  emitted codec buffers into response bytes without copying, and permanently
-  discard a codec after an output or allocation failure.
+- Bound Brotli, gzip, and Zstandard logical output before accepting excess
+  encoded bytes, transfer emitted codec buffers into response bytes without
+  copying, and permanently discard a codec after an output or allocation
+  failure.
 - Fail response compression closed for malformed `Accept-Encoding` fields,
   honor explicit coding rejection over wildcard acceptance, and suppress
   compression for qualified `Cache-Control: private="..."` responses.