Fix size_t overflow in encoder CreatePreparedDictionaryWithParams (32-bit)

[0xazanul]

Summary

• Fix integer overflow in CreatePreparedDictionaryWithParams() (c/enc/compound_dictionary.c) where sizeof(uint32_t) * source_size wraps size_t on 32-bit platforms when source_size approaches 2^30 bytes
• The undersized allocation leads to heap buffer overflow in subsequent hash table writes

The Bug

Lines 22–26 compute alloc_size as a single expression:

size_t alloc_size = (sizeof(uint32_t) << slot_bits) +
    (sizeof(uint32_t) << slot_bits) +
    (sizeof(uint16_t) << bucket_bits) +
    (sizeof(uint32_t) << bucket_bits) +
    (sizeof(uint32_t) * source_size);  // overflows on 32-bit

On 32-bit platforms (size_t = 32 bits), the last term 4 * source_size overflows when source_size >= 2^30 (~1 GB). When alloc_size wraps to a small value:

• BROTLI_ALLOC allocates an undersized buffer
• BROTLI_IS_NULL is a no-op in non-analyzer builds (#define BROTLI_IS_NULL(A) (!!0))
• Subsequent writes to slot_size, slot_limit, num, bucket_heads, and next_bucket overflow the heap buffer

Impact

• DoS: Crash on 32-bit platforms with a ~1 GB dictionary
• Heap buffer overflow: If malloc returns a valid pointer for the wrapped size, all subsequent writes go out of bounds — potential code execution on 32-bit targets

Requires a 32-bit platform and a large (~1 GB) dictionary passed to the encoder via BrotliEncoderPrepareDictionary().

The Fix

Split alloc_size into a fixed header_size (slot/bucket tables) and the variable source_size-dependent term. Before combining, check that the multiplication and addition do not overflow size_t:

if (source_size > (BROTLI_SIZE_MAX - header_size) / sizeof(uint32_t)) {
  return NULL;
}
alloc_size = header_size + (sizeof(uint32_t) * source_size);

Uses the existing BROTLI_SIZE_MAX macro from brotli/types.h. Returns NULL (matching existing error-return convention) when the size would overflow.

Note

This is distinct from the decoder-side total_size overflow fixed in #1450. That fix addressed AttachCompoundDictionary() in c/dec/decode.c; this fix addresses the encoder-side allocation in c/enc/compound_dictionary.c.

Test plan

• Verify compilation on 32-bit and 64-bit targets with -Wall -Wextra
• On 64-bit, behavior is unchanged (overflow check is always false since size_t is 64 bits)
• On 32-bit, CreatePreparedDictionary() with source_size >= 2^30 returns NULL instead of overflowing

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[eustas]

Hi. Let's fix that on higher level. Dictionaries larger than ~64MiB are oxymōrum. With some generous amount forgiveness we can reject dictionaries larger than 256MiB (1<<28). So:

if (source_size > (1u<<28)) return NULL;

in CreatePreparedDictionary would be sufficient. Preferably with comment to calculate exact limit and make it a constant rather than magic number.

[0xazanul]

Thanks for the review! That makes sense capping at the protocol level is cleaner than guarding the arithmetic. I'll update the PR with:

/* Brotli cannot reference dictionary data beyond 64 MiB (1 << 26).
Accept up to 256 MiB (1 << 28) as a generous upper bound. */
#define BROTLI_MAX_PREPARED_DICTIONARY_SIZE (1u << 28)

if (source_size > BROTLI_MAX_PREPARED_DICTIONARY_SIZE) return NULL;

Will also double-check the exact distance limit from the spec to tighten the constant if appropriate.

[0xazanul]

Updated the PR with the changes you suggested:

• Replaced the arithmetic overflow guard with a protocol-level size cap at 256 MiB (1 << 28)
• Added BROTLI_MAX_PREPARED_DICTIONARY_SIZE as a named constant with a comment explaining the rationale
• Returns NULL early if source_size exceeds the limit

Let me know if anything else needs adjusting!