Heap buffer overflow in LZW byte-list allocation

[Oculytic]

Summary

Integer overflow in LZW_GenerateStream causes undersized heap allocation for the LZW byte-list buffer. Subsequent writes by create_byte_list overflow the buffer, resulting in heap corruption.

Vulnerability

src/cgif_raw.c, lines 352–353:

uint64_t MaxByteListLen = MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull;
uint64_t MaxByteListBlockLen = MAX_CODE_LEN * lzwPos * (BLOCK_SIZE + 1ull) / 8ull / BLOCK_SIZE + 2ull + 1ull + 1ull;

MAX_CODE_LEN is int (value 12). lzwPos is uint32_t. The multiplication MAX_CODE_LEN * lzwPos is evaluated as int * uint32_t, which is uint32_t, and wraps when lzwPos > UINT32_MAX / 12 ~ 357,913,941. The wrapped result is then promoted to uint64_t for the /8ull division, but the damage is already done.

Trigger Conditions

1. Image with width * height > ~358 million pixels (e.g. 65535*10923)
2. Pixel data must be sufficiently non-compressible (varied values) so that lzwPos (number of LZW codes generated) exceeds the threshold
3. Any API path (raw or RGB) - all frames go through LZW encoding

Impact

Heap buffer overflow (write). create_byte_list writes the full LZW stream into the undersized byteList buffer, overwriting adjacent heap memory.

Reproduction

I've attached a harness testcase that can be used to trigger the bug (Same harness is attached to another bug as well). It can be compiled and run with:

cc -g -fsanitize=address -I inc harness.c \
   src/cgif.c src/cgif_raw.c src/cgif_rgb.c -o harness -lm
printf '\xff\xff\xab\x2a\x00\xDE\xAD\xBE\xEF' | ./harness

ASan reports:

heap-buffer-overflow in create_byte_list cgif_raw.c:251
allocated by LZW_GenerateStream cgif_raw.c:354

Patch

--- a/src/cgif_raw.c
+++ b/src/cgif_raw.c
@@ -349,8 +349,8 @@
   // pack the generated LZW data into blocks of 255 bytes
   uint8_t *byteList; // lzw-data packed in byte-list
   uint8_t *byteListBlock; // lzw-data packed in byte-list with 255-block structure
-  uint64_t MaxByteListLen = MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull; // conservative upper bound
-  uint64_t MaxByteListBlockLen = MAX_CODE_LEN * lzwPos * (BLOCK_SIZE + 1ull) / 8ull / BLOCK_SIZE + 2ull + 1ull +1ull; // conservative upper bound
+  uint64_t MaxByteListLen = (uint64_t)MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull;
+  uint64_t MaxByteListBlockLen = (uint64_t)MAX_CODE_LEN * lzwPos * (BLOCK_SIZE + 1ull) / 8ull / BLOCK_SIZE + 2ull + 1ull + 1ull;
   byteList      = malloc(MaxByteListLen);
   byteListBlock = malloc(MaxByteListBlockLen);

Casting MAX_CODE_LEN to uint64_t ensures the multiplication is performed in 64-bit arithmetic, preventing the wrap.

Harness: harness.c

Credit

Oculytic