wpng: fix integer overflow in NetPBM size calculations

[Naveed8951]

Summary

This patch fixes an integer overflow vulnerability in contrib/gregbook/wpng.c when computing image buffer sizes from NetPBM header fields (width, height). Crafted inputs could previously trigger under-allocation followed by an out-of-bounds heap write.

Vulnerability Details

wpng derives buffer sizes using arithmetic based on attacker-controlled width and height values, for example:

• rowbytes = width * {1,3,4}
• image_bytes = rowbytes * height

These computations were performed without overflow checks. On overflow, malloc() could allocate a smaller-than-required buffer, while fread() would still read the (overflowed) size, resulting in heap memory corruption.

Both interlaced and non-interlaced decode paths were affected.

Fix

• Switched size computations to size_t

• Added explicit overflow checks for:

  • rowbytes derivation
  • image_bytes calculation
  • row_pointers allocation (height * sizeof(pointer))

• Ensured allocation sizes and fread() sizes are consistent and validated

Malformed inputs with excessive dimensions are now rejected before allocation or I/O.

Impact

• Prevents heap overflow from untrusted NetPBM inputs
• Eliminates a memory-corruption class vulnerability
• Improves robustness of the wpng utility when used in automated pipelines

Testing

• Verified normal behavior with valid P5/P6/P8 inputs
• Confirmed early failure on crafted headers with extreme dimensions
• No ASan/UBSan findings after the fix

This change is limited to input validation and does not alter correct behavior for valid files.