avoid rounding boundary case in color quantization - #125
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(bug fix thanks to Arthur Chan)
AI summary:
crawl_decision_tree's partition loop assumes the slice's mean is a true separator: every value below it in pPalette, at most one above. That mean was computed in get_mean using single-precision float accumulators. For a slice whose weighted sum exceeds float's 24-bit exact-integer range (2^24), the accumulated sum loses low-order bits, and the final division can round the mean below every value in the slice. When that happens, the descending pointer k in the partition loop never finds a value at or below the pivot and walks past idxMin, underflowing the uint32_t index. The resulting child node claims range [idxMin, UINT32_MAX], and any subsequent walk of it (e.g. get_variance) is an unbounded out-of-bounds read, reachable at cgif_rgb_addframe with attacker-controlled pixel data.
Fix: compute the mean and variance in double throughout. sum/m in get_mean stay bounded by 255 * numPixel, and numPixel is bounded by the uint16_t width/height limit (~4.3 * 10^9), so all sums remain exactly representable in double (2^53). The resulting division is a single correctly-rounded IEEE-754 operation, with an error several orders of magnitude smaller than the smallest possible true distance from the mean to an integer boundary. This removes the rounding-direction failure mode structurally, not just empirically.
Changed:
pImageDataRGBfloat (the dithering error-diffusion buffer) is intentionally left as float: it holds individual pixel values (0-255, exact in float), not an accumulating sum, so it isn't part of this bug.