fix: overflow for large bit counts and large document counts in b… - #99
Merged
Conversation
…itpacker (#97) ## What Fixes a panic in `FileSlice::combine_ranges` that occurs when querying segments with high document counts (e.g., >67M docs with 64-bit values). It was introduced in #56. The issue was caused by `u32` integer overflow during `doc_id * num_bits` calculations. This resulted in wrapped-around offsets and invalid ranges where `start > end`, triggering an assertion failure in the file slicing logic. ## How Cast from `u32` to `usize` before multiplication in a variety of locations. Additionally, add an assert in `block_oblivious_range` to provide a clear failure if an out-of-bounds start is requested, preventing invalid ranges from reaching `FileSlice`. ## Tests Adds a test that fails with overflow before the fix.
rebasedming
added a commit
that referenced
this pull request
Feb 10, 2026
Co-authored-by: Stu Hood <stuhood@paradedb.com>
rebasedming
added a commit
that referenced
this pull request
Feb 10, 2026
Co-authored-by: Stu Hood <stuhood@paradedb.com>
rebasedming
added a commit
that referenced
this pull request
Jun 22, 2026
Co-authored-by: Stu Hood <stuhood@paradedb.com> (cherry picked from commit 35502b2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…itpacker (#97)
What
Fixes a panic in
FileSlice::combine_rangesthat occurs when querying segments with high document counts (e.g., >67M docs with 64-bit values). It was introduced in #56.The issue was caused by
u32integer overflow duringdoc_id * num_bitscalculations. This resulted in wrapped-around offsets and invalid ranges wherestart > end, triggering an assertion failure in the file slicing logic.How
Cast from
u32tousizebefore multiplication in a variety of locations.Additionally, add an assert in
block_oblivious_rangeto provide a clear failure if an out-of-bounds start is requested, preventing invalid ranges from reachingFileSlice.Tests
Adds a test that fails with overflow before the fix.