perf: Optimize TermSet for very large sets of terms. - #75
Merged
Conversation
rebasedming
approved these changes
Oct 25, 2025
stuhood
added a commit
that referenced
this pull request
Oct 25, 2025
* Removes allocation in a bunch of places * Removes sorting of terms if we're going to use the fast field execution method * Adds back the (accidentally dropped) cardinality threshold * Removes `bool` support -- using the posting lists is always more efficient for a `bool`, since there are at most two of them * More eagerly constructs the term `HashSet` so that it happens once, rather than once per segment
stuhood
added a commit
to paradedb/paradedb
that referenced
this pull request
Oct 26, 2025
## What Further optimizes `pdb.term_set` and `paradedb.term_set`, and deprecates using `pdb.term_set` in aggregate position. ## Why #3351 optimized `term_set` queries for very large input term sets by switching to using fast fields when more than `1024` terms were used in the set. But there was more that could be done. For a `paradedb.aggregate` query using a `pdb.term_set` constructed from an `array_agg` containing 10mm distinct inputs and 8 segments, this PR further optimizes the fast field execution path: | version | runtime | | ------- | -------- | | pre-#3351 - 0 workers | 35.851 s | | pre-#3351 - 8 workers | swapping - did not complete | | #3351 - 0 workers | 12.573 s | | #3351 - 8 workers | 13.708 s | | #3412 - 0 workers | 5.532 s | | #3412 - 8 workers | 8.538 s | Before #3351, the posting-list based execution mode for term sets was not able to complete on my machine with multiple workers, because it required enough memory to trigger swapping. Critical to note: in the case of a massive `pdb.term_set` like this, additional workers might be a pessimization. That's because the cost of propagating and creating the query is expensive enough that it can dwarf the actual aggregate time. For larger segment counts or larger aggregates, the results might be different. Additionally, this change deprecates using `pdb.term_set` in aggregate position (as added in #3336): using `pdb.term_set` in function position with an `array_agg` is equivalent, and sometimes slightly faster. ## How * Incorporates paradedb/tantivy#75 * Removes some allocation from `pdb.term_set` and `paradedb.term_set` creation. * Skips allocation `FieldName::path` arrays if they contain a single component.
stuhood
added a commit
to paradedb/paradedb
that referenced
this pull request
Oct 26, 2025
Further optimizes `pdb.term_set` and `paradedb.term_set`, and deprecates using `pdb.term_set` in aggregate position. switching to using fast fields when more than `1024` terms were used in the set. But there was more that could be done. For a `paradedb.aggregate` query using a `pdb.term_set` constructed from an `array_agg` containing 10mm distinct inputs and 8 segments, this PR further optimizes the fast field execution path: | version | runtime | | ------- | -------- | | pre-#3351 - 0 workers | 35.851 s | | pre-#3351 - 8 workers | swapping - did not complete | | #3351 - 0 workers | 12.573 s | | #3351 - 8 workers | 13.708 s | | #3412 - 0 workers | 5.532 s | | #3412 - 8 workers | 8.538 s | Before #3351, the posting-list based execution mode for term sets was not able to complete on my machine with multiple workers, because it required enough memory to trigger swapping. Critical to note: in the case of a massive `pdb.term_set` like this, additional workers might be a pessimization. That's because the cost of propagating and creating the query is expensive enough that it can dwarf the actual aggregate time. For larger segment counts or larger aggregates, the results might be different. Additionally, this change deprecates using `pdb.term_set` in aggregate position (as added in #3336): using `pdb.term_set` in function position with an `array_agg` is equivalent, and sometimes slightly faster. * Incorporates paradedb/tantivy#75 * Removes some allocation from `pdb.term_set` and `paradedb.term_set` creation. * Skips allocation `FieldName::path` arrays if they contain a single component.
stuhood
added a commit
to paradedb/paradedb
that referenced
this pull request
Oct 26, 2025
## What Further optimizes `pdb.term_set` and `paradedb.term_set`, and deprecates using `pdb.term_set` in aggregate position. ## Why #3351 optimized `term_set` queries for very large input term sets by switching to using fast fields when more than `1024` terms were used in the set. But there was more that could be done. For a `paradedb.aggregate` query using a `pdb.term_set` constructed from an `array_agg` containing 10mm distinct inputs and 8 segments, this PR further optimizes the fast field execution path: | version | runtime | | ------- | -------- | | pre-#3351 - 0 workers | 35.851 s | | pre-#3351 - 8 workers | swapping - did not complete | | #3351 - 0 workers | 12.573 s | | #3351 - 8 workers | 13.708 s | | #3412 - 0 workers | 5.532 s | | #3412 - 8 workers | 8.538 s | Before #3351, the posting-list based execution mode for term sets was not able to complete on my machine with multiple workers, because it required enough memory to trigger swapping. Critical to note: in the case of a massive `pdb.term_set` like this, additional workers might be a pessimization. That's because the cost of propagating and creating the query is expensive enough that it can dwarf the actual aggregate time. For larger segment counts or larger aggregates, the results might be different. Additionally, this change deprecates using `pdb.term_set` in aggregate position (as added in #3336): using `pdb.term_set` in function position with an `array_agg` is equivalent, and sometimes slightly faster. ## How * Incorporates paradedb/tantivy#75 * Removes some allocation from `pdb.term_set` and `paradedb.term_set` creation. * Skips allocation `FieldName::path` arrays if they contain a single component.
stuhood
added a commit
to paradedb/paradedb
that referenced
this pull request
Oct 27, 2025
## What Further optimizes `pdb.term_set` and `paradedb.term_set`, and deprecates using `pdb.term_set` in aggregate position. ## Why #3351 optimized `term_set` queries for very large input term sets by switching to using fast fields when more than `1024` terms were used in the set. But there was more that could be done. For a `paradedb.aggregate` query using a `pdb.term_set` constructed from an `array_agg` containing 10mm distinct inputs and 8 segments, this PR further optimizes the fast field execution path: | version | runtime | | ------- | -------- | | pre-#3351 - 0 workers | 35.851 s | | pre-#3351 - 8 workers | swapping - did not complete | | #3351 - 0 workers | 12.573 s | | #3351 - 8 workers | 13.708 s | | #3412 - 0 workers | 5.532 s | | #3412 - 8 workers | 8.538 s | Before #3351, the posting-list based execution mode for term sets was not able to complete on my machine with multiple workers, because it required enough memory to trigger swapping. Critical to note: in the case of a massive `pdb.term_set` like this, additional workers might be a pessimization. That's because the cost of propagating and creating the query is expensive enough that it can dwarf the actual aggregate time. For larger segment counts or larger aggregates, the results might be different. Additionally, this change deprecates using `pdb.term_set` in aggregate position (as added in #3336): using `pdb.term_set` in function position with an `array_agg` is equivalent, and sometimes slightly faster. ## How * Incorporates paradedb/tantivy#75 * Removes some allocation from `pdb.term_set` and `paradedb.term_set` creation. * Skips allocation `FieldName::path` arrays if they contain a single component.
stuhood
added a commit
to paradedb/paradedb
that referenced
this pull request
Oct 27, 2025
## What Further optimizes `pdb.term_set` and `paradedb.term_set`, and deprecates using `pdb.term_set` in aggregate position. ## Why #3351 optimized `term_set` queries for very large input term sets by switching to using fast fields when more than `1024` terms were used in the set. But there was more that could be done. For a `paradedb.aggregate` query using a `pdb.term_set` constructed from an `array_agg` containing 10mm distinct inputs and 8 segments, this PR further optimizes the fast field execution path: | version | runtime | | ------- | -------- | | pre-#3351 - 0 workers | 35.851 s | | pre-#3351 - 8 workers | swapping - did not complete | | #3351 - 0 workers | 12.573 s | | #3351 - 8 workers | 13.708 s | | #3412 - 0 workers | 5.532 s | | #3412 - 8 workers | 8.538 s | Before #3351, the posting-list based execution mode for term sets was not able to complete on my machine with multiple workers, because it required enough memory to trigger swapping. Critical to note: in the case of a massive `pdb.term_set` like this, additional workers might be a pessimization. That's because the cost of propagating and creating the query is expensive enough that it can dwarf the actual aggregate time. For larger segment counts or larger aggregates, the results might be different. Additionally, this change deprecates using `pdb.term_set` in aggregate position (as added in #3336): using `pdb.term_set` in function position with an `array_agg` is equivalent, and sometimes slightly faster. ## How * Incorporates paradedb/tantivy#75 * Removes some allocation from `pdb.term_set` and `paradedb.term_set` creation. * Skips allocation `FieldName::path` arrays if they contain a single component.
stuhood
added a commit
to paradedb/paradedb
that referenced
this pull request
Oct 27, 2025
## What Further optimizes `pdb.term_set` and `paradedb.term_set`, and deprecates using `pdb.term_set` in aggregate position. ## Why #3351 optimized `term_set` queries for very large input term sets by switching to using fast fields when more than `1024` terms were used in the set. But there was more that could be done. For a `paradedb.aggregate` query using a `pdb.term_set` constructed from an `array_agg` containing 10mm distinct inputs and 8 segments, this PR further optimizes the fast field execution path: | version | runtime | | ------- | -------- | | pre-#3351 - 0 workers | 35.851 s | | pre-#3351 - 8 workers | swapping - did not complete | | #3351 - 0 workers | 12.573 s | | #3351 - 8 workers | 13.708 s | | #3412 - 0 workers | 5.532 s | | #3412 - 8 workers | 8.538 s | Before #3351, the posting-list based execution mode for term sets was not able to complete on my machine with multiple workers, because it required enough memory to trigger swapping. Critical to note: in the case of a massive `pdb.term_set` like this, additional workers might be a pessimization. That's because the cost of propagating and creating the query is expensive enough that it can dwarf the actual aggregate time. For larger segment counts or larger aggregates, the results might be different. Additionally, this change deprecates using `pdb.term_set` in aggregate position (as added in #3336): using `pdb.term_set` in function position with an `array_agg` is equivalent, and sometimes slightly faster. ## How * Incorporates paradedb/tantivy#75 * Removes some allocation from `pdb.term_set` and `paradedb.term_set` creation. * Skips allocation `FieldName::path` arrays if they contain a single component.
mdashti
pushed a commit
that referenced
this pull request
Dec 3, 2025
* Removes allocation in a bunch of places * Removes sorting of terms if we're going to use the fast field execution method * Adds back the (accidentally dropped) cardinality threshold * Removes `bool` support -- using the posting lists is always more efficient for a `bool`, since there are at most two of them * More eagerly constructs the term `HashSet` so that it happens once, rather than once per segment
stuhood
added a commit
that referenced
this pull request
Dec 3, 2025
* Removes allocation in a bunch of places * Removes sorting of terms if we're going to use the fast field execution method * Adds back the (accidentally dropped) cardinality threshold * Removes `bool` support -- using the posting lists is always more efficient for a `bool`, since there are at most two of them * More eagerly constructs the term `HashSet` so that it happens once, rather than once per segment
stuhood
added a commit
that referenced
this pull request
Dec 10, 2025
* Removes allocation in a bunch of places * Removes sorting of terms if we're going to use the fast field execution method * Adds back the (accidentally dropped) cardinality threshold * Removes `bool` support -- using the posting lists is always more efficient for a `bool`, since there are at most two of them * More eagerly constructs the term `HashSet` so that it happens once, rather than once per segment
stuhood
added a commit
that referenced
this pull request
Dec 10, 2025
* Removes allocation in a bunch of places * Removes sorting of terms if we're going to use the fast field execution method * Adds back the (accidentally dropped) cardinality threshold * Removes `bool` support -- using the posting lists is always more efficient for a `bool`, since there are at most two of them * More eagerly constructs the term `HashSet` so that it happens once, rather than once per segment
stuhood
added a commit
that referenced
this pull request
Jan 13, 2026
stuhood
added a commit
that referenced
this pull request
Jan 13, 2026
stuhood
added a commit
that referenced
this pull request
Jan 13, 2026
stuhood
added a commit
that referenced
this pull request
Jan 14, 2026
rebasedming
pushed a commit
that referenced
this pull request
Feb 10, 2026
rebasedming
pushed a commit
that referenced
this pull request
Feb 10, 2026
rebasedming
pushed a commit
that referenced
this pull request
Jun 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
boolsupport -- using the posting lists is always more efficient for abool, since there are at most two of themHashSetso that it happens once, rather than once per segment