Use bloom filters to collect large DFs #25009
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raunaqmorarka
force-pushed
the
df-bloom-final
branch
from
12f2fdd to
35e139b
Compare
raunaqmorarka
force-pushed
the
df-bloom-final
branch
2 times, most recently
from
ef14d7f to
454c630
Compare
raunaqmorarka
requested review from
lukasz-stec,
Dith3r,
sopel39 and
losipiuk
|
Ignore the red tests for now, cleaning them up |
lukasz-stec
reviewed
Feb 20, 2025
core/trino-main/src/main/java/io/trino/sql/gen/columnar/BloomColumnarFilter.java
Outdated
Show resolved
Hide resolved
losipiuk
reviewed
Feb 25, 2025
core/trino-main/src/main/java/io/trino/sql/planner/DynamicFilterDomain.java
Outdated
Show resolved
Hide resolved
losipiuk
reviewed
Feb 25, 2025
core/trino-main/src/main/java/io/trino/sql/planner/DynamicFilterDomain.java
Outdated
Show resolved
Hide resolved
losipiuk
reviewed
Feb 25, 2025
core/trino-main/src/main/java/io/trino/sql/planner/DynamicFilterDomain.java
Outdated
Show resolved
Hide resolved
losipiuk
reviewed
Feb 25, 2025
Comment on lines
252
to
265
| { | ||
| return new DynamicFilterDomain(Domain.all(type)); | ||
| } |
There was a problem hiding this comment.
feels like it should just be none
losipiuk
reviewed
Feb 25, 2025
core/trino-main/src/main/java/io/trino/sql/planner/BloomFilterWithRange.java
Outdated
Show resolved
Hide resolved
losipiuk
reviewed
Feb 25, 2025
core/trino-main/src/main/java/io/trino/sql/planner/BloomFilterWithRange.java
Outdated
Show resolved
Hide resolved
losipiuk
reviewed
Feb 25, 2025
Comment on lines
284
to
281
| // returned mask sets 3 bits based on portions of given hash | ||
| // Extract 38th to 43rd bits |
There was a problem hiding this comment.
All this bit numbers is sheer magic. Please provide a structure of bloom filter in javadoc at the top of the class.
Some ascii art with bit marking would be appreciated.
losipiuk
reviewed
Feb 25, 2025
| | (1L << ((hashCode >> 33) & 63)); | ||
| } | ||
|
|
||
| private static int getBloomFilterSize(int valuesCount) |
There was a problem hiding this comment.
english explanation what it does
losipiuk
reviewed
Feb 25, 2025
| import static java.util.Objects.requireNonNull; | ||
| import static java.util.stream.Collectors.toMap; | ||
|
|
||
| public class DynamicFilterTupleDomain<T> |
There was a problem hiding this comment.
Would it be feasible to fight for reusing more code from TupleDomain instead of copy pasting?
losipiuk
reviewed
Feb 25, 2025
raunaqmorarka
force-pushed
the
df-bloom-final
branch
from
454c630 to
97c0284
Compare
raunaqmorarka
force-pushed
the
df-bloom-final
branch
2 times, most recently
from
d3af94e to
a9f1a72
Compare
raunaqmorarka
force-pushed
the
df-bloom-final
branch
3 times, most recently
from
200e239 to
04d8e93
Compare
raunaqmorarka
force-pushed
the
df-bloom-final
branch
5 times, most recently
from
09bee49 to
53cdc3b
Compare
raunaqmorarka
force-pushed
the
df-bloom-final
branch
10 times, most recently
from
6c32f54 to
216c41b
Compare
raunaqmorarka
force-pushed
the
df-bloom-final
branch
5 times, most recently
from
e000b4a to
6cf30bd
Compare
BenchmarkDynamicPageFilter.filterPages
(filterSize) (inputDataSet) (inputNullChance) (nonNullsSelectivity) (nullsAllowed) Mode Cnt Score using fastutil set Score using bloom filter
1000 INT64_RANDOM 0.05 0.2 false thrpt 20 30.282 ± 0.792 ops/s 65.017 ± 0.566 ops/s
10000 INT64_RANDOM 0.05 0.2 false thrpt 20 33.799 ± 0.511 ops/s 63.218 ± 1.783 ops/s
100000 INT64_RANDOM 0.05 0.2 false thrpt 20 29.464 ± 0.469 ops/s 63.482 ± 1.626 ops/s
1000000 INT64_RANDOM 0.05 0.2 false thrpt 20 18.854 ± 0.558 ops/s 63.690 ± 1.662 ops/s
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect, maxDistinctValuesCount = 600572
Collection type (positionsPerPage) Mode Cnt Score Error Units
Hash set 4096 avgt 45 39.950 ± 0.281 ms/op
Bloom filter 4096 avgt 45 10.297 ± 0.065 ms/op
Min-max 4096 avgt 45 5.845 ± 0.038 ms/op
no-op 4096 avgt 45 0.075 ± 0.001 ms/op
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect, maxDistinctValuesCount = 6001215
Collection type (positionsPerPage) Mode Cnt Score Error Units
Hash set 4096 avgt 45 590.042 ± 22.009 ms/op
Bloom filter 4096 avgt 45 98.025 ± 0.750 ms/op
Min-max 4096 avgt 45 61.982 ± 6.330 ms/op
no-op 4096 avgt 45 0.092 ± 0.001 ms/op
raunaqmorarka
force-pushed
the
df-bloom-final
branch
from
6cf30bd to
db6464b
Compare
|
This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack. |
raunaqmorarka
added
stale-ignore
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: