[Feature] Support for builtin inverted index #66541
Open
+2,334
−81
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
🧪 CI InsightsHere's what we observed from your CI run for e3bf220. 🟢 All jobs passed!But CI Insights is watching 👀 |
Contributor
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
be/src/storage/index/inverted/builtin/builtin_inverted_index_iterator.cpp
Outdated
Show resolved
Hide resolved
Contributor
|
@cursor review |
Contributor
|
@cursor review |
be/src/storage/index/inverted/builtin/builtin_inverted_index_iterator.cpp
Show resolved
Hide resolved
Contributor
|
@cursor review |
srlch
force-pushed
the
builtin_gin
branch
2 times, most recently
from
8644f5c to
97d36e4
Compare
Contributor
|
@cursor review |
Contributor
|
@cursor review |
be/src/storage/index/inverted/builtin/builtin_inverted_index_iterator.cpp
Outdated
Show resolved
Hide resolved
Contributor
|
@cursor review |
1 similar comment
Contributor
|
@cursor review |
be/src/storage/index/inverted/builtin/builtin_inverted_index_iterator.cpp
Show resolved
Hide resolved
sevev
previously approved these changes
Dec 26, 2025
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 19 / 19 (100.00%) file detail
|
[BE Incremental Coverage Report]✅ pass : 383 / 420 (91.19%) file detail
|
Contributor
|
@cursor review |
fe/fe-core/src/main/java/com/starrocks/server/OlapTableFactory.java
Outdated
Show resolved
Hide resolved
wyb
reviewed
Dec 27, 2025
be/src/storage/index/inverted/builtin/builtin_inverted_index_iterator.cpp
Outdated
Show resolved
Hide resolved
| return st; | ||
| } | ||
|
|
||
| Status BuiltinInvertedIndexIterator::_equal_query(const Slice* search_query, roaring::Roaring* bit_map) { |
Contributor
There was a problem hiding this comment.
If it’s tokenized at write time, can we still perform an equal query using bitmap index?
Contributor
Author
There was a problem hiding this comment.
equal query means match keyword. It always need MATCH predicate to trigger inverted index
| }; | ||
| Slice from_value = Slice(lower.second); | ||
| size_t search_size = upper.first - lower.first; | ||
| auto res = _bitmap_itr->seek_dictionary_by_predicate(predicate, from_value, search_size); |
Contributor
There was a problem hiding this comment.
If it’s tokenized at write time, is the like predicate on dictionary right?
Contributor
Author
There was a problem hiding this comment.
Yes, it performs like on dictionary
be/src/storage/index/inverted/builtin/builtin_inverted_index_iterator.cpp
Outdated
Show resolved
Hide resolved
wyb
reviewed
Dec 27, 2025
fe/fe-core/src/main/java/com/starrocks/sql/analyzer/IndexAnalyzer.java
Outdated
Show resolved
Hide resolved
|
This PR introduces builtin inverted index including the entire read and write logic. It builtin based on the bitmap index infrastructure. Signed-off-by: srlch <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I'm doing:
This PR introduces builtin inverted index including the entire read and write logic. It builtin based on the bitmap index infrastructure.
Fixes #67167
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
Note
Adds a builtin inverted index implementation using bitmap index infrastructure with complete reader/writer/iterator and planner/runtime integration.
builtininverted index:BuiltinInvertedWriter/Reader/IndexIteratorandBuiltinPlugin; simple tokenizer for English; wired into CMakeColumnReader/Writer,Segment,SegmentIteratorupdated to load/use builtin inverted index viaIndexReadOptions; iterator close path added; new GIN metrics (GinFilter*) and profile countersadd_value_with_current_rowid,incre_rowid,finishoverload,seek_dictionary_by_predicate,next_batch_dictionary,read_union_bitmap(Buffer)) and minor size/memory optimizationsenable_prune_column_after_index_filterandenable_gin_filter; propagate to tablet reader paramsIMP_LIB=builtin, default to builtin in shared-data mode; restrict clucene in shared-data; refine replicated storage rule when GIN present; assign index id for CN tables (non-vector)BUILTIN_INVERTED_INDEXandBuiltinInvertedIndexPB; thriftTLakeScanNodegains enable flagsWritten by Cursor Bugbot for commit e3f10ad. This will update automatically on new commits. Configure here.