[branch-4.1][feature](variant) add variant doc mode (#59183)#61281
Open
csun5285 wants to merge 4 commits intoapache:branch-4.1from
Open
[branch-4.1][feature](variant) add variant doc mode (#59183)#61281csun5285 wants to merge 4 commits intoapache:branch-4.1from
csun5285 wants to merge 4 commits intoapache:branch-4.1from
Conversation
**DOC encoding**: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document. doc: apache/doris-website#3253
…hmarks (apache#60511) Improve VARIANT doc-mode ingestion and compaction performance, and add a write/compaction benchmark UT. Key changes: - Add config `enable_variant_doc_compaction_sparse_write` (default: true) to enable sparse writing for materialized subcolumns during doc-mode compaction. - Optimize doc value column generation: deduplicate paths, sort paths once, and serialize values directly into the binary column to reduce allocations/copies. - Optimize bucket sharding/stats collection with phmap and cached path->bucket mapping; remove per-row doc value sorting in writer. - Refactor VariantDocCompactWriter finalize flow with an explicit write plan; skip invalid array types and fix doc materialization offset handling. - Update variant-related UTs and add VariantDocModeCompactionTest to benchmark segment import and doc-mode compaction. UT Testing: - BE UT: VariantColumnWriterReaderTest.*, VariantUtilTest.*, ColumnVariantTest.* Benchmark Testing(Release compile): - BE UT (Release-only, slow): VariantDocModeCompactionTest.* <img width="902" height="778" alt="image" src="https://github.com/user-attachments/assets/08fb9341-2620-484d-9cf9-f3a64c95d4bd" />
…index_fault_injection (apache#60518)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick from master #59183, #60511, #60518, #60668