[improvement](compaction) Support lazy chunked row ID conversion - #67333
Open
mymeiyi wants to merge 3 commits into
Open
[improvement](compaction) Support lazy chunked row ID conversion#67333mymeiyi wants to merge 3 commits into
mymeiyi wants to merge 3 commits into
Conversation
Issue Number: None
Related PR: None
Problem Summary: Cloud cumulative compaction normally selects multiple rowsets, so a singleton rowset containing many overlapping segments can remain uncompacted. Add configurable selection and grouped merge support for eligible single rowsets, merging bounded segment ranges through the common compaction output lifecycle and preserving overlap semantics across non-empty output groups. Restrict the grouped path to size-based cumulative policy after normal policy filtering, exclude cluster-key tablets, retain the input cumulative point for this grouped path, and keep row ID conversion, vertical writer segment metadata, inverted index metadata, and vertical progress reporting consistent across all segment groups. Snapshot the grouped-mode decision and segment group size when input rowsets are selected so queued tasks preserve their selected execution semantics if dynamic settings change before merging begins. Add focused BE unit coverage and a cloud regression case for grouped compaction behavior.
Support configurable cloud cumulative compaction of a single overlapping rowset in bounded segment groups. Queued tasks preserve their selected grouped execution mode when dynamic settings change. Cluster-key tablets continue to use the existing compaction path.
- Test: Unit Test / Regression test
- Added focused BE unit coverage and a cloud regression case.
- Targeted CloudCompactionTest attempted with run-be-ut.sh; blocked during CMake configuration because OpenMP_C is unavailable.
- Behavior changed: Yes. Eligible cloud single rowsets with overlapping segments can be compacted in bounded groups while retaining their previous cumulative point and task-local grouped settings.
- Does this need documentation: No
Issue Number: None Related PR: apache#65907 Problem Summary: Single-rowset grouped compaction produces rowsets whose segments are non-overlapping only within recorded groups. Follow-up compaction, vertical reading, row ID conversion, cloud snapshot conversion, and schema change did not consistently preserve or interpret that layout. Repeated compaction also selected input by logical group count while planning ranges by physical segment count, so it could rewrite a rowset without reducing groups. Preserve and validate grouped metadata across cloud paths, read ordered segments lazily within each group, plan follow-up merges from logical group boundaries, use logical merge-way counts, and skip configurations that cannot make progress. Run the regression case non-concurrently only in cloud mode on the existing cloud cluster, temporarily disable SQL cache and mutable BE settings with guaranteed restoration, and use the configured MetaService endpoint. Reuse ReadSchema in the repeated-compaction unit test so it uses the current storage reader API. None - Test: No tests executed (targeted BE UT build was previously blocked during CMake configuration because OpenMP_C was unavailable; later runs were skipped per request); git diff --check passed - Behavior changed: Yes (grouped rowsets are preserved and read correctly, repeated compaction reduces logical groups, and the cloud regression case is skipped outside cloud mode) - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: Row ID conversion previously allocated a dense destination entry for every source row, which wastes memory when a compaction reads only sparse source ranges. Add an opt-in lazy chunked mode that allocates mappings only for touched rows, preserves dense mode for existing callers, and propagates allocation failures through merge paths. ### Release note None ### Check List (For Author) - Test: Not run (per request) - Behavior changed: No - Does this need documentation: No
mymeiyi
requested review from
gavinchou,
liaoxin01,
luwei16 and
yiguolei
as code owners
August 31, 2026 07:26
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Doris BE compaction infrastructure to support segment-range (chunked) processing and introduces an opt-in lazy, chunked RowIdConversion mode to reduce memory usage when only sparse source row ranges are touched (a prerequisite for future parallel / sub-range compactions). It also adds cloud single-rowset grouped compaction metadata (segment groups) and corresponding readers/compaction logic, plus broad unit/regression coverage.
Changes:
- Add
RowIdConversion::Mode::LAZY_CHUNKEDto allocate row-id mappings on-demand in 4096-row chunks, and plumbStatus-returning APIs through merge paths. - Introduce
NONOVERLAPPING_WITHIN_GROUP+segment_group_sizesin rowset proto/meta and implement cloud single-rowset grouped compaction using segment-range merges. - Extend vertical/horizontal merge and
VerticalBlockReaderto respect segment ranges and grouped-segment iterator initialization, with new BE UTs and a cloud regression suite.
Checklist Conclusions (from review checkpoints)
- Goal & tests: The goal (reduce RowIdConversion memory + enable segment-range merge paths for grouped compaction) is implemented and backed by extensive UTs and a new cloud regression suite.
- Scope/focus: The PR is broad (proto + compaction execution path + readers + tests). Changes are cohesive but touch high-risk storage/compaction code paths.
- Concurrency: No new explicit concurrency primitives were introduced in the changed code; the work mainly restructures compaction execution and reader initialization. (Still high-risk due to compaction’s operational sensitivity.)
- Compatibility: Proto is extended (new enum + repeated field). Conversions/copy paths are updated for cloud metadata; rolling-upgrade behavior depends on invariants around when the new enum value can appear.
- Config: New cloud configs are added with defaults; they are used as runtime flags to gate behavior.
- Observability: A new INFO log is added for grouped compaction completion with useful tags.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
regression-test/suites/cloud_p0/compaction/test_cloud_single_rowset_grouped_compaction.groovy |
New cloud regression coverage for grouped rowset compaction behavior and follow-up compaction to fully non-overlapping. |
gensrc/proto/olap_file.proto |
Add NONOVERLAPPING_WITHIN_GROUP and segment_group_sizes fields to rowset metadata protos. |
be/test/storage/rowset/rowset_meta_test.cpp |
UT ensuring grouped overlap is treated as “overlapping” semantics and compaction-produced behavior. |
be/test/storage/rowid_conversion_test.cpp |
UTs for lazy chunked RowIdConversion plus an end-to-end grouped-compaction rowid correctness test. |
be/test/storage/pb_convert_test.cpp |
UT validating Doris↔Cloud rowset meta conversion for segment_group_sizes. |
be/test/storage/iterator/vertical_block_reader_test.cpp |
New UT for grouped iterator initialization flags and invalid-input behavior. |
be/test/storage/cloud_file_cache_write_index_only_test.cpp |
Extend UT assertions around allocated segment ID progression. |
be/test/cloud/cloud_snapshot_mgr_test.cpp |
Update snapshot conversion UT to include grouped overlap + group sizes. |
be/test/cloud/cloud_cumulative_compaction_policy_test.cpp |
Add policy test ensuring cumulative point advancement logic with overlapping output. |
be/test/cloud/cloud_compaction_test.cpp |
Add extensive UT coverage for grouped-compaction candidate checks, merge-range construction, and selection behaviors. |
be/src/storage/rowset/vertical_beta_rowset_writer.cpp |
Adjust segment row-count tracking and reset segment-writer state after final flush. |
be/src/storage/rowset/rowset_meta.h |
Add segment-group size validation APIs and treat within-group overlap as compaction-produced/overlapping semantics. |
be/src/storage/rowid_conversion.h |
Implement LAZY_CHUNKED mode, add memory tracking hooks, and return Status from add(). |
be/src/storage/merger.h / be/src/storage/merger.cpp |
Add optional segment_range plumbing and propagate Status from rowid conversion updates. |
be/src/storage/iterator/vertical_block_reader.h / be/src/storage/iterator/vertical_block_reader.cpp |
Add grouped overlap iterator-init logic and make iterator-init flags respect grouped layouts and segment ranges. |
be/src/storage/compaction/compaction.h / be/src/storage/compaction/compaction.cpp |
Refactor merge into overridable stages and add segment-range merge execution helper with progress mapping. |
be/src/cloud/pb_convert.cpp |
Copy/swap segment_group_sizes in Doris↔Cloud rowset meta conversion. |
be/src/cloud/config.h / be/src/cloud/config.cpp |
Add cloud single-rowset compaction feature/config knobs with defaults. |
be/src/cloud/cloud_snapshot_mgr.cpp |
Copy segment_group_sizes when creating snapshot rowset metas. |
be/src/cloud/cloud_schema_change_job.cpp |
Force conservative overlap semantics when schema change rewrites grouped rowsets. |
be/src/cloud/cloud_cumulative_compaction.h / be/src/cloud/cloud_cumulative_compaction.cpp |
Implement grouped single-rowset compaction flow with segment-range merges and output group-size metadata. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Summary:
Later, we will support parallel compaction, which means one compaction sub task only handles somes rows of the input rowsets.
Now, Row ID conversion allocated a dense destination entry for every source row, which wastes memory when a compaction reads only sparse source ranges.
Add an opt-in lazy chunked mode that allocates mappings only for touched rows.
This pr is based on #65907