[Enhancement] Optimize large tablet lake compaction by support parallel compaction #66586
+5,409
−78
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
🧪 CI InsightsHere's what we observed from your CI run for 0b34040. 🟢 All jobs passed!But CI Insights is watching 👀 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces support for parallel lake compaction, allowing multiple compaction subtasks to run concurrently within a single tablet. The feature aims to improve compaction throughput by enabling non-overlapping rowsets within the same tablet to be compacted in parallel, rather than sequentially.
Key Changes:
- Adds per-tablet parallel compaction manager in BE to coordinate concurrent subtasks with non-overlapping rowset selection
- Extends protobuf definitions for parallel compaction configuration and autonomous compaction results
- Adds FE and BE configuration options to enable/disable parallel compaction and configure parallelism limits
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| gensrc/proto/lake_types.proto | Adds CompactionResultPB for storing autonomous compaction results |
| gensrc/proto/lake_service.proto | Adds TabletParallelConfig for parallel compaction settings and TabletSubtaskStatus for tracking subtask progress |
| fe/fe-core/src/main/java/com/starrocks/common/Config.java | Adds 8 new configuration options for autonomous and parallel compaction modes |
| fe/fe-core/src/main/java/com/starrocks/lake/compaction/CompactionScheduler.java | Extends compaction requests to include parallel config and visible version; fixes typo |
| be/src/common/config.h | Adds 14 BE configuration options for autonomous/parallel compaction and recovery |
| be/src/storage/lake/tablet_parallel_compaction_manager.h | Defines TabletParallelCompactionManager class to orchestrate parallel compaction within tablets |
| be/src/storage/lake/tablet_parallel_compaction_manager.cpp | Implements parallel compaction logic: rowset selection, subtask execution, TxnLog merging |
| be/src/storage/lake/tablet_manager.h | Adds overload of compact() accepting pre-selected rowsets |
| be/src/storage/lake/tablet_manager.cpp | Implements compact() overload to support parallel compaction with pre-selected rowsets |
| be/src/storage/lake/compaction_scheduler.h | Adds TabletParallelCompactionManager member and process_parallel_compaction method |
| be/src/storage/lake/compaction_scheduler.cpp | Integrates parallel compaction path with fallback to non-parallel mode on failure |
| be/src/storage/lake/compaction_policy.h | Adds PartialCompactionState struct and PartialCompactionSelector for autonomous compaction support |
| be/src/storage/lake/compaction_policy.cpp | Implements pick_rowsets_with_limit() for rowset selection with exclusion and byte limits; adds partial compaction helpers |
| be/src/storage/CMakeLists.txt | Adds tablet_parallel_compaction_manager.cpp to build |
fe/fe-core/src/main/java/com/starrocks/lake/compaction/CompactionScheduler.java
Show resolved
Hide resolved
meegoo
force-pushed
the
parallel_compaction
branch
from
bb76f7c to
0b34040
Compare
meegoo
changed the title
meegoo
changed the title
Contributor
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
meegoo
force-pushed
the
parallel_compaction
branch
from
0b34040 to
2b492e4
Compare
Contributor
|
@cursor review |
meegoo
force-pushed
the
parallel_compaction
branch
3 times, most recently
from
fd9d1a2 to
3fc0c7c
Compare
Contributor
|
@cursor review |
meegoo
force-pushed
the
parallel_compaction
branch
2 times, most recently
from
b852e25 to
243dcad
Compare
Contributor
|
@cursor review |
meegoo
force-pushed
the
parallel_compaction
branch
2 times, most recently
from
d019ca7 to
1188394
Compare
Contributor
|
@cursor review |
| if (context->partition_id == 0) { | ||
| context->partition_id = id_pair.second; | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
should handle else not ok
Contributor
Author
There was a problem hiding this comment.
else the id already set.
| public static int lake_compaction_max_parallel_per_tablet = 3; | ||
|
|
||
| @ConfField(mutable = true, comment = "Maximum data volume (bytes) per parallel subtask (1GB default)") | ||
| public static long lake_compaction_max_bytes_per_subtask = 1073741824L; |
Contributor
There was a problem hiding this comment.
is 1GB too small?
Contributor
Author
There was a problem hiding this comment.
This will be adjusted after large-scale testing.
Contributor
|
what does this feature do to |
| bool is_rowset_compacting(uint32_t rowset_id) const { return compacting_rowsets.count(rowset_id) > 0; } | ||
|
|
||
| // Check if all subtasks are completed | ||
| bool is_complete() const { return running_subtasks.empty() && total_subtasks_created > 0; } |
Contributor
There was a problem hiding this comment.
need mutex protection? since these values are written under mutex
Contributor
Author
There was a problem hiding this comment.
This function is protected by a mutex wherever it is used.
Contributor
Author
yes |
meegoo
force-pushed
the
parallel_compaction
branch
2 times, most recently
from
5d5d48b to
b32170c
Compare
Contributor
|
@cursor review |
be/test/storage/lake/tablet_parallel_compaction_manager_test.cpp
Outdated
Show resolved
Hide resolved
meegoo
force-pushed
the
parallel_compaction
branch
from
b32170c to
a4a287b
Compare
Contributor
|
@cursor review |
meegoo
force-pushed
the
parallel_compaction
branch
from
a4a287b to
e1a1b2a
Compare
Contributor
|
@cursor review |
meegoo
force-pushed
the
parallel_compaction
branch
from
e1a1b2a to
3f6660a
Compare
Contributor
|
@cursor review |
meegoo
force-pushed
the
parallel_compaction
branch
2 times, most recently
from
b5a7e76 to
7251ffd
Compare
meegoo
force-pushed
the
parallel_compaction
branch
from
7251ffd to
c759494
Compare
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 16 / 19 (84.21%) file detail
|
[BE Incremental Coverage Report]❌ fail : 709 / 1059 (66.95%) file detail
|
Contributor
|
@cursor review |
Why I'm doing: - Lake table compaction currently executes in a single thread per tablet, which becomes a bottleneck for large tablets with significant data volume or many rowsets, causing data accumulation and query performance degradation. - To improve compaction throughput and reduce compaction lag, we need the ability to run multiple compaction subtasks concurrently within a single tablet. What I'm doing: - Introduce TabletParallelCompactionManager to orchestrate parallel compaction subtasks within a single tablet by selecting non-overlapping rowset groups. - Modify CompactionScheduler to process parallel compaction requests from FE, creating multiple subtasks that can run concurrently on the thread pool. - Add rows mapper functionality to track row mappings across parallel subtasks, enabling proper conflict resolution for primary key tables. - Optimize compaction score calculation by skipping large segments that are close to max_segment_file_size (configurable via lake_compaction_skip_large_ segment_ratio), avoiding unnecessary compaction of already optimized segments. - Defer SST compaction to execute once after all subtasks complete, preventing multiple subtasks from competing to compact the same SST files. - Add FE configurations: lake_compaction_enable_parallel_per_tablet, lake_compaction_max_parallel_per_tablet, lake_compaction_max_bytes_per_subtask. - Add BE configurations: enable_lake_compaction_skip_large_segment. - Extend protobuf messages (TabletParallelConfig, CompactionResultPB) to support parallel compaction coordination between FE and BE. - Add comprehensive unit tests for parallel compaction manager and scheduler. Signed-off-by: meegoo <[email protected]>
meegoo
force-pushed
the
parallel_compaction
branch
from
c759494 to
e7d3503
Compare
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
becomes a bottleneck for large tablets with significant data volume or many
rowsets, causing data accumulation and query performance degradation.
to run multiple compaction subtasks concurrently within a single tablet.
What I'm doing:
subtasks within a single tablet by selecting non-overlapping rowset groups.
creating multiple subtasks that can run concurrently on the thread pool.
enabling proper conflict resolution for primary key tables.
close to max_segment_file_size (configurable via lake_compaction_skip_large_
segment_ratio), avoiding unnecessary compaction of already optimized segments.
multiple subtasks from competing to compact the same SST files.
lake_compaction_max_parallel_per_tablet, lake_compaction_max_bytes_per_subtask.
lake_compaction_max_bytes_per_subtask, enable_lake_compaction_skip_large_segment.
parallel compaction coordination between FE and BE.
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
Note
Introduces parallel compaction for Lake tablets and optimizations to reduce unnecessary work.
TabletParallelCompactionManagerorchestrates per-tablet subtasks, mergesTxnLogPB, and defers SST compaction until all subtasks finishCompactionSchedulerhandlesparallel_config, creates subtasks, tracks/renders parallel task states, and falls back to non-parallel when neededMultiRowsMapperIterator), per-subtask mapper files,subtask_idplumbed through writers and contextsTabletManager::compact(context, input_rowsets)overload to run compaction on pre-picked rowsetscalc_effective_segment_count()and config flags (enable_lake_compaction_skip_large_segment,lake_compaction_skip_large_segment_ratio) skip large segments in score/selectionsubtask_id; skip SST compaction inside subtasks for PK tables (run once post-merge)Written by Cursor Bugbot for commit c759494. This will update automatically on new commits. Configure here.