Skip to content

[feat][cp][DynamicMerge] Add lazy multi merge#409

Open
guhaiyan0221 wants to merge 2 commits intobytedance:mainfrom
guhaiyan0221:fix_cp_lazy_spill
Open

[feat][cp][DynamicMerge] Add lazy multi merge#409
guhaiyan0221 wants to merge 2 commits intobytedance:mainfrom
guhaiyan0221:fix_cp_lazy_spill

Conversation

@guhaiyan0221
Copy link
Collaborator

What problem does this PR solve?

Issue Number: close #191

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 🚀 Performance improvement (optimization)
  • ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
  • 🔨 Refactoring (no logic changes)
  • 🔧 Build/CI or Infrastructure changes
  • 📝 Documentation only

Description

Summary:
The LocalMerge operator may start only a subset of the sources at a time to limit memory usage. Hence, it performs multiple partial sort-merge operations, each handling a subset of the merge sources. It spills the results produced by each subset and finishes the spill to ensure the order both within and across all spill files for each partial merge. Finally, creates a ConcatFilesSpillMergeStream for each group of spill files of each partial merge , setup a SpillMerger using those ConcatFilesSpillMergeStream to perform the final sort-merge of the outputs.

Parts of facebookincubator/velox#13260

Corresponding PR: facebookincubator/velox#13586

Performance Impact

  • No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).

  • Positive Impact: I have run benchmarks.

    Click to view Benchmark Results
    Paste your google-benchmark or TPC-H results here.
    Before: 10.5s
    After:   8.2s  (+20%)
    
  • Negative Impact: Explained below (e.g., trade-off for correctness).

Release Note

Please describe the changes in this PR

Release Note:

Release Note:
- Fixed a crash in `substr` when input is null.
- optimized `group by` performance by 20%.

Checklist (For Author)

  • I have added/updated unit tests (ctest).
  • I have verified the code with local build (Release/Debug).
  • I have run clang-format / linters.
  • (Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
  • No need to test or manual test.

Breaking Changes

  • No

  • Yes (Description: ...)

    Click to view Breaking Changes
    Breaking Changes:
    - Description of the breaking change.
    - Possible solutions or workarounds.
    - Any other relevant information.
    

Summary:
The `LocalMerge` operator may start only a subset of the sources at a time
to limit memory usage. Hence, it performs multiple partial sort-merge operations,
each handling a subset of the merge sources. It spills the results produced by each subset
and finishes the spill to ensure the order both within and across all spill files for each partial merge.
Finally, creates a `ConcatFilesSpillMergeStream` for each group of spill files of each partial merge ,
setup a `SpillMerger` using those  `ConcatFilesSpillMergeStream` to perform the final sort-merge
of the outputs.

Parts of facebookincubator/velox#13260

Corresponding PR: facebookincubator/velox#13586
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants