branch-4.1:[fix](build) Keep Arrow 17 and 24 in shared thirdparty - #67320
Closed
hubgeter wants to merge 3 commits into
Closed
branch-4.1:[fix](build) Keep Arrow 17 and 24 in shared thirdparty#67320hubgeter wants to merge 3 commits into
hubgeter wants to merge 3 commits into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#66546 Problem Summary: Current branch-4.1 includes the Arrow 24 upgrade and can consume the same thirdparty installation as master. Installing Arrow 24 and Paimon against the legacy root prefix overwrites the Arrow 17 ABI expected by pre-upgrade branch-4.1 revisions and can leave mixed or stale artifacts after interrupted rebuilds. Keep Arrow 17 and Paimon 17 in the unversioned compatibility prefix, install the Arrow 24 and Paimon 24 ABI-coupled stack in a versioned prefix selected by the current branch, fingerprint and validate both stacks independently, and rebuild only the invalid stack. Preserve branch-4.1-specific build and test layout while backporting the shared-stack lifecycle coverage. ### Release note None ### Check List (For Author) - Test: Manual test - PASS: ./thirdparty/test/arrow-paimon-lifecycle-test.sh - PASS: bash -n on all changed shell scripts - PASS: ./build-support/clang-format.sh - PASS: ./build-support/check-format.sh - PASS: non-patch git diff --check and Arrow 17 patch byte-identity checks - Not run: full BE build; the filesystem is 99% used and this worktree does not have a valid dual-stack thirdparty installation, so rebuilding both stacks risks exhausting disk space - Behavior changed: Yes. Current branch-4.1 selects versioned Arrow 24/Paimon 24 while preserving the root Arrow 17/Paimon 17 compatibility stack. - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review completed against the verified PR head 0ed1bb2 (base 2c3ae38). I found three actionable correctness issues in the shared Arrow/Paimon lifecycle. The user focus file specifies no additional focus; the review covered the full changed-file set and the Arrow 17/24 prefix, source, patch, and CMake interactions.
Validation performed: all changed shell scripts pass bash -n; thirdparty/test/arrow-paimon-lifecycle-test.sh passes. No build was attempted, per the review instructions. The remaining candidate points were either dismissed with concrete evidence (alias/package closure, INT96/LZO/Thrift parity, Arrow Compute target gating, and the unsupported custom-prefix cache path) or are covered by the inline comments below.
### What problem does this PR solve? Issue Number: None Related PR: apache#66546 Problem Summary: The branch-4.1 BE UT job can consume the master automation prebuilt, whose unversioned Arrow/Paimon stack no longer satisfies the shared Arrow 17/24 layout. The download path also certified only the versioned stack, library stripping assumed lib was a symlink, and record batch readers ended early for a leading empty row group. Rebuild missing stacks in a source checkout, validate both stacks before reuse or replacement, strip explicit library directories, and skip empty row groups before reading the next batch. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - `./thirdparty/test/arrow-paimon-lifecycle-test.sh` - Arrow 17 and Arrow 24 `GetRecordBatchReaderSkipsEmptyLeadingRowGroup` tests - `./thirdparty/build-thirdparty.sh -j 8 arrow_17 paimon_cpp_17 arrow paimon_cpp` - `./build-support/check-format.sh` - Shell syntax and `git diff --check` - Behavior changed: Yes. Shared prebuilt reuse now requires both Arrow/Paimon stacks, and empty row groups no longer terminate the patched reader early. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66546 Problem Summary: Shared Arrow/Paimon recovery could validate the default tree while BE UT consumed independently overridden ABI roots, Paimon installation depended on the platform default libdir, Arrow empty-row-group skipping left Paimon on a zero-width logical range, and archive validation accepted incomplete SDK artifacts. Select one versioned build home for all BE entrypoints, fix Paimon installation at lib64, synchronize its row-group ranges with Arrow, and validate consumed headers plus nonempty static archives before publishing or installing a prebuilt. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - `./thirdparty/test/arrow-paimon-lifecycle-test.sh` - `./thirdparty/build-thirdparty.sh -j 8 arrow_17 paimon_cpp_17 arrow paimon_cpp` - `./thirdparty/build-thirdparty.sh -j 8 paimon_cpp_17 paimon_cpp` - Arrow 17 and Arrow 24 Paimon empty-row-group smoke tests with production static libraries - Paimon `file_reader_wrapper_test.cpp` compilation against Arrow 17 and Arrow 24 - `./build-support/check-format.sh` - Shell syntax and `git diff --check` - Behavior changed: Yes. BE build entrypoints enforce one ABI pair, shared prebuilts require consumable SDK artifacts, and Paimon skips zero-width row-group ranges. - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #66546
Problem Summary: Current branch-4.1 includes the Arrow 24 upgrade and can consume the same thirdparty installation as master. Installing Arrow 24 and Paimon against the legacy root prefix overwrites the Arrow 17 ABI expected by pre-upgrade branch-4.1 revisions and can leave mixed or stale artifacts after interrupted rebuilds. Keep Arrow 17 and Paimon 17 in the unversioned compatibility prefix, install the Arrow 24 and Paimon 24 ABI-coupled stack in a versioned prefix selected by the current branch, fingerprint and validate both stacks independently, and rebuild only the invalid stack. Preserve branch-4.1-specific build and test layout while backporting the shared-stack lifecycle coverage.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)