[feat](iceberg) HDFS lazy open + iceberg delete file file_size propagation - #66773
Open
cambyzju wants to merge 14 commits into
Open
[feat](iceberg) HDFS lazy open + iceberg delete file file_size propagation#66773cambyzju wants to merge 14 commits into
cambyzju wants to merge 14 commits into
Conversation
cambyzju
requested review from
924060929,
CalvinKirs,
Gabriel39,
gavinchou,
liaoxin01,
luwei16,
morningman and
yiguolei
as code owners
August 14, 2026 07:33
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17642 ms |
Contributor
TPC-DS: Total hot run time: 85447 ms |
Contributor
ClickBench: Total hot run time: 14.6 s |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
|
/review |
cambyzju
force-pushed
the
opt-master-lazyopen-260814
branch
from
August 19, 2026 01:28
9c0e9b1 to
8f93283
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-20T03:35:00Z. Please trigger /review again after that time. |
cambyzju
force-pushed
the
opt-master-lazyopen-260814
branch
from
August 19, 2026 14:59
143126e to
71bb025
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17212 ms |
Contributor
TPC-DS: Total hot run time: 83307 ms |
Contributor
ClickBench: Total hot run time: 16.33 s |
Contributor
TPC-H: Total hot run time: 17523 ms |
Contributor
TPC-DS: Total hot run time: 83680 ms |
Contributor
ClickBench: Total hot run time: 16.45 s |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-DS: Total hot run time: 83748 ms |
Contributor
ClickBench: Total hot run time: 16.27 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Gabriel39
previously approved these changes
Aug 22, 2026
Contributor
Author
|
run check_coverage |
…ileSystem coverage
…ard null _handle in read_at_impl Two review comments on HDFS lazy open: 1. HdfsFileHandle::ensure_open() called hdfs_error() outside the call_once lambda, so waiting callers would query libhdfs thread-local last-error in their own thread context and see empty/unrelated errors. Move hdfs_error() and Status construction inside the lambda, store as _open_status, return the same Status to every caller. 2. HdfsFileReader::read_at_impl() called _handle->ensure_open() before any null guard. After a failed read sets _handle=nullptr, a subsequent read would dereference null. Add null guard before ensure_open(). Tests: - OpenFailurePreservesStatusInsideCallOnce: serial test verifies _open_status is preserved across calls (hdfs_error mock changed between calls). - ConcurrentOpenFailureReturnsSameStatusToAllCallers: 8 threads verify all callers see NOT_FOUND (simulates libhdfs thread-local last-error). - SecondReadAfterFailureDoesNotCrash: second read_at after first failed read returns InternalError instead of crashing. Also refactor MockHandleGuard to use a set_mock_return helper.
cambyzju
force-pushed
the
opt-master-lazyopen-260814
branch
from
August 31, 2026 07:54
cdcf29c to
0072668
Compare
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16796 ms |
Contributor
TPC-DS: Total hot run time: 81392 ms |
Contributor
ClickBench: Total hot run time: 14.67 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
When block cache hits, HdfsFileReader still calls hdfsOpenFile on every construction, wasting NameNode RPCs for data that's never read from HDFS.
How To Fix：
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)