Skip to content

[opt](lance) add some profiles in lance node - #67005

Merged
yiguolei merged 6 commits into
apache:branch-4.1from
zhangstar333:lance-count
Aug 27, 2026
Merged

[opt](lance) add some profiles in lance node#67005
yiguolei merged 6 commits into
apache:branch-4.1from
zhangstar333:lance-count

Conversation

@zhangstar333

@zhangstar333 zhangstar333 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

  1. update lance version to 0.1.7
  2. add some profiles in lance node.

Release note

None

Check List (For Author)

  • Test

    • [] Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@zhangstar333
zhangstar333 requested a review from yiguolei as a code owner August 20, 2026 14:32
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: four concrete issues remain.

Correctness and compatibility: the new metadata COUNT path can return a wrong COUNT(*) for supported Lance fragments whose deletion count is absent from metadata. Snapshot pinning, COUNT argument classification, pushed/residual predicate gating, runtime-filter gating, and zero/multi-fragment advancement otherwise checked out.

Lifecycle: the new metadata shortcut bypasses FileScannerV2's asynchronous IO stop signal. Scanner/dataset reuse, split reset, EOF/error/abort cleanup, Arrow stream ownership, and timer RAII otherwise checked out.

Profiles and physical-plan diagnostics: the changed InitReader average is re-averaged across relation request blocks, and the new FlatKnn counter can label an unsegmented default-index fallback as flat. The new Lance timer units, per-task ownership, serialization, optional mixed-version keys, and backend aggregation otherwise checked out.

Tests: the added tests exercise current/synthetic no-deletion metadata, a null IO context, and preformatted profile strings, so they do not cover the four failing branches. No additional user focus was supplied. Review coverage is complete and converged after two rounds: both normal full-coverage reviewers and the separate risk reviewer returned NO_NEW_VALUABLE_FINDINGS in Round 2. Builds were not run, per the review-task instructions.

Comment thread be/src/exec/rowid_fetcher.cpp
Comment thread be/src/format_v2/table/lance_reader.cpp Outdated
Comment thread be/src/format_v2/table/lance_reader.cpp Outdated
@zhangstar333 zhangstar333 changed the title [opt](lance) support count(*) pushdown in lance and add profiles [opt](lance) add some profiles in lance node Aug 21, 2026
Comment thread be/src/exec/operator/materialization_opertor.cpp
@zhangstar333

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for six new findings (two P1 and four P2): a deterministic C++ type-declaration compile conflict, stale distribution/license metadata for the lance-c bump, incomplete execution statistics on successful early-stop scans, ambiguous fragment-local top-k labels, loss of vector-column identity in EXPLAIN, and Lance metrics attached outside the V2 profile ownership hierarchy.

The existing live row-ID average thread at rowid_fetcher.cpp:1132 still applies and was intentionally not duplicated. I reviewed all changed files plus the required V2 design/checklist, callback lifecycle, row-ID profile propagation, dependency patch/API/ABI/license state, and current CI evidence. The macOS-arm64 failure occurs earlier in unrelated libunwind setup and is not a basis for these findings. No builds were run, as required by the review bundle.

Comment thread be/src/format_v2/table/lance_reader.h Outdated
Comment thread be/src/format_v2/table/lance_reader.cpp Outdated
Comment thread thirdparty/vars.sh
Comment thread be/src/format_v2/table/lance_reader.cpp
Comment thread be/src/format_v2/table/lance_reader.cpp Outdated
@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 76.36% (168/220) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.18% (31195/42052)
Line Coverage 58.36% (345814/592584)
Region Coverage 55.16% (288207/522532)
Branch Coverage 56.00% (129622/231472)

@yiguolei
yiguolei merged commit 8740cb4 into apache:branch-4.1 Aug 27, 2026
27 of 33 checks passed
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants