Skip to content

Conversation

@roeap
Copy link
Collaborator

@roeap roeap commented Oct 3, 2025

Description

We currently create a dedicated session with a dedicated planner in some operations to collect metrics. This creates a situation where - even if we pass a session - we need to recreate the session on the fly, potentially replacing an external planner and thus potentially loosing support for UDFs or other custom nodes.

This PR consolidates the various extension planner we currently have into a single planner to eventually make it easy for us to centralise session creation or allow others to extend their own planner with the delta extension planners.

cc @abhi-airspace-intelligence

Related Issue(s)

part-of: #3799

@roeap roeap requested review from hntd187 and rtyler as code owners October 3, 2025 07:34
@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Oct 3, 2025
@roeap roeap requested a review from ion-elgreco October 3, 2025 07:34
@codecov
Copy link

codecov bot commented Oct 3, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.07%. Comparing base (fb73c6f) to head (a776a97).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3804      +/-   ##
==========================================
+ Coverage   76.05%   76.07%   +0.02%     
==========================================
  Files         145      145              
  Lines       45274    45313      +39     
  Branches    45274    45313      +39     
==========================================
+ Hits        34433    34472      +39     
  Misses       9150     9150              
  Partials     1691     1691              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@roeap roeap force-pushed the refactor/ext-planners branch from e87e09c to 48e5ef9 Compare October 3, 2025 07:46
@ion-elgreco
Copy link
Collaborator

Correct me if I am wrong, but adding all extensions planner together will cause the plan become messy and do multiple metric counts

@roeap
Copy link
Collaborator Author

roeap commented Oct 3, 2025

Correct me if I am wrong, but adding all extensions planner together will cause the plan become messy and do multiple metric counts

The individual planners, while all downcasting to the same MetricObserver node, also match on unqiue IDs on the node for their respective metrics, as long as these don't overlap, only one planner should ever return a plan..

In case there are duplicate IDs we would potentially match wrong depending on the first planner that signifies it can plan the extension node (returning Some(...)). However we do always plan a node at most once .

@ion-elgreco
Copy link
Collaborator

Correct me if I am wrong, but adding all extensions planner together will cause the plan become messy and do multiple metric counts

The individual planners, while all downcasting to the same MetricObserver node, also match on unqiue IDs on the node for their respective metrics, as long as these don't overlap, only one planner should ever return a plan..

In case there are duplicate IDs we would potentially match wrong depending on the first planner that signifies it can plan the extension node (returning Some(...)). However we do always plan a node at most once .

Yeah, I thought our IDs werent unique but it's been a while that I've touched these extension planners

@roeap
Copy link
Collaborator Author

roeap commented Oct 3, 2025

it's been a while that I've touched these extension planners

same 😆

In a post kernelized world, I do believe we'll see much more of the likes though ...

@roeap roeap enabled auto-merge (squash) October 3, 2025 08:31
@roeap roeap merged commit c5eb4c2 into delta-io:main Oct 3, 2025
62 of 66 checks passed
@roeap roeap deleted the refactor/ext-planners branch October 3, 2025 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/rust Issues for the Rust crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants