Skip to content

Conversation

@roeap
Copy link
Collaborator

@roeap roeap commented Oct 6, 2025

Description

part-of: #3733

Most friction in integrating kernel arises when we need to translate kernels blocking iterators to streams to conform to our compute model.

Thus far we had an extension trait for kernel scans to translate the scan iterators into arrow record batches. However when building our own scans, we need to process more of the scan metadata.

To facilitate implementation of kernel based scans we now provide shims around kernels Scan and ScanBuilder to expose their operations via more idiomatic abstractions without altering the returned data.

@roeap roeap requested a review from ion-elgreco October 6, 2025 11:53
@roeap roeap requested review from hntd187 and rtyler as code owners October 6, 2025 11:53
@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Oct 6, 2025
@codecov
Copy link

codecov bot commented Oct 6, 2025

Codecov Report

❌ Patch coverage is 69.38776% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.32%. Comparing base (b0eca62) to head (5733233).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/kernel/snapshot/scan.rs 63.44% 29 Missing and 5 partials ⚠️
crates/core/src/kernel/snapshot/mod.rs 79.24% 3 Missing and 8 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3818      +/-   ##
==========================================
- Coverage   74.38%   74.32%   -0.07%     
==========================================
  Files         146      147       +1     
  Lines       39479    39495      +16     
  Branches    39479    39495      +16     
==========================================
- Hits        29367    29354      -13     
- Misses       8714     8742      +28     
- Partials     1398     1399       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@roeap roeap enabled auto-merge (squash) October 6, 2025 12:02
@roeap
Copy link
Collaborator Author

roeap commented Oct 6, 2025

The test coverage dropped a bit - looking at the report though, this is mainly the unused "proxy-methods" that directly delegate ti fields in the inner types. THis will get covered as we start to use these abstractions for our own scans.

@roeap roeap merged commit 4301422 into delta-io:main Oct 6, 2025
25 of 26 checks passed
@roeap roeap deleted the feat/kernel-shims branch October 6, 2025 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/rust Issues for the Rust crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants