2025-09-18 kernel meeting notes

[zachschuermann]

Summary

0.16.0 today/tomorrow?

Attendees

@zachschuermann @roeap @mightyshazam @DrakeLin @MannDP @hntd187 @nicklan @scovich

notes stream

• @rtyler: possible bug larger table causes overflow deep in arrow land delta-rs#3767
  • @roeap: delta-rs concats everything? maybe we have offset overflow?
  • need to peek the checkpoint/transaction log (5GB log, 4GB checkpoint)
  • large file stats (MB-GB of stats for strings)
  • 1.6 (ish?) billion rows active in the table (data files)
  • 674 in a snapshot.files (is that record batches or files?)
  • ideally we should somehow share tables. can we get access to Scribd tables?
• For @mightyshazam - people asking about dotnet connector
• @zachschuermann: 0.16.0 soon! (hopefully)
  • new transforms (largest breaking change)
  • new snapshot builder APIs
  • row tracking
  • log compaction writes
  • lots more!
• @zachschuermann to update summer 2025 kernel work #1184

leftovers/follow-ups

• kernel side follow up: to check on checkpoint data iterator shouldn't return batches with different schemas #1099
• ask: why does daft/ray need serializing/deserializing ListedLogFiles
  • need Snapshot to be serializable/deserializable
  • basically just need some IO-less way to make these things
  • consumers: Daft (delta-rs) and Ray (deltalake/python)
• follow up on Urls discussion?