2025-10-02 kernel meeting notes

[zachschuermann]

Summary

panic in delta-rs fixed, mem investigation

Attendees

@zachschuermann @roeap @nicklan @OussamaSaoudi @rtyler @mightyshazam @DrakeLin @scovich

notes stream

• following up on larger table causes overflow deep in arrow land delta-rs#3767
  • robert has fix!
• possibly related issue: memory usage +100% (doubling-ish) (TODO do we have an issue?)
• robert: stats/parsed_stats both tracked, need to drop one
• need more memory debug
• robert: parse_json into expression framework? StringArray col into json, now StructArray into json
  • todo: PR to move, issue to improve (both kernel and arrow)
• tyler: ask to split out default engine [wip] split out default engine into separate crate #941, [WIP] split out default engine to a separate crate #914
  • split out DefaultEngine to a separate delta_arrow crate #847
  • zach will share doc
• async APIs? hard to work with from async rust right now, potential duplication - robert
• should log replay be single core? no.
• [Pathfinding]: Dataframe #1257
• ryan's 2-phase log replay. read commit deltas + checkpoint manifest do on driver, distribute checkpoint parts reads
  • [VERY WIP] Pathfinding for distributed log replay #1160

todos from last time

• main issue: can we mitigate panic in delta-rs (taking above as follow-up)
• secondary issue: debug memory usage issue
• follow-up: do a test binary to hand to tyler

leftovers/follow-ups

• @zachschuermann to update summer 2025 kernel work #1184
• kernel side follow up: to check on checkpoint data iterator shouldn't return batches with different schemas #1099
• ask: why does daft/ray need serializing/deserializing ListedLogFiles: need Snapshot to be serializable/deserializable
  • basically just need some IO-less way to make these things
  • consumers: Daft (delta-rs) and Ray (deltalake/python)
• follow up on Urls discussion?