chore: modified add_actions_table to stream

[zen-zap]

Description

Changed add_actions_table to return BoxStream in crates/core/src/table/state.rs. This helps by making sure we don't load everything into memory all at once. Added BoxStreamToReaderAdapter in python/src/reader.rs to convert the BoxStreams into RecordBatchReader. Modified the python bindings and tests to use .read_all() to fetch them all at once.

Related Issue(s)

Some tests fail I think. I got this:
438 passed, 4 skipped, 2 xfailed, 5 xpassed, 22 warnings in 14.72s

• closes Move add_actions_table onto stream #3811

[codecov]

Codecov Report

❌ Patch coverage is 0% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 24.98%. Comparing base (f1727c9) to head (b05a9ed).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/table/state.rs	0.00%	37 Missing ⚠️
crates/core/src/delta_datafusion/find_files.rs	0.00%	5 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (f1727c9) and HEAD (b05a9ed). Click for more details.

HEAD has 3 uploads less than BASE

Flag	BASE (f1727c9)	HEAD (b05a9ed)
	8	5

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #3836       +/-   ##
===========================================
- Coverage   74.37%   24.98%   -49.39%     
===========================================
  Files         147      120       -27     
  Lines       39670    20270    -19400     
  Branches    39670    20270    -19400     
===========================================
- Hits        29503     5065    -24438     
- Misses       8777    14834     +6057     
+ Partials     1390      371     -1019     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ion-elgreco]

This action already materializes all batches into memory

[zen-zap]

Sorry, I missed that.

I did make some changes to give BoxStream a 'static lifetime parameter and tried to have it stream instead of collecting it all.

[abhiaagarwal]

I don't think this PR is safe.

Sure, the 'static lifetime helps avoid the lifetime issues, but at the cost of essentially cloning the table's file state at "a point in time". This means that if you get the record batch stream, then perform operations on the delta table (which may delete a file, say a compact) and then you try to consume the stream, you could be reading a file that got invalidated.

Someone should double-check my logic, but a 'static lifetime is a bit scary unless you can prove it really does live for the duration of the program.

[abhiaagarwal]

I guess that these are log files, it's pretty rare they get deleted, but it still would get stale table state if another action added some files. Maybe that's the right abstraction on the python side, but definitely not the rust side.

[zen-zap]

Maybe I could take a snapshot like - get the current version of the delta table and start the stream from that? If I tie it to the version, it should be okay I think..

Do you have any suggestions?

[abhiaagarwal]

The only real way to do it in a safe way would be to have the BoxStream lifetime be 'a which is tied into &'a self, so the boxstream borrows from self.

pub fn add_actions_table<'a>(
    &'a self,
    flatten: bool,
) -> BoxStream<'a, DeltaResult<arrow::record_batch::RecordBatch>> {
    ...
}

[abhiaagarwal]

Now that I think about it, in theory, the self.files parameter should never change per snapshot so perhaps this is safe, but it's not enforced in the type system. On the python side though, this race condition absolutely does exist, though I'm not sure what the proper semantics are anyways

[zen-zap]

I think version pinning would help dealing with the race condition on rust side. The python side uses this so that should get fixed. Let me know if you have any ideas about this. Thanks!