reports of many WARNs with "read_json receiver end of channel dropped before sending completed"

[zachschuermann]

From SpiceAI guys: they were seeing lots of WARN delta_kernel::engine::default::json: read_json receiver end of channel dropped before sending completed in 0.14

will add more details

[zachschuermann]

cc @lukekim @phillipleblanc

[roeap]

@zachschuermann - I think this might be somewhat "expected" as the sender is not cancelled when the receiver is dropped so we would expect this for most P&M replays or whenever we might not fully exhaust an iterator?

For delta-rs I adopted some code from datafusion which handles this, specifically the ReceiverStreamBuilder and thought about upstreaming this to delta-kernel. However I also wanted to get rid of the task executor before doing that :D.

[NightTrain]

I've got small reproducer using Databricks to write the delta.

Create a dummy table using:

CREATE TABLE catalog_1.schema_1.my_delta_table (
column1 INT,
column2 STRING
);

INSERT INTO catalog_1.schema_1.my_delta_table (column1, column2) VALUES
(1, 'Row1'),
(2, 'Row2');

This generates a delta table with the following properties:

This does not reproduce the issue.

Then add liquid clustering using:

ALTER TABLE catalog_1.schema_1.my_delta_table CLUSTER BY (column1);
OPTIMIZE catalog_1.schema_1.my_delta_table FULL;

This adds additional table properties:

Once liquid clustering is added, the warnings start to happen.

[tcldr]

As an additional data point, I'm using plain delta-rs via rust (so no liquid clustering as far as I'm aware) and getting the same warnings.

It seems to happen on tables which have the save mode set to SaveMode::Overwrite. I then feed them a stream of arrow record batches in append mode via the DeltaOps write builder.

Not seeing it on my other delta tables.

[tfostertexel]

I also have this issue with a delta table when I read it with 1.2.0 or later. Reading with deltalake 1.1.4 does not produce the same warnings. This is a delta table written with delta-rs from a polars dataframe. There are no issues I can see when reading the delta table either with polars or pyspark, just these warnings.

[wleftwich]

I got the same warning with delta-rs (python) after restoring a table version. deltalake 1.2.1, polars 1.34