Invalid argument error: all columns in a record batch must have the specified row count

[ryzhyk]

Environment

Delta-rs version: 0.22.2

Binding: Rust

Environment:

• Cloud provider: AWS
• OS: Linux
• Other:

---

Bug

We ran into this error when reading from a managed Databricks Delta Table:

Invalid argument error: all columns in a record batch must have the specified row count

This happens when running the select * from my_table query via datafusion. This occurs in the customer environment, so we don't have a reliable reproduction yet.

FWIW, I found a similar issue that someone ran into when reading from Iceberg, along with the explanation that the number of physical and logical records in a batch may not match:
apache/datafusion-comet#973

Is it possible that a similar issue exists with Delta?

Thanks!

[ion-elgreco]

@ryzhyk please provide an MRE, it's not clear how you are reading delta with Datafusion

[ryzhyk]

Thanks for your reply @ion-elgreco. I will work with the customer on a reproduction. Like I said, we don't have access to the delta table, so can't reproduce on our end. I was hoping that someone familiar with the code base might have some suggestions. Maybe there are some specific things we can ask the customer to check, e.g., could this happen if the table has deletion vectors enabled?

To query the DeltaLake using datafusion, we use register_table to register the Delta table provider with datafusion and then issue queries via

let options: SQLOptions = SQLOptions::new()
    .with_allow_ddl(false)
    .with_allow_dml(false);

let df = self.datafusion.sql_with_options("select * from my_table", options).await?;

let mut stream = match df.execute_stream().await?;

while let Some(batch) = stream.next().await {
    let batch = match batch {
        Ok(batch) => batch,
        Err(e) => {
            /*THE ERROR IS REPORTED HERE */
        }
    };
}

[ion-elgreco]

The TableProvider uses parquetExec underneath, so I think it's rather an issue surfacing from Datafusion.

Have you confirmed, whether you can read it properly with Polars in python or pyArrow datasets in python?

[ryzhyk]

Nope, I don't have a way to do this yet. I'll try it if I can get my hands on a table that causes this error.

[ryzhyk]

I can confirm that this was caused by the presence of deletion vectors in the table. The user worked around the problem by creating a copy of the tables without deletion vectors and it went away for a while. But they are now seeing the same error again in one table even without deletion vectors:

External error: Arrow error: Invalid argument error: all columns in a record batch must have the specified row count

As before I don't have access to the table, but the user shared the table config with us (it's a databricks managed table):

Any ideas what could potentially be causing this error?

PS. This is with delta-rs 0.24.

[ryzhyk]

according to the user, the errors seem correlated with autoCompact and optimizeWrite properties. Out of hundreds of tables they have this only seems to happen on tables where these are enabled.

[ion-elgreco]

This will be addressed in the next release since cliënt will hard fail on unsupported features

[ryzhyk]

Thanks, @ion-elgreco , this is good to know! I am aware that deletion vectors are unsupported, but I don't see any other unsupported features in the list I shared (based on supported features listed in the README).

PS. To clarify, the error now occurs on a table without deletion vectors.

[ion-elgreco]

Thanks, @ion-elgreco , this is good to know! I am aware that deletion vectors are unsupported, but I don't see any other unsupported features in the list I shared (based on supported features listed in the README).

PS. To clarify, the error now occurs on a table without deletion vectors.

It has column mapping enabled in your screenshot. I guess that might be the issue

[ryzhyk]

Thank you!

[ryzhyk]

This will be addressed in the next release since cliënt will hard fail on unsupported features

@ion-elgreco , is this functionality (compatibility check) already in main? Is there a list of supported features somewhere?