`write_deltalake` with `schema_mode=None` will overwrite `nullable` properties of columns

[FrankPortman]

Environment

Delta-rs version: 1.1.4

Binding: Python

Environment:

• Cloud provider: Azure/Local
• OS: Mac/Linux
• Other:

---

Bug

What happened: Instantiating a Delta table with a specific schema, including nullable=False columns, and later writing to it via write_deltalake with mode="overwrite", schema_mode=None, and predicate=.<some predicate to emulate replaceWhere>, leads to overwriting the nullability properties of the columns.

What you expected to happen: Coercing all column safely to the types that match the existing Delta table, including adhering to existing nullability properties. In the case where all columns and dtypes from the to-be-written data already match the Delta table on disk, the nullability of the to-be-written data shouldn't even be considered. It should only validate data via the Delta protocol based on the existing schema if the existing schema says some columns are not nullable.

How to reproduce it: https://gist.github.com/FrankPortman/4a967f5bcb0bf5e7136eff5087ed6880

More details:

I went back and forth on whether this should be filed as a bug or not, but the behavior IMO is unexpected, and it behaves differently from Spark (setting no schema mode, attempts all safe column coercions to existing types but doesn't override nullability). Also this is in the Rust code /// whether to overwrite the schema or to merge it. None means to fail on schmema drift, and IMO failing on schema drift does not implicitly mean to overwrite column nullability.

[FrankPortman]

Followup - this does not happen with write_deltalake(mode="append"), only write_deltalake(mode="overwrite").

[FrankPortman]

Unclear if merge is impacted, can investigate that tomorrow.

[ion-elgreco]

Could you put in a fix for it?

[FrankPortman]

I can try but I am very unfamiliar with Rust. Do you have some general pointers on where you think the issue is and/or which files would need to be looked at? Then I can try putting up a "work in progress" PR and we can discuss from there.

[ion-elgreco]

@FrankPortman likely here:

delta-rs/crates/core/src/operations/write/mod.rs

Line 625 in d148c32

if &delta_schema != snapshot.schema() {

the eq implementation problaby doesn't consider nullability.

[FrankPortman]

@ion-elgreco thanks - do you know why there is even a codepath around overwriting metadata for schema_mode=None just because write mode is "overwrite"? Perhaps that is the unintended behavior? Or is that load bearing for something else?

[ion-elgreco]

@ion-elgreco thanks - do you know why there is even a codepath around overwriting metadata for schema_mode=None just because write mode is "overwrite"? Perhaps that is the unintended behavior? Or is that load bearing for something else?

Looks unintended, we also check the schema change twice, so I think that should be collapsed into a multiple if that can trigger the schema action creation

[FrankPortman]

Put up an initial PR here: #3747