Allow changing a column non-nullable to nullable in `SchemaMode::Merge` mode.

[debajyoti-truefoundry]

Environment

Delta-rs version: 0.27.0

Binding: I am using the Rust library

---

Bug

What happened:
delta-rs does not allow making a column non-nullable to nullable with SchemaMode::Merge.

What you expected to happen:
delta-rs should allow making a column non-nullable to nullable. Nullable to non-nullable should not be permitted with a concept of a default value (Not in the scope of this issue).

How to reproduce it:

#[tokio::test]
async fn null_to_non_null_column() {
    let schema_without_null_column = vec![StructField::new("id", DeltaDataType::INTEGER, false)];

    let non_null_batch = RecordBatch::try_from_iter_with_nullable(vec![(
        "id",
        Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef,
        false,
    )]);

    let null_batch = RecordBatch::try_from_iter_with_nullable(vec![(
        "id",
        Arc::new(Int32Array::from(vec![Some(1), None, Some(3)])) as ArrayRef,
        true,
    )]);

    let ops = DeltaOps::try_from_uri("memory:///").await.unwrap();
    let table = ops
        .create()
        .with_table_name("my_table")
        .with_save_mode(SaveMode::Ignore)
        .with_columns(schema_without_null_column)
        .await
        .unwrap();

    DeltaOps::from(table.clone())
        .write(non_null_batch)
        .with_schema_mode(SchemaMode::Merge)
        .with_save_mode(SaveMode::Append)
        .await
        .unwrap();

    DeltaOps::from(table)
        .write(null_batch)
        .with_schema_mode(SchemaMode::Merge)
        .with_save_mode(SaveMode::Append)
        .await
        .unwrap();
}

The above code raises InvalidData { violations: ["Non-nullable column violation for id, found 1 null values"] }.

More details:

delta-rs/crates/core/src/kernel/schema/cast/merge_schema.rs

Line 258 in ef9e077

match new_field.try_merge(right) {

https://github.com/apache/arrow-rs/blob/6c1b96f877cbab99ae4dfcbfd503590323bf83cb/arrow-schema/src/field.rs#L825C9-L825C40

[rtyler]

The non-nullable to nullable transition should be fine, according to @roeap. This is something I will need to poke at the Delta/Spark implementation to see how they're handling this for consistency's sake

[FrankPortman]

related-ish? #3744 and the fix #3747

I was noticing automatic non-nullable -> nullable coercion with SchemaMode::None and Mode:Overwrite which I consider a bug and inconsistent with Delta/Spark land.

Not sure what #3747 does to this issue though.