[Bug]: Kernel Error when vacuuming with Null Partition

[ekatz-BLA]

What happened?

This appears to be a regression from 1.1.4 to 1.2.0. The following example shows a small table with "year" as a partition column, and we append data with a null value for year. The data is correctly written out with the placeholder name, but when we try to vacuum the table we get a kernel error.

Expected behavior

Should be able to vacuum the files within the null partition

Operating System

macOS

Binding

Python

Bindings Version

1.2.1

Steps to reproduce

# Schema with a nullable partition column
schema = {
    "date": pl.Date,
    "player_id": pl.Int64,
    "year": pl.Int64,
}

# Data to append, including a null in the partition column
data = pl.DataFrame(
    {
        "date": [datetime.date(2023, 1, 3), datetime.date(2023, 1, 4)],
        "player_id": [3, 4],
        "year": [2023, None],  # Null partition
    },
    schema=schema,
)

# Append new data
print("Appending data with null partition...")
write_deltalake(
    table_path,
    data,
    partition_by=["year"],
    mode="append",
)
dt = DeltaTable(table_path)
files_before_optimize = dt.file_uris()

# # Optimize the table to make old files obsolete
dt.optimize.z_order(["date", "player_id"])
files_after_optimize = dt.file_uris()
assert files_before_optimize != files_after_optimize

# Vacuum the table
dt.vacuum(retention_hours=0, dry_run=False, enforce_retention_duration=False)

Relevant logs

An error occurred during vacuum: Generic error: Kernel error: Invalid argument error: Found unmasked nulls for non-nullable StructArray field "value"

[ion-elgreco]

@ekatz-BLA can you open an issue in this repo instead: https://github.com/delta-io/delta-kernel-rs

[ekatz-BLA]

@ekatz-BLA can you open an issue in this repo instead: https://github.com/delta-io/delta-kernel-rs

delta-io/delta-kernel-rs#1426