-
Couldn't load subscription status.
- Fork 118
Open
Description
Describe the bug
This appears to be a regression from that shows up when upgrading delta-rs from 1.1.4 to 1.2.0. The following example shows a small table with "year" as a partition column, and we append data with a null value for year. The data is correctly written out with the placeholder name, but when we try to vacuum the table we get a kernel error.
To Reproduce
# Schema with a nullable partition column
schema = {
"date": pl.Date,
"player_id": pl.Int64,
"year": pl.Int64,
}
# Data to append, including a null in the partition column
data = pl.DataFrame(
{
"date": [datetime.date(2023, 1, 3), datetime.date(2023, 1, 4)],
"player_id": [3, 4],
"year": [2023, None], # Null partition
},
schema=schema,
)
# Append new data
print("Appending data with null partition...")
write_deltalake(
table_path,
data,
partition_by=["year"],
mode="append",
)
dt = DeltaTable(table_path)
files_before_optimize = dt.file_uris()
# # Optimize the table to make old files obsolete
dt.optimize.z_order(["date", "player_id"])
files_after_optimize = dt.file_uris()
assert files_before_optimize != files_after_optimize
# Vacuum the table
dt.vacuum(retention_hours=0, dry_run=False, enforce_retention_duration=False)Relevant logs
An error occurred during vacuum: Generic error: Kernel error: Invalid argument error: Found unmasked nulls for non-nullable StructArray field "value"Expected behavior
Should be able to vacuum the files within the null partition
Additional context
No response
Metadata
Metadata
Assignees
Labels
No labels