Skip to content

Dennis/polarsio partition metadata key#332

Open
dehume wants to merge 3 commits into
mainfrom
dennis/polarsio-partition-metadata-key
Open

Dennis/polarsio partition metadata key#332
dehume wants to merge 3 commits into
mainfrom
dennis/polarsio-partition-metadata-key

Conversation

@dehume

@dehume dehume commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary & Motivation

Fixes #330. PolarsDeltaIOManager with metadata={"partition_by": ...} regressed in 0.27.12: writing a TimeWindowPartitionsDefinition-partitioned asset failed with Cast error: Cannot cast string '2025' to value of Date32 type.

PR #314 changed get_predicate to emit a DATE '...' literal whenever the asset used a TimeWindowPartitionsDefinition:

if isinstance(partitions_def, TimeWindowPartitionsDefinition):
    return f"DATE '{key}'"

This assumes a time-window partition always maps to a Date column. But the column type is determined by the DataFrame being written, not the partition definition — a %Y yearly partition stored in a string year column produced year = DATE '2025', which deltalake can't cast against a string column.

How I Tested These Changes

  • Updated test_partition_filters_predicate to assert the new OR-chain predicate format.
  • Added test_polars_delta_native_partitioning_time_window_string_column — end-to-end reproduction of PolarsDeltaIOManager regression in 0.27.12 - partition_by metadata key breaks with Cannot cast string to Date32 #330 (yearly %Y → string column). Verified it fails on the old logic with the exact reported error.
  • Added test_polars_delta_native_partitioning_datetime_column_predicate — verifies the overwrite predicate works for a Datetime column. Verified it fails on the old logic with Invalid comparison operation: Timestamp(µs) <= Date32.
  • Existing Date-column and multi-partition tests continue to pass.
  • Full delta suite passes (21 tests); ruff and ty clean.

Changelog

Ensure that an entry has been created in CHANGELOG.md outlining additions, deletions, and/or modifications.

See: keepachangelog.com

@dehume dehume marked this pull request as ready for review July 15, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PolarsDeltaIOManager regression in 0.27.12 - partition_by metadata key breaks with Cannot cast string to Date32

1 participant