[Bug]: `write_deltalake` fails when writing date64 type

[diederikperdok]

What happened?

The below Python code raises

_internal.DeltaError: Kernel error: Json error: whilst decoding field 'minValues': whilst decoding field 'sales_date': failed to parse 1760918400000 as Date32"

on python 3.13.7, deltalake 1.2.0 and pyarrow 21.0.0.

from datetime import date
from tempfile import TemporaryDirectory

import pyarrow as pa
from deltalake import write_deltalake

with TemporaryDirectory() as tmp_dir:
    delta_table_path = f"{tmp_dir}/DELTA"
    schema = pa.schema([pa.field("sales_date", pa.date64())])
    table = pa.table([pa.array([date(2025, 10, 20)])], schema=schema)

    write_deltalake(delta_table_path, table)

When changing the type to pa.date32 it works as expected. Explicitly creating the data as date64 before writing (pa.table([pa.array([date(2025, 10, 20)], type=pa.date64())], schema=schema)) does not make a difference.

Expected behavior

The table should be written without error. Reading it back with

Deltalake(delta_table_path).to_pandas()

should result in a dataframe with 1 row and a column "sales_date" with value "2025-10-20"

Operating System

macOS

Binding

Python

Bindings Version

1.2.0

Steps to reproduce

Execute my code example as is

Relevant logs

[ion-elgreco]

cc @roeap this worked afaik before deeper kernel adoption

[ion-elgreco]

@diederikperdok can you check at which version this exactly stopped working?

[roeap]

After a quick glance the error seems quite straightforward, but not sure how to best fix this.

i.e. the arrow to kernel schema conversion converts both date32 and date64 to deltas date type. In delta there is only one date type. When reading back the data, deltas date type is always converted to date32. Sine we allowed that data to be written though it is now in a mismatching format.

Could we check if spark can read this table? If not i fear we may have to live with that regression. IMO potentially breaking tables for other readers is not acceptable. If it can read it, we would likely need to default to the wider type somehow.

[ion-elgreco]

@roeap date64 should simply be downcasted to date32, it's just a loss of precision. Parquet only has int32 data type so when writing date64 afaik, its gets casted regardless

[roeap]

@ion-elgreco - just checked it out and it seems we can store 64 bit ints in parquet. w.r.t. casting i fear this would potentially lead to loss of information of overflows if we try to cast? This is days since unix epoch so anything above i32::MAX would overflow or alter the data, no?

[diederikperdok]

@diederikperdok can you check at which version this exactly stopped working?

That should be 0.25.0 or 0.25.1 (because 0.24.0 works, 0.25.1 doesn't and 0.25.0 is not available for Linux)

[ion-elgreco]

Hmm I only see a minor bump of kernel between 0.24.0 and 0.25.0