athena.to_iceberg: Type casting incompatible with pandas 2

[andreasbolstad]

Describe the bug

When you add a new column to an existing iceberg table using the athena.to_iceberg method, you get a conversion error:

TypeError: Casting to unit-less dtype 'datetime64' is not supported. Pass e.g. 'datetime64[ns]' instead.

This is caused by the mapping in the _data_types module's function athena2pandas. Specifically the code on lines 381-382:

if dtype in ("timestamp", "timestamp with time zone"):
    return "datetime64" if dtype_backend != "pyarrow" else "timestamp[ns][pyarrow]"

Potential fix:

return "datetime64[ns]", instead of datetime64 which is not supported for pandas 2+

How to Reproduce

Create an iceberg table. Then use the awswrangler athena to_iceberg method with a dataframe having a new datetime/timestamp column.

Expected behavior

No response

Your project

No response

Screenshots

No response

OS

Mac, Sequoia 15.6.1

Python version

3.11.13

AWS SDK for pandas version

3.11.0

Additional context

No response

[andreasbolstad]

Just to be clear. This bug occurs when you use the to_iceberg function to UPDATE an iceberg table with a new datetime column. It does not occur when you create an iceberg table for the first time.