Pandas 3 compat and more recent pyarrow versions

[fjetter]

This restores compatibility with pandas 3.x and tests against latest pyarrow versions

The changes can be summarized as

• Tests have to account for default datetime precision in pandas changing. All of our filtering logic is coercing properly from what I can tell so I don't expect any fallout
• Similarly, pandas loads strings now as str data type which is an extension array backed by pyarrow. Since parquet files are not impacted, this is purely a runtime type issue and future/backwards compatibility won't be affected
• The default string extension type is using pyarrow.large_string instead of pyarrow.string dtypes. This is indeed affecting our arrow level (possibly parquet?) code which is primarily user facing when casting values in predicate evaluation, i.e. filters might not work as expected on older plateau versions with newer pandas
• The biggest impact on the code base that actually required me to introduce compat code branches is a behavior in how pandas.groupby.apply deals with group_keys. I haven't found a way how to pass the group keys to the apply functions (even though there is a kwarg for this). While I checked for bug reports (there are none about this), I haven't deeply investigated where this is coming from. This may be a pandas bug or an expected breakage. Either way, we should be covered. This mostly affects the more sophisticated pipelines that shuffle dataframes before storage.
• I added more arrow versions to CI. We typically only support four versions but I feel an upcoming 5.0 release would be better suited for this and this PR is certainly not a good place for cutting other stuff. I think this should be a release before any intended breakage

[fjetter]

Things are green for the nightly as well, see https://github.com/data-engineering-collective/plateau/actions/runs/16490435763

[Copilot]

Pull Request Overview

This PR implements compatibility for Pandas 3.x and support for newer PyArrow versions (19.0.1 and 20.0.0). The changes focus on handling Pandas 3's new string inference behavior and PyArrow schema differences.

• Updates compatibility handling for Pandas 3's string type inference
• Adds support for PyArrow versions 19.0.1 and 20.0.0 with compatibility fixes
• Refactors deprecated DataFrame operations to use modern syntax

Reviewed Changes

Copilot reviewed 25 out of 30 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
plateau/core/_compat.py	Adds PANDAS_3 flag and pandas_infer_string() helper
tests/serialization/test_parquet.py	Updates string dtype handling for Pandas 3 compatibility
tests/serialization/test_filter.py	Adds datetime precision filtering tests
plateau/serialization/_parquet.py	Updates large_string type handling and coercion parameters
plateau/io_components/metapartition.py	Updates metadata schema for string inference
pyproject.toml	Updates PyArrow version constraints

[fjetter]

I'll update the date once approved and release afterwards

[xhochy]

🚢

[fjetter]

nightly build https://github.com/data-engineering-collective/plateau/actions/runs/16493436729