dagster deltalake packages broken due to deltalake upgrade to 1.0 #30337
Description
What's the issue?
dagster-deltalake-polars is failing at the moment due to the update of delta lake to 1.0.
The error i am seeing is:
The above exception was caused by the following exception:
AttributeError: type object 'deltalake._internal.Schema' has no attribute 'from_pyarrow'. Did you mean: 'from_arrow'?
File "/usr/src/app/.venv/lib/python3.12/site-packages/dagster_deltalake/handler.py", line 53, in handle_output
delta_schema = Schema.from_pyarrow(reader.schema)
What did you expect to happen?
I had processes running normally in production until our latest deployment. i traced it back to deltalake being upgraded from 0.25.5 to 1.0.0
How to reproduce?
I am using dagster-deltalake and dagster-deltalake-polars version 0.26.13
I fixed my issue by pinning deltalake==0.25.5
without pinning, the packages will use deltalake 1.0.0
Dagster version
dagster, version 1.10.13
Deployment type
Other Docker-based deployment
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.