Skip to content

Update dagster-polars lockfile#314

Merged
cmpadden merged 5 commits into
mainfrom
update-dagster-polars-lockfile
May 22, 2026
Merged

Update dagster-polars lockfile#314
cmpadden merged 5 commits into
mainfrom
update-dagster-polars-lockfile

Conversation

@cmpadden

Copy link
Copy Markdown
Contributor

Summary

  • refresh dagster-polars uv.lock with latest compatible dependency versions

Testing

  • uv lock --upgrade in libraries/dagster-polars

@cmpadden

Copy link
Copy Markdown
Contributor Author

Pulled the failing check logs with gh. The failures were all AttributeError: '*UPath' object has no attribute '_kwargs' after the lockfile upgraded universal-pathlib to 0.3.x. Dagster's UPath IO manager still expects the 0.2.x API here, so I constrained universal-pathlib to <0.3 and regenerated the lockfile. Verified locally with uv run pytest dagster_polars_tests/test_polars_parquet.py::test_polars_parquet_io_manager_read_write -q and pushed the fix.

@cmpadden

Copy link
Copy Markdown
Contributor Author

Pulled the new failing check logs with gh after the UPath fix. The remaining failure was in test_polars_delta_native_multi_partitions with a Delta kernel Date32 <= Utf8View comparison error after upgrading deltalake to 1.6.0. Constrained the optional deltalake dependency to <1.6, regenerated the lockfile to 1.5.1, and verified locally with uv run pytest dagster_polars_tests/test_polars_delta.py::test_polars_delta_native_multi_partitions -q. Pushed the fix.

@cmpadden

Copy link
Copy Markdown
Contributor Author

Pulled the latest failing logs with gh. Delta Lake 1.5.1 still reproduced the CI-only multi-partition comparison error on Python 3.12. Tightened the optional deltalake constraint to <1.2, regenerating the lockfile to 1.1.4 (close to the previously working 1.1.x line). Verified the failing test locally again with uv run pytest dagster_polars_tests/test_polars_delta.py::test_polars_delta_native_multi_partitions -q and pushed the fix.

@cmpadden

Copy link
Copy Markdown
Contributor Author

Follow-up to avoid adding version caps: removed both temporary constraints (universal-pathlib<0.3 and deltalake<1.2) and updated the integration to work with the latest resolved versions instead.

Changes:

  • Override BasePolarsUPathIOManager.storage_options to use the public UPath.storage_options attribute when available, with a fallback to legacy _kwargs. This handles universal-pathlib 0.3 without pinning it back.
  • Generate typed Delta predicates for Dagster time-window partitions using DATE 'YYYY-MM-DD', avoiding the Delta 1.6 Date32 <= Utf8View comparison error.
  • Regenerated the lockfile back to latest universal-pathlib and deltalake.

Validation:

  • uv run ruff check
  • uv run ruff format
  • uv run pytest dagster_polars_tests/test_polars_delta.py::test_polars_delta_native_multi_partitions dagster_polars_tests/test_polars_parquet.py::test_polars_parquet_io_manager_read_write -q

A full local pytest run still needs the optional patito extra installed; CI covers the full workflow.

@cmpadden cmpadden merged commit f72afe4 into main May 22, 2026
7 checks passed
@cmpadden cmpadden deleted the update-dagster-polars-lockfile branch May 22, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant