v0.5.0

[kevinzwang]

‼️ v0.4 -> v0.5 Migration Guide ‼️

General

• The getdaft Python package was deprecated in v0.4 and is now unsupported. Please import the daft package instead
• The PyRunner was deprecated in v0.4 and is now removed. Please use the native runner instead, which is now the default runner for local execution:
  • Change the DAFT_RUNNER env var from py to native
  • Use daft.context.set_runner_native() instead of daft.context.set_runner_py()

Python API

• daft.Series

  • Passing a Series into daft.lit or daft.Expression methods is no longer supported.

• daft.sql.SQLCatalog

  • This class is marked as deprecated and will be removed in v0.6. You can now use keyword arguments with daft.sql(...) to add DataFrames to the query:

  # before
  catalog = daft.sql.SQLCatalog({ "my_df": df })
  daft.sql("SELECT * FROM my_df", catalog=catalog)
  
  # after
  daft.sql("SELECT * FROM my_df", my_df=df)

• These functions in daft.catalog were deprecated in v0.4 and have now been removed.

  • daft.catalog.read_table - use daft.read_table instead
  • daft.catalog.register_table - use daft.attach_table instead
  • daft.catalog.register_python_catalog - use daft.attach_catalog instead
  • daft.catalog.unregister_catalog - use daft.detach_catalog instead

• These functions in daft.io.catalog are deprecated and marked for removal in v0.6. Please use our unified catalog API instead.

  • daft.io.catalog.DataCatalog
  • daft.io.catalog.DataCatalogTable

What's Changed 🚀

✨ Features

• feat(catalogs): Rust-based in-memory catalog and table @kevinzwang (feat(catalogs): Rust-based in-memory catalog and table #4445)
• feat(flotilla): Materialize pipeline @colin-ho (feat(flotilla): Materialize pipeline #4457)
• feat: adds python partition fields to the DataSource API @rchowell (feat: adds python partition fields to the DataSource API #4449)
• feat: adds deserialize and try_deserialize with support for json @rchowell (feat: adds deserialize and try_deserialize with support for json #4438)
• feat(flotilla): LogicalPlan to PipelineNode translation @srilman (feat(flotilla): LogicalPlan to PipelineNode translation #4442)
• feat: Flotilla scheduler and dispatcher actors @colin-ho (feat: Flotilla scheduler and dispatcher actors #4375)
• feat: enables scalar function lowering e.g. function overloads @rchowell (feat: enables scalar function lowering e.g. function overloads #4431)
• feat(flotilla): Additional PipelineNodes for map pipelines @srilman (feat(flotilla): Additional PipelineNodes for map pipelines  #4439)
• feat: Flotilla default scheduler @colin-ho (feat: Flotilla default scheduler #4376)

🐛 Bug Fixes

• fix: Treat series literals as list type @colin-ho (fix: Treat series literals as list type #4370)

♻️ Refactor

• refactor(ordinals): move binding step above micropartition + local execution @kevinzwang (refactor(ordinals): move binding step above micropartition + local execution #4425)

👷 CI

• ci: remove publish with name 'getdaft' @kevinzwang (ci: remove publish with name 'getdaft' #4456)

🔧 Maintenance

• chore: Remove the PyRunner @srilman (chore: Remove the PyRunner #4458)
• chore: adds CTE map to daft.sql and removes deprecated catalog APIs @rchowell (chore: adds CTE map to daft.sql and removes deprecated catalog APIs #4460)

Full Changelog: v0.4.18...v0.5.0

---

This discussion was created from the release v0.5.0.