v0.4.0

[samster25]

What's Changed 🚀

• build: Use uv for maturin builds instead @raunakab (build: Use uv for maturin builds instead #3540)

💥 Breaking Changes

• feat: Default native runner @colin-ho (feat: Default native runner #3608)
• chore!: upgrade Ray pins and pyarrow pins @jaychia (chore!: upgrade pyarrow pin to >=8.0.0 #3612)
• chore!: drop support for Python 3.8 @kevinzwang (chore!: drop support for Python 3.8 #3592)
• chore!: remove pyarrow-based file reader @kevinzwang (chore!: remove pyarrow-based file reader #3587)

✨ Features

• feat: Default native runner @colin-ho (feat: Default native runner #3608)
• feat(swordfish): Progress Bar @colin-ho (feat(swordfish): Progress Bar #3571)
• feat(connect): df.show @universalmind303 (feat(connect): df.show #3560)
• feat(connect): support DdlParse @andrewgazelka (feat(connect): support DdlParse #3580)
• feat(swordfish): Optimize grouped aggregations @colin-ho (feat(swordfish): Optimize grouped aggregations #3534)
• feat(swordfish): Enable left/right joins to build probe table on either side @colin-ho (feat(swordfish): Enable left/right joins to build probe table on either side #3548)
• feat: Add DataType inference from Python types @jaychia (feat: Add DataType inference from Python types #3555)
• feat(shuffles): Locality aware pre shuffle merge @colin-ho (feat(shuffles): Locality aware pre shuffle merge #3505)
• feat: Implement count-distinct for sql @raunakab (feat: Implement count-distinct for sql #3553)
• feat(connect): add drop support @andrewgazelka (feat(connect): add drop support #3345)
• feat: support for basic subquery execution @kevinzwang (feat: support for basic subquery execution #3536)
• feat(connect): add df.filter @andrewgazelka (feat(connect): add df.filter #3346)
• feat: Make serialization code not unwrap and panic on failures @raunakab (feat: Make serialization code not unwrap and panic on failures #3546)
• feat: Unity Catalog writes using daft.DataFrame.write_deltalake() @anilmenon14 (feat: Unity Catalog writes using daft.DataFrame.write_deltalake() #3522)
• feat(connect): add parquet support @andrewgazelka (feat(connect): add parquet support #3360)
• feat: Add iterators to more types @raunakab (feat: Add iterators to more types #3539)
• feat(optimizer): Add scaffolding to create join graphs from logical plans @desmondcheongzx (feat(optimizer): Add scaffolding to create join graphs from logical plans #3501)
• feat(tpcds-benchmarking): Add basic tpcds benchmarking for local testing @raunakab (feat(tpcds-benchmarking): Add basic tpcds benchmarking for local testing #3509)
• feat(list): add fixed-size list support for value_counts @andrewgazelka (feat(list): add fixed-size list support for value_counts #3521)
• feat(parquet): Limit parallel tasks in remote parquet reader @colin-ho (feat(parquet): Limit parallel tasks in remote parquet reader #3490)
• feat(parquet): Target parquet writes by size bytes instead of rows @colin-ho (feat(parquet): Target parquet writes by size bytes instead of rows #3457)
• feat: cross join @kevinzwang (feat: cross join #3437)
• [FEAT] connect: remove excessive warnings from spark connect @universalmind303 ([FEAT] connect: remove excessive warnings from spark connect #3499)
• [CHORE] connect, test: df.withColumn @andrewgazelka ([CHORE] connect, test: df.withColumn #3359)
• [FEAT]: expr simplifier @universalmind303 ([FEAT]: expr simplifier #3393)
• [FEAT] shuffle testing @raunakab ([FEAT] shuffle testing #3492)
• [FEAT]: add coalesce to dataframe and SQL @universalmind303 ([FEAT]: add coalesce to dataframe and SQL #3482)
• [FEAT] add register-table helper to sql-catalog @chuanlei-coding ([FEAT] add register-table helper to sql-catalog #2837)
• [FEAT] Respect resource request for projections in swordfish @colin-ho ([FEAT] Respect resource request for projections in swordfish #3460)
• [FEAT] Enable Actor Pool UDFs by default @kevinzwang ([FEAT] Enable Actor Pool UDFs by default #3488)
• [FEAT] connect: add modulus operator and withColumns support @andrewgazelka ([FEAT] connect: add modulus operator and withColumns support #3351)
• [FEAT] connect: createDataFrame @andrewgazelka ([FEAT] connect: createDataFrame #3363)
• [FEAT] Support parquet RLE decoding for booleans @desmondcheongzx ([FEAT] Support parquet RLE decoding for booleans #3477)
• [FEAT] Cap parallelism on local parquet reader @colin-ho ([FEAT] Cap parallelism on local parquet reader #3310)
• [FEAT] connect: add binary operators @andrewgazelka ([FEAT] connect: add binary operators #3350)
• [FEAT] connect: support basic column operations @andrewgazelka ([FEAT] connect: support basic column operations #3362)
• [FEAT] extend build-commit workflow to support different compile-archs @raunakab ([FEAT] extend build-commit workflow to support different compile-archs #3459)
• [FEAT] Add count-distinct aggregation @raunakab ([FEAT] Add count-distinct aggregation #3455)

🐛 Bug Fixes

• fix(udf): udf call with empty table and batch size @kevinzwang (fix(udf): udf call with empty table and batch size #3604)
• fix: use arrow's schema instead of spark's for local rel @universalmind303 (fix: use arrow's schema instead of spark's for local rel #3602)
• fix: guard concurrent extension datatype setting with a lock @jaychia (fix: guard concurrent extension datatype setting with a lock #3589)
• fix(parquet): Fix parquet reads of required fields nested within optional fields @desmondcheongzx (fix(parquet): Fix parquet reads of required fields nested within optional fields #3598)
• fix: boolean and/or expressions with null @kevinzwang (fix: boolean and/or expressions with null #3544)
• fix(run-cluster-workflow): Add null check when parsing metadata @raunakab (fix(run-cluster-workflow): Add null check when parsing metadata #3507)
• fix(tpcds): fix bugs in tpcds datagen script @universalmind303 (fix(tpcds): fix bugs in tpcds datagen script #3495)
• [BUG] Fix build commit workflow @raunakab ([BUG] Fix build commit workflow #3487)
• [BUG]: dont panic on count(distinct) @universalmind303 ([BUG]: dont panic on count(distinct) #3481)
• [BUG] Block on parquet schema future in estimate_size_bytes @colin-ho ([BUG] Block on parquet schema future in estimate_size_bytes #3484)

🚀 Performance

• perf: filter null join key optimization rule @kevinzwang (perf: filter null join key optimization rule #3583)
• perf: lazily import pyiceberg and unity catalog if available @jaychia (perf: lazily import pyiceberg and unity catalog if available #3565)

♻️ Refactor

• refactor: allow InMemory to take in non python based entries @universalmind303 (refactor: allow InMemory to take in non python based entries #3554)
• refactor: create a rust based PartitionSet @universalmind303 (refactor: create a rust based PartitionSet  #3515)
• refactor(swordfish): Generic broadcast state bridge @colin-ho (refactor(swordfish): Generic broadcast state bridge #3508)

📖 Documentation

• docs: update tpch benchmark link @ccmao1130 (docs: update tpch benchmark link #3542)
• docs: Enable Linting of docstrings @samster25 (docs: Enable Linting of docstrings #3506)
• [FEAT] Enable Actor Pool UDFs by default @kevinzwang ([FEAT] Enable Actor Pool UDFs by default #3488)

✅ Tests

• test(connect): add more tests for createDataFrame @andrewgazelka (test(connect): add more tests for createDataFrame #3607)
• test: Add more size estimation tests from our s3 bucket @jaychia (test: Add more size estimation tests from our s3 bucket #3514)

👷 CI

• ci: Always download logs @jaychia (ci: Always download logs #3588)
• ci: Add ability to array-ify args and run multiple jobs @raunakab (ci: Add ability to array-ify args and run multiple jobs #3584)
• ci: Add "build" label type to accepted PR titles @raunakab (ci: Add "build" label type to accepted PR titles #3541)
• ci: add a tool to launch workloads on cluster @jaychia (ci: add a tool to launch workloads on cluster #3516)
• ci(release-drafter): use conventional commit labels @andrewgazelka (ci(release-drafter): use conventional commit labels #3503)

🔧 Maintenance

• chore!: upgrade Ray pins and pyarrow pins @jaychia (chore!: upgrade pyarrow pin to >=8.0.0 #3612)
• chore: add warning for native runner @jaychia (chore: add warning for native runner #3613)
• chore!: drop support for Python 3.8 @kevinzwang (chore!: drop support for Python 3.8 #3592)
• chore!: remove pyarrow-based file reader @kevinzwang (chore!: remove pyarrow-based file reader #3587)
• chore: Fix ordering in sql tests + pin docker images in read_sql tests @colin-ho (chore: Fix ordering in sql tests + pin docker images in read_sql tests #3596)
• chore: move symbolic and boolean algebra code into new crate @kevinzwang (chore: move symbolic and boolean algebra code into new crate #3570)
• [CHORE] use conventional commits @andrewgazelka ([CHORE] use conventional commits #3493)
• [CHORE] connect, test: df.withColumn @andrewgazelka ([CHORE] connect, test: df.withColumn #3359)
• [CHORE] Add tests for parquet size estimations @jaychia ([CHORE] Add tests for parquet size estimations #3405)
• [CHORE] Move all python wrapping logic to separate module @raunakab ([CHORE] Move all python wrapping logic to separate module #3458)

Full Changelog: v0.3.15...v0.3.16

---

This discussion was created from the release v0.4.0.