v0.3.15

[samster25]

Changes

✨ New Features

• [FEAT] run cluster on commit @raunakab ([FEAT] run cluster on commit #3461)
• [FEAT]: Support .clip function @conradsoon ([FEAT]: Support .clip function #3136)
• [FEAT] Add cluster profiles @raunakab ([FEAT] Add cluster profiles #3426)
• [FEAT] add pyiceberg 0.8.0 support @rongfengliang ([FEAT] add pyiceberg 0.8.0 support #3448)
• [FEAT] migrate schema inference → async, block at py boundary @andrewgazelka ([FEAT] migrate schema inference → async, block at py boundary #3432)
• [CHORE] connect: df.schema @andrewgazelka ([CHORE] connect: df.schema #3353)
• [CHORE] connect test: df.get_attr @andrewgazelka ([CHORE] connect test: df.get_attr #3349)
• [FEAT] Get native execution enablement from DAFT_RUNNER @desmondcheongzx ([FEAT] Get native execution enablement from DAFT_RUNNER #3409)
• [FEAT] Add ability to download log files from ray-cluster @raunakab ([FEAT] Add ability to download log files from ray-cluster #3406)
• [FEAT] Add ability to run arbitrary command on a set working directory @raunakab ([FEAT] Add ability to run arbitrary command on a set working directory #3404)
• [FEAT] Add steps to spin up, submit job, and spin down ray clusters @raunakab ([FEAT] Add steps to spin up, submit job, and spin down ray clusters #3403)
• [CHORE] connect: add tests for df.take() method @andrewgazelka ([CHORE] connect: add tests for df.take() method #3385)
• [FEAT] Create new run workflow @raunakab ([FEAT] Create new run workflow #3402)
• [FEAT] Enable group by keys in aggregation expressions @kevinzwang ([FEAT] Enable group by keys in aggregation expressions #3399)
• [FEAT] Build release python wheels and upload to AWS S3 @raunakab ([FEAT] Build release python wheels and upload to AWS S3 #3398)
• [FEAT] connect: Add support for select @andrewgazelka ([FEAT] connect: Add support for select #3344)
• [FEAT] connect: add df.limit and df.first @andrewgazelka ([FEAT] connect: add df.limit and df.first #3309)
• [FEAT] connect: to_daft_* use ref instead of value @andrewgazelka ([FEAT] connect: to_daft_* use ref instead of value #3355)
• [FEAT] connect: add alias support @andrewgazelka ([FEAT] connect: add alias support #3342)
• [FEAT] Filter predicates in SQL join @kevinzwang ([FEAT] Filter predicates in SQL join #3371)
• [FEAT] connect: collect @andrewgazelka ([FEAT] connect: collect #3326)

🚀 Performance Improvements

• [PERF] Improve hash table probe side decisions for Swordfish @desmondcheongzx ([PERF] Improve hash table probe side decisions for Swordfish #3327)

👾 Bug Fixes

• [BUG] Fix extension type display @jaychia ([BUG] Fix extension type display #3456)
• [BUG] Remove enum imports from match statements @raunakab ([BUG] Remove enum imports from match statements #3436)
• [BUG] Explicitly set IO config in unity catalog load table @colin-ho ([BUG] Explicitly set IO config in unity catalog load table #3453)
• [BUG] Include storage options in lance write commit @colin-ho ([BUG] Include storage options in lance write commit #3451)
• [BUG] Replace semicolons in filenames with underscore @raunakab ([BUG] Replace semicolons in filenames with underscore #3430)
• [BUG] Terminate nodes instead of stopping them @raunakab ([BUG] Terminate nodes instead of stopping them #3427)
• [BUG] Fix run-cluster passing in environment variables wrongly @jaychia ([BUG] Fix run-cluster passing in environment variables wrongly #3422)

📖 Documentation

• [FEAT]: Support .clip function @conradsoon ([FEAT]: Support .clip function #3136)
• [DOCS] Shorten union of Literals @desmondcheongzx ([DOCS] Shorten union of Literals #3449)
• [DOCS] Add missing list expression entries @desmondcheongzx ([DOCS] Add missing list expression entries #3428)

🧰 Maintenance

• [CHORE] Add warning in PyRunner to switch to Native @colin-ho ([CHORE] Add warning in PyRunner to switch to Native #3472)
• [CHORE] Address comments on previous PR @raunakab ([CHORE] Address comments on previous PR #3473)
• [CHORE] Write tpch parquet files one at a time @colin-ho ([CHORE] Write tpch parquet files one at a time #3396)
• [CHORE] Remove CountMode and ResourceRequest from public API @desmondcheongzx ([CHORE] Remove CountMode and ResourceRequest from public API #3429)
• [CHORE] Add schemas for remaining local plan ops @colin-ho ([CHORE] Add schemas for remaining local plan ops #3446)
• [CHORE] Put empty table when building probe table @colin-ho ([CHORE] Put empty table when building probe table #3445)
• [CHORE] Explain block_on function in common-runtime @colin-ho ([CHORE] Explain block_on function in common-runtime #3442)
• [CHORE] connect: df.schema @andrewgazelka ([CHORE] connect: df.schema #3353)
• [CHORE] Update execution config to turn on Ray tracing @jaychia ([CHORE] Update execution config to turn on Ray tracing #3431)
• [CHORE] connect test: df.get_attr @andrewgazelka ([CHORE] connect test: df.get_attr #3349)
• [CHORE] Cleanup ExprResolver @kevinzwang ([CHORE] Cleanup ExprResolver #3401)
• [CHORE] connect: add tests for df.take() method @andrewgazelka ([CHORE] connect: add tests for df.take() method #3385)
• [CHORE] Change IOConfig to be serialized into binary instead of JSON @kevinzwang ([CHORE] Change IOConfig to be serialized into binary instead of JSON #3400)
• [CHORE] Pin PyIceberg version to <0.8 @kevinzwang ([CHORE] Pin PyIceberg version to <0.8 #3391)
• [CHORE] Add TPC-H queries in SQL @kevinzwang ([CHORE] Add TPC-H queries in SQL #3392)
• [CHORE] connect: Optimize plans in connect @colin-ho ([CHORE] connect: Optimize plans in connect #3378)
• [CHORE] delete empty file xyz @andrewgazelka ([CHORE] delete empty file xyz #3370)

⬆️ Dependencies

14 changes

• Bump orjson from 3.10.11 to 3.10.12 @dependabot (Bump orjson from 3.10.11 to 3.10.12 #3464)
• Bump grpcio from 1.67.0 to 1.68.1 @dependabot (Bump grpcio from 1.67.0 to 1.68.1 #3465)
• Bump arrow-buffer from 51.0.0 to 53.3.0 @dependabot (Bump arrow-buffer from 51.0.0 to 53.3.0 #3467)
• Bump regex-syntax from 0.7.5 to 0.8.4 @dependabot (Bump regex-syntax from 0.7.5 to 0.8.4 #3468)
• Bump memmap2 from 0.9.4 to 0.9.5 @dependabot (Bump memmap2 from 0.9.4 to 0.9.5 #3470)
• Bump image from 0.25.4 to 0.25.5 @dependabot (Bump image from 0.25.4 to 0.25.5 #3471)
• Bump bytes from 1.7.1 to 1.8.0 @dependabot (Bump bytes from 1.7.1 to 1.8.0 #3411)
• Bump astral-sh/setup-uv from 3 to 4 @dependabot (Bump astral-sh/setup-uv from 3 to 4 #3410)
• Bump serde_json from 1.0.124 to 1.0.133 @dependabot (Bump serde_json from 1.0.124 to 1.0.133 #3413)
• Bump sample-arrow2 from 0.1.0 to 0.17.2 @dependabot (Bump sample-arrow2 from 0.1.0 to 0.17.2 #3414)
• Bump chrono-tz from 0.8.6 to 0.10.0 @dependabot (Bump chrono-tz from 0.8.6 to 0.10.0 #3415)
• Bump azure-storage-blob from 12.17.0 to 12.24.0 @dependabot (Bump azure-storage-blob from 12.17.0 to 12.24.0 #3416)
• Bump opencv-python from 4.8.1.78 to 4.10.0.84 @dependabot (Bump opencv-python from 4.8.1.78 to 4.10.0.84 #3417)
• Bump sqlalchemy from 2.0.25 to 2.0.36 @dependabot (Bump sqlalchemy from 2.0.25 to 2.0.36 #3418)

---

This discussion was created from the release v0.3.15.