v0.7.21

What's Changed

🐛 Bug Fixes

• fix(ai): keep UDF options out of OpenAI requests by @FANNG1 in #7277

📖 Documentation

• docs(benchmark): add hand-tracking workload A/B benchmark for lerobot batched decode by @ykdojo in #7267
• docs: add physical AI examples (hand tracking + reward scoring) by @ykdojo in #7271

Full Changelog: v0.7.20...v0.7.21

v0.7.20

What's Changed

✨ Features

• feat(iceberg): push down starts_with and is_nan/not_nan filters by @jackylee-ch in #7266

🐛 Bug Fixes

• fix(dashboard): support OUT_DIR on a different filesystem from the source tree by @alejandro-soto-franco in #7246
• fix(daft-file): release GIL in PyDaftFile methods to prevent deadlock by @zhouwenjiajia in #7257
• fix: fd and thread leak from tqdm progress bar in Jupyter by @ykdojo in #7262

♻️ Refactor

• refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions by @BABTUNA in #7221

🔧 Maintenance

• chore(deps): lower opt-level for opendal-service-oss release builds by @alejandro-soto-franco in #7249
• chore(config): deprecate min_cpu_per_task by @XiaoHongbo-Hope in #7125

New Contributors

• @alejandro-soto-franco made their first contribution in #7249
• @zhouwenjiajia made their first contribution in #7257

Full Changelog: v0.7.19...v0.7.20

v0.7.19

What's Changed

✨ Features

• feat: add mcap pip extra and dependency checks for read_mcap by @everettVT in #7220
• feat(writers): add single_file kwarg to write_parquet by @aaron-ang in #7004

🐛 Bug Fixes

• fix: normalize HF bucket tree paths by @everettVT in #7218
• fix(session): allow set_provider to accept Provider instance for auto… by @DieselZhang in #7041
• fix(telemetry): prevent interpreter-shutdown segfault from Scarf telemetry by @FANNG1 in #7203
• fix(io): route oss:// through S3-compatible storage options for write_lance by @AnayGarodia in #7228
• fix: Reuse file format config in Iceberg and Delta by @srilman in #7163

📖 Documentation

• docs: add GooseFS connector documentation by @XuQianJin-Stars in #7193
• docs: Clean up formatting in Common Crawl docs by @srilman in #7237
• docs(checkpoint): add user guide and connector cross-references by @rohitkulshreshtha in #6985

👷 CI

• ci: Include New Contributers in Automated Draft Releases by @srilman in #7234

New Contributors

• @DieselZhang made their first contribution in #7041
• @FANNG1 made their first contribution in #7203
• @AnayGarodia made their first contribution in #7228

Full Changelog: v0.7.18...v0.7.19

v0.7.18

What's Changed 🚀

✨ Features

• feat: Remove dynamodb dependency for concurrent writes in deltalake on s3 @arpatilmh (#7187)
• feat(lerobot): batch video-frame decode by shard @ykdojo (#7184)
• feat(io): export GooseFSConfig from daft.io @XuQianJin-Stars (#7194)

🚀 Performance

• perf(inline-agg): pack two-string-column keys into u64 for typed FNV grouping @BABTUNA (#6924)

Full Changelog: v0.7.17...v0.7.18

v0.7.17

What's Changed 🚀

✨ Features

• feat: streamline HDF5 file access @everettVT (#7160)
• feat: Use HF Bucket as as alternative for Common Crawl @lhoestq (#7103)
• feat: Support Xet reads for HuggingFace IO @srilman (#7183)
• feat(inline-agg): add AnyValue accumulator @BABTUNA (#7036)
• feat(io): add GooseFS support via OpenDAL services-goosefs @XuQianJin-Stars (#7109)
• feat(ai): support text file inputs in transformers prompter @YuangGao (#7174)
• feat: asof_join_aligned distributed @euanlimzx (#7177)
• feat: add HDF5 file support @everettVT (#7159)
• feat(ai): add transformers provider for prompt() @YuangGao (#7152)
• feat(lerobot): Add daft.datasets.lerobot for working with LeRobot v3 datasets @srilman (#7090)
• feat(session): resolve catalog-qualified identifiers in namespace methods @Lucas61000 (#7171)

🐛 Bug Fixes

• fix: add pillow to video extra @everettVT (#7213)
• fix: Format nested Rust errors as a simplified backtrace in Python @srilman (#7178)
• fix(iceberg): pass ignore_corrupt_files through table reads @jackylee-ch (#7147)
• fix(io): default https scheme for scheme-less S3 endpoint_url @plusplusjiajia (#7111)

♻️ Refactor

• refactor(inline-agg): consolidate Sum/Product/Min/Max into one macro @BABTUNA (#7151)

🔧 Maintenance

• chore(deps): bump the all group with 4 updates @dependabot[bot] (#7190)
• chore: Rename GoosefsConfig to GooseFSConfig @srilman (#7192)

⬆️ Dependencies

• chore(deps): bump the all group with 4 updates @dependabot[bot] (#7190)

Full Changelog: v0.7.16...v0.7.17

v0.7.15

What's Changed 🚀

✨ Features

• feat: add try_cast function for safe type conversion @XuQianJin-Stars (#6960)
• feat(shuffle): default flight shuffle compression to lz4 @colin-ho (#7071)
• feat(iceberg): support branch and tag reads @jackylee-ch (#7042)
• feat: thread assume_sorted_and_aligned_partitions parameter through ASOF join @euanlimzx (#7067)
• feat(checkpoint): distributed observability counters @rohitkulshreshtha (#7026)
• feat(functions): add UUIDv7 timestamp-extraction partition transforms @jaychia (#7032)
• feat: tests for range repartition hints in clustering spec @euanlimzx (#7044)
• feat: allow custom datasources to specify clustering spec @jaychia (#7031)
• feat(iceberg): accept PathLike metadata paths in read_iceberg @jackylee-ch (#7040)
• feat(ext): native extension support on ray @rchowell (#6996)
• feat: support PostgreSQL as a structured data source via Gravitino @qingfeng-occ (#6989)
• feat: Better extension display @universalmind303 (#6802)
• feat(gcs): implement delete for GCS object store @daiping8 (#6958)
• feat(inline-agg): symbolize string group keys in multi-column grouped aggregation @BABTUNA (#6748)
• feat: first_value / last_value aggs for window functions @euanlimzx (#6974)
• feat(iceberg): auto-configure IOConfig for Alibaba Cloud OSS @plusplusjiajia (#6993)
• feat: nearest asof joins @euanlimzx (#6953)
• feat(functions): add sample_interval_seconds to video_frames @TheR1sing3un (#6832)
• feat: wrappers for using Daft's FileIO / open from top-level ops @srilman (#6986)
• feat: configurable flight shuffle compression @colin-ho (#6979)
• feat(checkpoint): idempotent write_deltalake + cross-sink helpers @rohitkulshreshtha (#6932)

🐛 Bug Fixes

• fix(asof-join): use unknown clustering spec instead of hash @euanlimzx (#7075)
• fix(ray): configure worker startup timeout on runner @colin-ho (#7055)
• fix(shuffle): concat recordbatches before repartition @colin-ho (#7064)
• fix(flight-shuffle): reduce coordinator memory to O(map_tasks + partitions) @colin-ho (#7056)
• fix(planner): propagate clustering through bound projections @jaychia (#7033)
• fix(filesystem): fix pyarrow fs memory by caching by value, not identity @rchowell (#7025)
• fix: strip trailing slash from resolved write/read paths @colin-ho (#6987)
• fix(integration): ray data deprecated api fix @kyuds (#6871)
• fix(io): use checked_shl and saturating_mul in retry jitter calculation @ARDA7787 (#6949)
• fix: cast generic file arrays to media-specific types @everettVT (#6848)
• fix(daft-distributed): remove unused hash_map_macro feature gate @mikedep333 (#6981)
• fix: gate Self import on Python 3.11+ @everettVT (#6971)

🚀 Performance

• perf: update jemalloc 5.3.0 → 5.3.1 to fix muzzy decay performance bug @everettVT (#7059)
• perf(window): reduce unnecessary copies of data in finalize() step of window functions to reduce memory usage @euanlimzx (#7006)
• perf(window): return Series from window ops to reduce intermediate RecordBatch copies @euanlimzx (#7011)
• perf(flotilla): batch flight partition refs per finalize @cckellogg (#7008)
• perf: Specialized Code Path for Hash Join on Ints @srilman (#6644)
• perf: enable per-column compression configuration @rchowell (#6884)

♻️ Refactor

• refactor(checkpoint): drop "seal" vocabulary from Rust API surface @rohitkulshreshtha (#7078)
• refactor(file): rename File byte-range fields to position/size @aaron-ang (#6747)
• refactor(distributed): rename needs_hash_repartition to can_skip_hash_repartition @euanlimzx (#7053)
• refactor(checkpoint): bincode-encoded versioned FileMetadata blobs @rohitkulshreshtha (#7001)

📖 Documentation

• docs: standardize Slack links to use daft.ai/slack @everettVT (#7066)
• docs: reorder UDF API docs to prioritize new API and deprecate legacy @everettVT (#7028)
• docs(paimon): clarify object-store IO config usage @jackylee-ch (#7029)
• docs: add shuffle algorithms tuning guide @colin-ho (#7017)
• docs(lance): fix daft-lance repository links @jackylee-ch (#7016)

✅ Tests

• test(lance): use daft-lance APIs directly @jackylee-ch (#7010)
• test(parquet): cover embedding and image roundtrips @jackylee-ch (#7021)
• test(iceberg): cover partition field transform conversion @jackylee-ch (#7024)
• test(lance): import test helpers from daft-lance @jackylee-ch (#7015)
• test(checkpoint): optional real-S3 routing for checkpoint test suites @rohitkulshreshtha (#6934)

👷 CI

• ci: catch undeclared runtime imports via TC strict + py version matrix @colin-ho (#6972)

🔧 Maintenance

• chore(deps): bump the all group across 1 directory with 59 updates @dependabot[bot] (#7000)
• chore: bump minimum PyArrow version to 16 @0xdeadd (#6868)

⬆️ Dependencies

• chore(deps): bump the all group across 1 directory with 59 updates @dependabot[bot] (#7000)

Full Changelog: v0.7.14...v0.7.15

v0.7.14

What's Changed 🚀

✨ Features

• feat(flotilla): hint users to switch to flight_shuffle on large shuffles @colin-ho (#6962)
• feat: add uuidv7 generation @everettVT (#6909)
• feat(flotilla): Distributed Limit Counter @colin-ho (#6942)
• feat: add JSON functions json_array_length, json_object_keys, json_tuple @XuQianJin-Stars (#6945)
• feat: add conv function for PySpark parity @YuangGao (#6910)
• feat: ASOF join benchmarking scripts @euanlimzx (#6940)
• feat(show): env defaults and auto alignment for preview output @BABTUNA (#6856)
• feat(temporal): add Spark-style temporal aliases @BABTUNA (#6830)

🐛 Bug Fixes

• fix: hash(-0.0) == hash(0.0) @rchowell (#6963)
• fix: short-circuit on empty micropartitions @rchowell (#6956)

🚀 Performance

• perf(parquet): rewrite reader with arrow-rs public decoder API @colin-ho (#6952)
• perf(flotilla): Coalesce task outputs based on byte threshold @colin-ho (#6943)
• perf(shuffle): Write one shuffle file per task instead of N partition files @colin-ho (#6948)

📖 Documentation

• docs: Update API references and fix code examples @colin-ho (#6938)
• docs(examples): migrate text-embeddings example to public bucket @colin-ho (#6941)

🔧 Maintenance

• chore: migrate daft.io.lance to daft_lance @rchowell (#6957)
• chore(deps): bump the all group across 1 directory with 4 updates @dependabot[bot] (#6863)

⬆️ Dependencies

• chore(deps): bump the all group across 1 directory with 4 updates @dependabot[bot] (#6863)

Full Changelog: v0.7.13...v0.7.14

v0.7.13

What's Changed 🚀

✨ Features

• feat: add forward ASOF joins @euanlimzx (#6918)

🐛 Bug Fixes

• fix: remove hallucinated lance-namespace code @rchowell (#6939)
• fix(sql): omit connection URL from read_sql error messages and __repr__ @madvart (#6933)
• fix(dashbaord): highlight failed operators for queries @cckellogg (#6930)
• fix: handle broadcast of scalars when calling to_struct @euanlimzx (#6931)

Full Changelog: v0.7.12...v0.7.13

v0.7.12

What's Changed 🚀

✨ Features

• feat(iceberg): honor table-level write.target-file-size-bytes property @YuangGao (#6912)
• feat(temporal): add Spark-style add_months and months_between @BABTUNA (#6913)

🐛 Bug Fixes

• fix: revert "feat(daft-ext): scalar daft_func macro with overloading (#6844)" @rchowell (#6925)

📖 Documentation

• docs: Adds daft-qdrant community extension to docs.daft.ai @Anush008 (#6921)

✅ Tests

• test: re-enable Iceberg map-type round-trip test @YuangGao (#6911)

Full Changelog: v0.7.11...v0.7.12

v0.7.11

What's Changed 🚀

✨ Features

• feat: Arrow PyCapsule Interface @aaron-ang (#6745)
• feat: add Python UDAF support (@daft.udaf) @chenghuichen (#6790)
• feat(dashboard): per-task rows/bytes stats with task-topology markers @samstokes (#6861)
• feat(iceberg): idempotent commits via daft.idempotence-key snapshot marker @rohitkulshreshtha (#6905)
• feat(lance): partitioned reads/writes @universalmind303 (#6898)
• feat(dashboard): make header smaller in query details @cckellogg (#6900)
• feat: streaming ASOF joins @euanlimzx (#6846)
• feat: Add map_keys function to extract keys from Map type columns @qingfeng-occ (#6875)
• feat: add event log config api @cckellogg (#6894)
• feat(dashboard): surface query failure details in query view @cckellogg (#6897)
• feat(dashboard): bound query state retention (DF-1970) @samstokes (#6896)
• feat(dashboard): per-task progress updates from flotilla workers @samstokes (#6838)
• feat(dashboard): plumb TaskScheduled events to distinguish pending vs running @samstokes (#6866)
• feat(dashboard): show task sources in tasks sidebar @samstokes (#6879)
• feat(ci): use larger runner for pr testing @cckellogg (#6890)
• feat(native): emit TaskStart events from native executor @cckellogg (#6840)
• feat(session): add non-materialized DataFrame temp views @BABTUNA (#6786)
• feat(dtype): add Float16 (half-precision) support @universalmind303 (#6859)
• feat: extension datatype casting @universalmind303 (#6803)
• feat(file): add Image media type to File @chenghuichen (#6847)
• feat(iceberg): idempotent snapshot commits via snapshot_properties @rohitkulshreshtha (#6841)
• feat(daft-ext): scalar daft_func macro with overloading @universalmind303 (#6844)
• feat(checkpoint): wire checkpoint= into csv, json, warc, iceberg, hudi, lance @rohitkulshreshtha (#6855)
• feat(checkpoint): bundle source config into CheckpointConfig with tunable filter knobs @rohitkulshreshtha (#6853)
• feat(checkpoint): decouple sealed-keys column name from source schema @rohitkulshreshtha (#6850)
• feat: use scan task metadata to parallelize initial limit reads @samstokes (#6763)
• feat(checkpoint): track in-flight writes and seal empty manifests @rohitkulshreshtha (#6835)
• feat(file): add byte-range read support to FileReference @chenghuichen (#6842)
• feat(list): add list_flatten expression @mytsvy (#6812)
• feat(extensions): add proc macro for making writing extension functio… @universalmind303 (#6837)

🐛 Bug Fixes

• fix: Handle dtype mismatch error in join_asof join keys @euanlimzx (#6904)
• fix(distributed): emit empty downstream task for limit(0) @madvart (#6916)
• fix(sql): redact password in SQL connection errors and repr @madvart (#6902)
• fix(dashboard): split task groups by local-plan shape @samstokes (#6899)
• fix(stats): close race in RuntimeStatsManager shutdown @samstokes (#6891)
• fix(writers): strip partition columns from Hive-partitioned file bodies @aaron-ang (#6770)
• fix(distributed): fire OperatorStart/End events per-node on Flotilla @samstokes (#6860)
• fix(test): handle transient iceberg rest catalog startup failures @cckellogg (#6880)
• fix(dashboard): handle offline builds in frontend @samstokes (#6867)
• fix: infer audio file return dtype @everettVT (#6849)

🚀 Performance

• perf(file): add buffer_size to File.open() to reduce wasteful pre-reads @chenghuichen (#6876)
• perf: iceberg predicate pushdowns @rchowell (#6815)

♻️ Refactor

• refactor(distributed): decouple shuffle cleanup from Ray @chenghuichen (#6809)
• refactor(dashboard): remove result preview tab and ResultOut event @samstokes (#6878)

📖 Documentation

• docs(dashboard): add user guide for Daft Dashboard @samstokes (#6882)
• docs: change affiliation for maintainer @gweaverbiodev (#6851)

✅ Tests

• test: restore fixed-size list parquet roundtrip @TuodiAunty (#6895)
• test(iceberg): make predicate-pushdown limit test order-deterministic @samstokes (#6843)

👷 CI

• ci(issues): auto-label issues from component selections @BABTUNA (#6820)
• ci: fix dependency check on external prs @srilman (#6737)

🔧 Maintenance

• chore: Revert "feat(lance): partitioned reads/writes" @universalmind303 (#6907)

👋 New Contributors

• @mytsvy made their first contribution in #6812
• @TuodiAunty made their first contribution in #6895

Full Changelog: v0.7.10...v0.7.11