Skip to content

Commit d24d6b3

Browse files
chore(deps): bump iceberg-rust (parallel file scanning) + standardize fork branch naming (spiceai#11328)
* chore(deps): bump iceberg-rust to parallel file scanning fork Bumps the spiceai/iceberg-rust pin from e519b221 to b652de2e, picking up the eager task-bucketing parallel-scan change (spiceai/iceberg-rust#42, port of apache/iceberg-rust#2298). IcebergTableProvider::scan() now plans files eagerly and distributes FileScanTasks across min(target_partitions, n_files) DataFusion partitions, so Iceberg file reads are scheduled concurrently instead of through a single partition. Identity-partitioned tables additionally declare Partitioning::Hash so downstream joins/aggregates can skip a RepartitionExec. No Spice code changes are required: Spice consumes the provider only via IcebergTableProvider::try_new, and the change is internal to the scan path. The fork preserves Spice's existing iceberg limit-pushdown. * chore(deps): adopt spiceai-<iceberg>-df-<df> fork naming; re-pin to spiceai-0.9.1-df-53 Re-pins iceberg-rust to the renamed `spiceai-0.9.1-df-53` branch (rev 9e6e1a00) and bumps the version requirement to `=0.9.1` to match the corrected crate version. Adds docs/dev/fork-branch-naming.md documenting the `spiceai-<iceberg>-df-<datafusion>` convention. The DF53 fork line was previously named `spiceai-0.9.0` but actually tracks a post-0.9.1 `main` snapshot; the rename + version correction make the branch name reflect its real contents. * chore(deps): re-pin iceberg-rust to merged spiceai-0.9.1-df-53 commit Fork PR spiceai/iceberg-rust#43 merged; re-pin from the branch head (9e6e1a00) to the merge commit b8fc79d9 on spiceai-0.9.1-df-53. * chore(deps): annotate iceberg pins with fork branch; drop naming doc Address Copilot review on spiceai#11328: add `# branch: spiceai-0.9.1-df-53` to the iceberg-rust pins (matching the convention used by delta_kernel/duckdb/arrow). Remove docs/dev/fork-branch-naming.md per review feedback (no separate doc).
1 parent 79f8cb6 commit d24d6b3

2 files changed

Lines changed: 39 additions & 39 deletions

File tree

Cargo.lock

Lines changed: 27 additions & 27 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -210,12 +210,12 @@ hf-hub = { version = "0.4", features = ["tokio"] }
210210
http = "1.3.1"
211211
httpdate = "1.0"
212212
humantime = "2.1.0"
213-
iceberg = "=0.9.0"
214-
iceberg-catalog-glue = "=0.9.0"
215-
iceberg-catalog-rest = "=0.9.0"
216-
iceberg-datafusion = "=0.9.0"
217-
iceberg-storage-opendal = { version = "=0.9.0", features = ["opendal-s3", "opendal-gcs", "opendal-fs"] }
218-
iceberg_test_utils = "=0.9.0"
213+
iceberg = "=0.9.1"
214+
iceberg-catalog-glue = "=0.9.1"
215+
iceberg-catalog-rest = "=0.9.1"
216+
iceberg-datafusion = "=0.9.1"
217+
iceberg-storage-opendal = { version = "=0.9.1", features = ["opendal-s3", "opendal-gcs", "opendal-fs"] }
218+
iceberg_test_utils = "=0.9.1"
219219
imap = { git = "https://github.com/jonhoo/rust-imap", rev = "6fe22ed11a1ccffe1799a73f48e589366d8d100f" }
220220
indexmap = "2"
221221
insta = { version = "1.46.3", features = ["filters"] }
@@ -455,12 +455,12 @@ tokio-rusqlite = { git = "https://github.com/spiceai/tokio-rusqlite.git", rev =
455455
# Tracking Issue: https://github.com/allan2/dotenvy/issues/113
456456
dotenvy = { git = "https://github.com/spiceai/dotenvy.git", rev = "e5cef1871b08003198949dfe2da988633eaad78f" }
457457

458-
iceberg = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "e519b22193e44667dde7d461e3cac1798fa0c9e2" }
459-
iceberg-catalog-glue = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "e519b22193e44667dde7d461e3cac1798fa0c9e2" }
460-
iceberg-catalog-rest = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "e519b22193e44667dde7d461e3cac1798fa0c9e2" }
461-
iceberg-datafusion = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "e519b22193e44667dde7d461e3cac1798fa0c9e2" }
462-
iceberg-storage-opendal = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "e519b22193e44667dde7d461e3cac1798fa0c9e2" }
463-
iceberg_test_utils = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "e519b22193e44667dde7d461e3cac1798fa0c9e2" }
458+
iceberg = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "b8fc79d9717d8576a45e2bd49aee52967b763f1d" } # branch: spiceai-0.9.1-df-53
459+
iceberg-catalog-glue = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "b8fc79d9717d8576a45e2bd49aee52967b763f1d" } # branch: spiceai-0.9.1-df-53
460+
iceberg-catalog-rest = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "b8fc79d9717d8576a45e2bd49aee52967b763f1d" } # branch: spiceai-0.9.1-df-53
461+
iceberg-datafusion = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "b8fc79d9717d8576a45e2bd49aee52967b763f1d" } # branch: spiceai-0.9.1-df-53
462+
iceberg-storage-opendal = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "b8fc79d9717d8576a45e2bd49aee52967b763f1d" } # branch: spiceai-0.9.1-df-53
463+
iceberg_test_utils = { git = "https://github.com/spiceai/iceberg-rust.git", rev = "b8fc79d9717d8576a45e2bd49aee52967b763f1d" } # branch: spiceai-0.9.1-df-53
464464

465465
# candle-* crates are already redirected to the spiceai/candle fork via the
466466
# [patch.crates-io] block higher up in this file. Keeping those here would be

0 commit comments

Comments
 (0)