Update datafusion by Jeadie · Pull Request #10422 · spiceai/spiceai

Jeadie · 2026-04-20T11:08:25Z

📝 Summary

Update datafusion to include:

Implement ExecutionPlan::fetch and ExecutionPlan::with_fetch for ProjectionExec
#155
Guard ProjectionExec::try_pushdown_sort against stale name/index metadata
#156

For # 156 we also need to fix a planning bug where rrf(...) queries using vector_search(...) with recency settings (time_column, decay args) could produce an invalid physical plan (physical_plan_error: SanityCheckPlan) due to a stale _score@idx sort reference after optimizer rewrites.

github-actions · 2026-04-20T11:08:36Z

✅ Pull with Spice Passed

Passing checks:

✅ Title meets minimum length requirement (10 characters)
✅ No banned labels detected
✅ Has a label from required category kind/
✅ Has a label from required category area/
✅ Has at least one assignee: Jeadie

github-actions · 2026-04-20T11:08:36Z

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

area/config
kind/dependencies
size/m

Passing checks:

✅ Title meets minimum length requirement (10 characters)
✅ No banned labels detected
✅ Has a label from required category kind/
✅ Has a label from required category area/
✅ Has at least one assignee: Jeadie

github-actions · 2026-04-20T11:08:36Z

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

area/config
kind/dependencies
size/m

Passing checks:

✅ Title meets minimum length requirement (10 characters)
✅ No banned labels detected
✅ Has a label from required category kind/
✅ Has a label from required category area/
✅ Has at least one assignee: Jeadie

Copilot

Pull request overview

Updates the pinned spiceai/datafusion git revision to include changes from spiceai/datafusion#155.

Changes:

Bump datafusion (and related datafusion-* crates) git rev in [patch.crates-io] to 2e4b04b4e9dd4b949195529cb0d9b92cb8d75eaf.
Regenerate Cargo.lock entries so all datafusion* packages point at the new git revision.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

File	Description
Cargo.toml	Updates the pinned DataFusion fork revision across all patched `datafusion*` crates.
Cargo.lock	Updates lockfile sources to match the new DataFusion git revision.

fix: Update test snapshots

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 49 out of 207 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 50 out of 209 changed files in this pull request and generated 2 comments.

Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai>

* update datafusion * fix: Update test snapshots * fix: Update test snapshots * fixes for _score in vector UDTF * remove bas snapshots * fix: Update Search integration test snapshots * search: stabilize vector_search score column for rrf recency plans * revert these snapshots * fix: Update Search integration test snapshots * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update snapshot * remove * fix build * fix: remove unused col import The col function from datafusion_expr is no longer used in embeddings/udtf.rs after the upstream datafusion update; the lint job runs with -D warnings, so the unused import broke the build. * snapshot * update DF * Fix DF * fixes * improved testing for RRF * snapshots * normalize remove * fix: Update Search integration test snapshots (#10567) Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai> * test: update insta snapshots --------- Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Evgenii Khramkov <evgenii@spice.ai> Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: claudespice <claude@spice.ai>

* Add self-hosted Spice connector support * Enhance RefreshTask to handle pre-delete rows for upserts and add tests for endpoint scheme validation * Refactor encode_data_update function to use lifetime annotations for better clarity * Refactor data exchange handling to support streaming snapshots and improve batch encoding * fix: enhance upsert handling with primary key validation and improve error reporting * fix: pass dataset name as a reference in upsert pre-delete rows function * Add DuckDB vector engine support * Address DuckDB vector engine PR feedback * Add DuckDB HNSW search integration coverage * Remove DuckDB ef_search alias * Remove DuckDB ef_construction alias * Address DuckDB vector engine PR feedback * Revert "Add self-hosted Spice connector support" This reverts commit 10af21f. * fix merge * better Error enum * Support views on DDL catalogs (spiceai#10554) * Support views on DDL catalogs * fix ref * fix variable * fix compile * fix: invert table_exists loop condition for view dependency wait The view dependency polling loop had an inverted condition: it retried while table_exists() returned true (table found) and broke when it returned false (table not found). This caused all view tests to fail — the loop would spin until the deadline with the table already present, log 'does not exist, retrying...' throughout, then exit on timeout and report the view as failed. Fix: negate the condition so the loop retries while the table is absent (!table_exists) and exits as soon as the table appears. * fix: table_exists takes &TableReference to satisfy clippy::needless_pass_by_value * bad merge * Update datafusion (spiceai#10422) * update datafusion * fix: Update test snapshots * fix: Update test snapshots * fixes for _score in vector UDTF * remove bas snapshots * fix: Update Search integration test snapshots * search: stabilize vector_search score column for rrf recency plans * revert these snapshots * fix: Update Search integration test snapshots * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update snapshot * remove * fix build * fix: remove unused col import The col function from datafusion_expr is no longer used in embeddings/udtf.rs after the upstream datafusion update; the lint job runs with -D warnings, so the unused import broke the build. * snapshot * update DF * Fix DF * fixes * improved testing for RRF * snapshots * normalize remove * fix: Update Search integration test snapshots (spiceai#10567) Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai> * test: update insta snapshots --------- Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Evgenii Khramkov <evgenii@spice.ai> Co-authored-by: Luke Kim <80174+lukekim@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: claudespice <claude@spice.ai> * Improve full-text search indexing performance (spiceai#10464) * Improve tantivy FTS ingest performance * Improve tests * Rollback on error path * remove index as unnecessary --------- Co-authored-by: Jack Eadie <jack@spice.ai> * add to search integration tests * fix compile * update docs * Address DuckDB vector review feedback * fix boxed * Address DuckDB vector follow-up review feedback * Bound DuckDB vector search default limit * fix: Update Search integration test snapshots * fix finding duckdb index * feat: Enhance DuckDB vector query handling for empty projections and filter pushdown * fix: Update Search integration test snapshots * fixes * fixes * better docs * chore: update datafusion-table-providers to add ignored_index_prefixes Pin to fork branch spiceai-hnsw-index-drift which adds `TableDefinition::add_ignored_index_prefix` so externally-managed HNSW indexes (named `__spice_vss_*`) are excluded from the DuckDB writer's index drift check, preventing spurious refresh failures. * chore: update datafusion-table-providers to upstream merged commit Switch from Jeadie fork back to datafusion-contrib upstream at df7dbc64, which includes the merged ignored_index_prefixes fix. * formatting * clippy * fix clippy: return closure result directly instead of let binding --------- Co-authored-by: jeadie <jack@spice.ai> Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Evgenii Khramkov <evgenii@spice.ai> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: claudespice <claude@spice.ai> Co-authored-by: Sergei Grebnov <sergei.grebnov@gmail.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: William <98815791+peasee@users.noreply.github.com>

update datafusion

230780b

Jeadie self-assigned this Apr 20, 2026

Jeadie requested a review from a team as a code owner April 20, 2026 11:08

Copilot AI review requested due to automatic review settings April 20, 2026 11:08

Jeadie added kind/bug Something isn't working area/datafusion labels Apr 20, 2026

github-actions Bot added area/config kind/dependencies Pull requests that update a dependency file size/m labels Apr 20, 2026

Copilot started reviewing on behalf of Jeadie April 20, 2026 11:08 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

sgrebnov previously approved these changes Apr 20, 2026

View reviewed changes

ewgenius previously approved these changes Apr 20, 2026

View reviewed changes

Spice Snapshot Update Bot and others added 2 commits April 20, 2026 06:21

fix: Update test snapshots

b632cf3

Merge pull request #10425 from spiceai/snapshot-update/24667765439

508adb6

fix: Update test snapshots

github-actions Bot dismissed stale reviews from ewgenius and sgrebnov via 508adb6 April 20, 2026 13:28

Spice Snapshot Update Bot and others added 2 commits April 20, 2026 13:47

fix: Update test snapshots

98eef04

Merge pull request #10426 from spiceai/snapshot-update/24667773332

13373ca

fix: Update test snapshots

Copilot AI review requested due to automatic review settings April 20, 2026 13:55

github-actions Bot added area/tests size/l labels Apr 20, 2026

Copilot started reviewing on behalf of ewgenius April 20, 2026 13:56 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

lukekim and others added 2 commits April 20, 2026 13:53

Merge branch 'trunk' into jeadie/26-04-20/df-increment

b6b1df2

fixes for _score in vector UDTF

b15bc0b

Copilot AI review requested due to automatic review settings April 21, 2026 03:52

snapshot

85c1eaa

peasee previously approved these changes Apr 27, 2026

View reviewed changes

Merge branch 'trunk' into jeadie/26-04-20/df-increment

85cd4ed

Copilot AI review requested due to automatic review settings April 27, 2026 03:09

Copilot started reviewing on behalf of Jeadie April 27, 2026 03:10 View session

update DF

6a762a3

Jeadie dismissed peasee’s stale review via 6a762a3 April 27, 2026 03:11

Copilot AI reviewed Apr 27, 2026

View reviewed changes

Jeadie added 2 commits April 27, 2026 13:53

Fix DF

02fad26

fixes

7f99dd1

Copilot AI review requested due to automatic review settings April 27, 2026 08:40

Copilot started reviewing on behalf of Jeadie April 27, 2026 08:41 View session

Copilot AI reviewed Apr 27, 2026

View reviewed changes

Comment thread crates/runtime/src/embeddings/udtf.rs

Comment thread crates/runtime/src/search/rrf.rs

Jeadie added 3 commits April 28, 2026 08:09

improved testing for RRF

2aad027

snapshots

0d4f26d

Merge branch 'trunk' into jeadie/26-04-20/df-increment

004865b

Copilot AI review requested due to automatic review settings April 27, 2026 22:55

Jeadie and others added 2 commits April 28, 2026 10:19

normalize remove

da36dec

fix: Update Search integration test snapshots (#10567)

b6d32cc

Co-authored-by: Spice Snapshot Update Bot <spiceaibot@spice.ai>

peasee previously approved these changes Apr 28, 2026

View reviewed changes

test: update insta snapshots

db8f548

Jeadie dismissed peasee’s stale review via db8f548 April 28, 2026 01:27

peasee approved these changes Apr 28, 2026

View reviewed changes

Jeadie added this pull request to the merge queue Apr 28, 2026

Merged via the queue into trunk with commit f24c163 Apr 28, 2026
66 of 67 checks passed

Jeadie deleted the jeadie/26-04-20/df-increment branch April 28, 2026 04:45

claudespice mentioned this pull request May 1, 2026

fix: preserve field and schema metadata in Vortex type transformation #10628

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update datafusion#10422

Update datafusion#10422
Jeadie merged 39 commits into
trunkfrom
jeadie/26-04-20/df-increment

Jeadie commented Apr 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

Jeadie commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

Uh oh!

github-actions Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Pull with Spice Passed

Passing checks:

Uh oh!

github-actions Bot commented Apr 20, 2026

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

Passing checks:

Uh oh!

github-actions Bot commented Apr 20, 2026

✅ Pull with Spice Passed

🏷️ Auto-applied labels:

Passing checks:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Jeadie commented Apr 20, 2026 •

edited

Loading

github-actions Bot commented Apr 20, 2026 •

edited

Loading