DuckLake QueryPlanner: intercept DML logical plans

[zfarrell]

Context

This is one ticket in a series carrying forward #12 foundation work. Read #12 first for repo context.

DataFusion natively plans INSERT against a TableProvider, but it does not natively plan DELETE, UPDATE, or MERGE. The fork adds a custom QueryPlanner that intercepts DataFusion's logical DML plans for DuckLake tables and routes them to DuckLake-specific execution plans. This is the integration point — every other DML ticket (#17 DELETE, #18 UPDATE, #19 MERGE) depends on it.

Reference branch

ducklake-features/integration:

• src/query_planner.rs — the planner. Read it end-to-end; it's ~600 LOC and not bigger than necessary.
• Inline #[cfg(test)] tests in the same file cover the routing logic (planner rejects unsupported plan shapes with explicit errors instead of silently emptying filters — a deliberate safety choice flagged in the audit).

Scope

1. Port src/query_planner.rs.
2. The planner intercepts only LogicalPlan::Dml(Delete | Update). INSERT continues to use DataFusion's native path via TableProvider::insert_into. MERGE is implemented as a custom logical extension node (see MERGE physical execution (INSERT/UPDATE/DELETE atomic) #19).
3. For DELETE/UPDATE, the planner extracts:
   • the target DuckLakeTable (downcast via TableProvider::as_any)
   • the filter expression (rejecting plans where DataFusion rewrote the filter through joins/subqueries — emit a clear error rather than silently dropping the predicate)
   • for UPDATE, the SET expressions, identified by positional matching against the target schema (see audit note below)
4. Construct the right physical plan: DeleteExec (DELETE physical execution (MOR delete files) #17) or UpdateExec (UPDATE physical execution (MOR delete + insert) #18).
5. Register the planner via SessionStateBuilder::with_query_planner and expose a helper, e.g. DuckLakeQueryPlanner::register(&mut state).

Acceptance criteria

• src/query_planner.rs compiles standalone (mock the exec types if DELETE physical execution (MOR delete files) #17/UPDATE physical execution (MOR delete + insert) #18 are not yet ported)
• Inline tests pass
• Explicit-error behavior preserved: a DELETE/UPDATE plan whose filter has been rewritten through a JOIN or subquery returns a planner error, not an empty filter
• No duckdb crate imports
• Documented integration example in the crate's top-level doc-comment showing how to register the planner

Dependencies

• Blocks: DELETE physical execution (MOR delete files) #17 (DELETE), UPDATE physical execution (MOR delete + insert) #18 (UPDATE), MERGE physical execution (INSERT/UPDATE/DELETE atomic) #19 (MERGE)
• Depends on: Foundation: rebase integration onto upstream, drop pass-throughs, triage SLT failures #12 (foundation)

Out of scope

• The DELETE/UPDATE/MERGE physical execs themselves — separate tickets
• DDL planning (CREATE TABLE etc. continues to flow through DataFusion's native path)

Notes

• Audit concern to address before merging: in the fork's implementation, UPDATE detection uses positional matching: projection_exprs[i].name == schema.fields()[i].name(). This depends on DataFusion never reordering projections. The audit flagged this as fragile but not currently broken. Add a runtime assertion that the names match by index and fail loudly with a planner error if they don't — that way a future DataFusion behavior change becomes a clear bug report rather than silent data corruption.
• The audit verdict on this file was "solid" — the planner's rejection of join/subquery-rewritten filters is called out as a "genuinely thoughtful safety check."

[zfarrell]

Ticket #16 — pushed feat/16-query-planner

Branch: https://github.com/zfarrell/datafusion-ducklake/tree/feat/16-query-planner

Investigation outcome

The foundation port (upstream-port/integration) already shipped a substantially complete src/query_planner.rs — DuckLakeQueryPlanner was wired into the public API and consumed by ~12 integration test files (sql_dml_tests, cross_engine_dml_tests, merge_tests, edge_case_tests, deep_edge_case_tests, etc.). The DML execs (DuckLakeDeleteExec, DuckLakeUpdateExec, DuckLakeMergeExec) were already constructible and reachable from the planner via DuckLakeTable::{delete, update, merge}. The foundation comment's "DML wiring deferred" warning was conservative — wiring was in place; what was missing was the polish called out in #16's audit notes.

What this PR adds

Concern	Before	After
Registration helper	only manual with_query_planner(Arc::new(...))	DuckLakeQueryPlanner::register_into(builder) -> SessionStateBuilder + new_arc()
Top-level doc example	only read-only example	dedicated DML example showing register_into
UPDATE projection drift	silent tracing::warn! if lengths differ, silent break if extra	hard DataFusionError::Plan with diagnostic if length mismatches or if the projection alias at index i does not equal schema.fields()[i].name()
Subquery DELETE rejection	already returned a planner error	now also covered by an end-to-end test that asserts data is untouched
Physical-plan-shape test	none	three new tests in tests/sql_dml_tests.rs

Acceptance criteria

• src/query_planner.rs exposes public DuckLakeQueryPlanner with register_into(builder) (and new_arc())
• Crate top-level rustdoc has a working register_into code example (src/lib.rs lines 38-69)
• Planner intercepts DELETE / UPDATE on DuckLakeTable targets and dispatches to DuckLakeTable::{delete,update}
• Planner returns an explicit error for filters rewritten through joins/subqueries — test_planner_rejects_delete_with_subquery verifies the error and that the underlying data is unchanged (no silent full-table delete)
• UPDATE planner asserts projection-schema alignment by name+index, fails loudly on mismatch
• cargo build / cargo build --all-features clean
• cargo clippy clean for my changes (pre-existing clippy 0.1.92 lints on the rest of the lib are out of scope — same set fails on upstream-port/integration)
• End-to-end planning tests: test_planner_delete_produces_delete_exec_node, test_planner_update_produces_update_exec_node, test_planner_rejects_delete_with_subquery
• No regression vs foundation baseline. Lib: 426 → 426 pass. tests/sql_dml_tests: 10 → 13 pass. Total integration delta: +3 pass, 0 new failures, 0 changed ignores (verified with cargo test --test '*' --all-features --no-fail-fast on both branches).

Commits

567349e test(planner): end-to-end planning tests for DELETE and UPDATE
a3c1875 feat(planner): assert UPDATE projection alignment by name and index
44064b6 feat(planner): expose DuckLakeQueryPlanner registration helper
64fae5d docs: document DuckLakeQueryPlanner registration in crate top-level rustdoc

Note for downstream tickets (#17 DELETE, #18 UPDATE, #19 MERGE)

The planner contract the execs receive is unchanged from the foundation port:

• DELETE (#17): DuckLakeTable::delete(state, &[Expr]) — Exprs are unnormalized (qualifier-stripped) column refs; predicate is split by split_conjunction. The planner guarantees the filter input is Filter(pred, TableScan) or bare TableScan (no-WHERE delete); any other shape produces a planner-level error before the exec is constructed.
• UPDATE (#18): DuckLakeTable::update(state, Vec<UpdateAssignment>, &[Expr]) — UpdateAssignment.column_index is now guaranteed to be a valid index into table.schema().fields() because the planner enforces projection_exprs.len() == schema.fields().len() and that the alias at index i matches field.name(). The expr is unnormalized.
• MERGE (#19): unchanged — MERGE reaches DuckLakeMergeExec through its custom extension node, not via this planner (see ticket comment in src/query_planner.rs).

Execs may rely on the new UPDATE invariant: if extract_update_info returned Ok, every column_index in the returned Vec<UpdateAssignment> is < schema.fields().len() and corresponds to the correct target column.