[META] Carry-forward of fork integration work — tracking issue

[zfarrell]

Tracking: Carry-forward of fork integration work

Background

A few months ago an agent team did substantial work on this fork to make it "1.0" feature-compliant with the DuckLake spec. The work landed on ducklake-features/integration (265 commits, +23k net src LOC, +14 new src files) but was never validated, reviewed, or upstreamed.

In 2026-05 we audited it across five dimensions: build correctness, test quality, code quality, feature inventory, and upstream overlap. Verdict: real, substantive engineering — not AI slop — but cannot be merged as-is. It branched from upstream commit 59eb3da (PR datafusion-contrib#79) and upstream has since landed datafusion 53 / arrow 58, TableProvider::statistics(), and a competing virtual-column design.

This issue tracks the decomposition of that work into focused upstreamable PRs.

Workstreams

Each child ticket is self-contained for an agent with no prior context. Read #12 first; every other ticket references back to it.

#	Ticket	Status	Depends on
#12	Foundation: rebase + cleanup + SLT triage	open	—
#13	Metadata writer dialect/macro layer	open	#12
#14	Postgres MetadataWriter	open	#12 #13
#15	MySQL MetadataWriter	open	#12 #13
#16	DuckLake QueryPlanner	open	#12
#17	DELETE physical execution	open	#12 #16
#18	UPDATE physical execution	open	#12 #16 #17
#19	MERGE physical execution	open	#12 #16 #17 #18
#20	ALTER/DROP/CREATE schema evolution	open	#12 #13
#21	CDC table functions (+ cdc_common bug fix)	open	#12
#22	Virtual columns reconciliation (DESIGN first)	open	#12
#23	Type system & inlined-data parsing	open	#12
#24	R10/R11 hardening safety net	open	#12
#25	Partitioned INSERT + write-path expansion	open	#12

Explicitly dropped from carry-forward

• src/compaction_functions.rs — DuckDB pass-through (shells out to INSTALL ducklake; LOAD ducklake; ATTACH ...; CALL ducklake_merge_adjacent_files();). Per project direction, no feature should depend on libduckdb at runtime for its actual logic. Compaction will be reintroduced later as a native Rust implementation.
• PRs chore: archive untracked artifacts before server teardown #2, feat: preserve early virtual-columns/backend writer thread #4–fix: preserve type roundtrip precision corrections #8, fix: preserve type-string and delete-file validation fixes #10, fix: preserve review-52 delete-file error handling follow-up #11 on this fork — either archival (chore: archive untracked artifacts before server teardown #2) or fully subsumed by ducklake-features/integration (others).
• The 56-file docs/ directory on the integration branch — agent-process artifacts (R1–R11 review cycles, retrospectives). Move to a separate archive branch or .audit/ after rebase; do not include in upstream PRs.

Side pickups worth doing

• docs/gap-analysis.md from PR docs: add phase 1/2 analysis artifacts #3 (859 lines) — only doc unique to that branch. Decide if it's still relevant post-rebase and either port to .audit/ or drop.
• assert!(record_count >= 0, ...) guard in DataFileInfo::new from PR fix: preserve record_count validation + test stability updates #9 — 15-line cherry-pick. Trivial; fold into Foundation: rebase integration onto upstream, drop pass-throughs, triage SLT failures #12 if convenient.

Suggested execution order

The dependency graph above suggests this rough order of attack:

1. Foundation: rebase integration onto upstream, drop pass-throughs, triage SLT failures #12 (foundation) — must complete before anything else
2. Metadata writer dialect/macro layer (prereq for PG + MySQL writers) #13 (dialect/macro layer) — prerequisite for Postgres MetadataWriter implementation #14, MySQL MetadataWriter implementation #15, ALTER/DROP/CREATE schema evolution DDL #20
3. DuckLake QueryPlanner: intercept DML logical plans #16 (planner) — prerequisite for DELETE physical execution (MOR delete files) #17, UPDATE physical execution (MOR delete + insert) #18, MERGE physical execution (INSERT/UPDATE/DELETE atomic) #19
4. Parallel: Postgres MetadataWriter implementation #14, MySQL MetadataWriter implementation #15, ALTER/DROP/CREATE schema evolution DDL #20 (metadata work), DELETE physical execution (MOR delete files) #17 (DELETE), CDC table functions (with cdc_common duplicate-column bug fix) #21 (CDC), Type system & inlined-data parsing (merge with upstream overlap) #23 (types), R10/R11 hardening fixes batch (correctness + atomicity safety net) #24 (hardening), Partitioned INSERT + write-path expansion (stats, footer-size, partition layout) #25 (INSERT expansion)
5. UPDATE physical execution (MOR delete + insert) #18 (UPDATE) after DELETE physical execution (MOR delete files) #17
6. MERGE physical execution (INSERT/UPDATE/DELETE atomic) #19 (MERGE) after DELETE physical execution (MOR delete files) #17, UPDATE physical execution (MOR delete + insert) #18
7. Reconcile virtual_column_exec with upstream row_id (DESIGN required first) #22 (virtual columns) — design first, then implement; can run in parallel but ship last because of the upstream reconciliation requirement

Audit summary

• Build: clean (1 dead-code warning)
• Tests: 827 pass / 97 fail / 0 ignored on the integration branch; SLT 180/276 self-acknowledged
• Code quality: 10/11 substantial files audited "solid", 1 "mixed" (merge_exec hash key collation — documented, not a bug), 1 real bug (cdc_common.rs:107-114 duplicate-column projection — fix tracked in CDC table functions (with cdc_common duplicate-column bug fix) #21)
• Test quality: zero #[ignore] outside external-service tests, zero TODO/FIXME, exact-value assertions throughout
• Real conflict zones with upstream: 3 (virtual columns / metadata_provider schema / types.rs partial overlap)
• ~30 empty test scaffold files exist on the branch — cleanup in Foundation: rebase integration onto upstream, drop pass-throughs, triage SLT failures #12

Full audit raw findings: available on request, summarized in the foundation ticket.