Use DataFusion Parquet virtual row_number for DuckLake row positions

[anoop-narang]

DataFusion PR apache/datafusion#22026 has merged reader-level Parquet virtual-column plumbing, including support for the Arrow/parquet RowNumber virtual column via TableSchema / ParquetOpener. This is the upstream mechanism DuckLake should eventually use for physical row positions.

Update (target release): apache/datafusion#22026 did not ship in DataFusion 54.0.0 — it landed after the 54 release branch was cut. It is now slated for DataFusion 55.0.0 (~Jun/Jul 2026), where it is tracked in the 55.0.0 release issue apache/datafusion#22393. This work is blocked until 55.0.0 is published to crates.io. (Latest published datafusion is 54.0.0, released 2026-06-08, which does not include it.)

Current state:

• This project currently depends on DataFusion 53.x, which does not include feat: Plumb Parquet virtual columns (row_number) through TableSchema and ParquetOpener apache/datafusion#22026.
• The local correctness fix synthesizes __ducklake_row_pos above a constrained Parquet scan using row-group-aligned partitions and disables unsafe reader pushdowns on positional paths.
• That local approach is correct, but it gives up some Parquet pruning/reordering optimizations for rowid / positional-delete paths.

Desired future state:

• After upgrading to a DataFusion release that includes feat: Plumb Parquet virtual columns (row_number) through TableSchema and ParquetOpener apache/datafusion#22026 (i.e. 55.0.0), request a Parquet virtual row-number column from the scan and use it as DuckLake's internal physical row position.
• Feed that column into positional delete filtering and synthetic rowid = row_id_start + physical_row_position.
• Drop the internal position column before user-visible output, preserving zero-column row counts.
• Keep the current fast path for scans that do not need rowid or positional deletes.

Why this matters:

• DuckLake positional deletes and row lineage require true physical row positions in the Parquet file.
• Reader-level row numbers should allow DataFusion/Parquet to preserve more pruning while still producing correct physical positions.

References:

• feat: Plumb Parquet virtual columns (row_number) through TableSchema and ParquetOpener apache/datafusion#22026: reader-level Parquet virtual column plumbing, merged May 28, 2026 (targeted for 55.0.0)
• Release DataFusion 55.0.0 (Jul / Aug 2026) apache/datafusion#22393: DataFusion 55.0.0 release tracking issue
• [EPIC] A collection of support for metadata columns in ListingTable apache/datafusion#20135: DataFusion metadata / virtual-column epic
• Add file_row_index UDF to query file-level row indexes from Parquet files apache/datafusion#22604: user-facing file_row_index() UDF work, separate from the lower-level API

Acceptance criteria:

• Upgrade to a DataFusion version that includes feat: Plumb Parquet virtual columns (row_number) through TableSchema and ParquetOpener apache/datafusion#22026 (55.0.0 or later).
• Replace or bypass FileRowNumberExec on positional paths with the Parquet virtual RowNumber column when available.
• Keep correctness tests for split files, positional deletes, filters, limits, and COUNT(*) passing.
• Verify that non-positional reads remain on the existing fast path.

[anoop-narang]

Heads-up for the eventual migration: PR #200 (fix for #178) adds a new FileRowNumberExec / PositionalFileSource call site — build_data_file_scan() in src/table_deletions.rs. The deletions feed now matches delete-file positions against the true physical position column, so it must be included when positional paths are switched to the DataFusion 55 virtual RowNumber column.

Notes for that migration:

• The position-matching logic in table_deletions.rs keys purely off the position column (no arrival-order assumptions), so the swap should be localized to build_data_file_scan().
• DeletedRowsExec was made single-partition with no DataFusion children as part of the ducklake_table_deletions: DeletedRowsExec is unsound under file-scan repartitioning #178 fix. Once positions come from the reader-level virtual column (correct under any split/pruning), that constraint can be relaxed to restore intra-file parallelism for the feed — worth folding into this issue's acceptance criteria.
• Regression tests to keep green during the swap: tests/table_deletions_repartition_tests.rs (added in fix(cdc): make DeletedRowsExec sound under scan repartitioning (#178) #200) alongside the existing tests/rowid_physical_position_tests.rs.

Still blocked upstream: crates.io latest is DataFusion 54.1.0 (2026-07-21); apache/datafusion#22026 ships in 55.0.0.