Is your feature request related to a problem or challenge?
We already write v3 row lineage (row ranges get assigned on the snapshot and
manifest during commit), but there's no way to read the two row-lineage
metadata columns back out during a scan:
_row_id
_last_updated_sequence_number
Both are defined in metadata_columns.rs but nothing actually produces their
values, so selecting _row_id in a scan currently errors out in the arrow read
pipeline, the field id passes projection validation and then there's no
constant/source for it.
For reference, Iceberg Java derives these on read rather than storing them per row.
_last_updated_sequence_number maps to the file's data sequence number:
the one that's preserved when a file is carried forward across a rewrite, not
the file sequence number (which changes on every commit that re-adds the file).
_row_id is first_row_id + position_in_file, where the per-file first_row_id is inherited when the
manifest is read (manifest-level first_row_id plus a running record count).
We don't do any of that inheritance/derivation on the read side yet.
Note the write side already does the right thing here . v3 stores compact ranges
and expects readers to derive the per-row values, so this is really just a
missing piece on the read path.
This is a building block for row-lineage consumers (CDC/MERGE-style engines on
top of the scan API), not a changelog/CDC scan feature itself.
Describe the solution you'd like
Happy to work on this. Part of the v3 row-lineage work (#2411) and complements
the write side (#1652); would follow the same metadata-column materialization
pattern as #2746 (_pos) and #2695 (_spec_id).
Willingness to contribute
I can contribute to this feature independently
Is your feature request related to a problem or challenge?
We already write v3 row lineage (row ranges get assigned on the snapshot and
manifest during commit), but there's no way to read the two row-lineage
metadata columns back out during a scan:
_row_id_last_updated_sequence_numberBoth are defined in
metadata_columns.rsbut nothing actually produces theirvalues, so selecting
_row_idin a scan currently errors out in the arrow readpipeline, the field id passes projection validation and then there's no
constant/source for it.
For reference, Iceberg Java derives these on read rather than storing them per row.
_last_updated_sequence_numbermaps to the file's data sequence number:the one that's preserved when a file is carried forward across a rewrite, not
the file sequence number (which changes on every commit that re-adds the file).
_row_idisfirst_row_id + position_in_file, where the per-filefirst_row_idis inherited when themanifest is read (manifest-level
first_row_idplus a running record count).We don't do any of that inheritance/derivation on the read side yet.
Note the write side already does the right thing here . v3 stores compact ranges
and expects readers to derive the per-row values, so this is really just a
missing piece on the read path.
This is a building block for row-lineage consumers (CDC/MERGE-style engines on
top of the scan API), not a changelog/CDC scan feature itself.
Describe the solution you'd like
Happy to work on this. Part of the v3 row-lineage work (#2411) and complements
the write side (#1652); would follow the same metadata-column materialization
pattern as #2746 (
_pos) and #2695 (_spec_id).Willingness to contribute
I can contribute to this feature independently