Skip to content

Support reading _row_id and _last_updated_sequence_number metadata columns #2879

Description

@anoopj

Is your feature request related to a problem or challenge?

We already write v3 row lineage (row ranges get assigned on the snapshot and
manifest during commit), but there's no way to read the two row-lineage
metadata columns back out during a scan:

  • _row_id
  • _last_updated_sequence_number

Both are defined in metadata_columns.rs but nothing actually produces their
values, so selecting _row_id in a scan currently errors out in the arrow read
pipeline, the field id passes projection validation and then there's no
constant/source for it.

For reference, Iceberg Java derives these on read rather than storing them per row.
_last_updated_sequence_number maps to the file's data sequence number:
the one that's preserved when a file is carried forward across a rewrite, not
the file sequence number (which changes on every commit that re-adds the file).
_row_id is first_row_id + position_in_file, where the per-file first_row_id is inherited when the
manifest is read (manifest-level first_row_id plus a running record count).
We don't do any of that inheritance/derivation on the read side yet.

Note the write side already does the right thing here . v3 stores compact ranges
and expects readers to derive the per-row values, so this is really just a
missing piece on the read path.

This is a building block for row-lineage consumers (CDC/MERGE-style engines on
top of the scan API), not a changelog/CDC scan feature itself.

Describe the solution you'd like

Happy to work on this. Part of the v3 row-lineage work (#2411) and complements
the write side (#1652); would follow the same metadata-column materialization
pattern as #2746 (_pos) and #2695 (_spec_id).

Willingness to contribute

I can contribute to this feature independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions