_rowoffset can be used in a scan filter only when the predicate lowers into TakeOperation::RowOffsets, that is a positive IN list of literal offsets. Anything else is rejected. On main at ebba5814c:
ds.scan().filter("_rowoffset > 2")
fails with Column _rowoffset does not exist from Projection::union_column, which has arms for _rowid, _rowaddr, _row_last_updated_at_version and _row_created_at_version but none for _rowoffset (rust/lance-core/src/datatypes/schema.rs:1272). Adding that arm is not sufficient on its own: the filtered read then rejects the predicate at rust/lance-datafusion/src/planner.rs:1089 with FieldNotFound { field: _rowoffset, valid_fields: [idx, _rowid, _rowaddr] }, because the column is never materialized into the schema the filter is planned against.
_rowoffset is a pure function of _rowaddr (the low 32 bits), so the shape of the fix is to request _rowaddr whenever a filter mentions _rowoffset and compute the derived column before filter refinement. That makes every predicate over it work, not just the IN shape.
This came up while fixing #9080: a negated _rowoffset list used to lower into a take of the listed offsets, which returned the complement of what was asked. That is now rejected instead, so the wrong-results bug is gone, but the predicate still cannot be answered.
_rowoffsetcan be used in a scan filter only when the predicate lowers intoTakeOperation::RowOffsets, that is a positiveINlist of literal offsets. Anything else is rejected. Onmainatebba5814c:fails with
Column _rowoffset does not existfromProjection::union_column, which has arms for_rowid,_rowaddr,_row_last_updated_at_versionand_row_created_at_versionbut none for_rowoffset(rust/lance-core/src/datatypes/schema.rs:1272). Adding that arm is not sufficient on its own: the filtered read then rejects the predicate atrust/lance-datafusion/src/planner.rs:1089withFieldNotFound { field: _rowoffset, valid_fields: [idx, _rowid, _rowaddr] }, because the column is never materialized into the schema the filter is planned against._rowoffsetis a pure function of_rowaddr(the low 32 bits), so the shape of the fix is to request_rowaddrwhenever a filter mentions_rowoffsetand compute the derived column before filter refinement. That makes every predicate over it work, not just theINshape.This came up while fixing #9080: a negated
_rowoffsetlist used to lower into a take of the listed offsets, which returned the complement of what was asked. That is now rejected instead, so the wrong-results bug is gone, but the predicate still cannot be answered.