TakeOperation::try_from_expr in rust/lance/src/dataset/scanner.rs matches Expr::InList without consulting negated, so a filter like _rowid NOT IN (...) is planned as a take of exactly the listed ids. That is the complement of what the user asked for. The arm returns no remainder, so the predicate is dropped from the plan rather than merely ignored by the take.
Small lists never reach the lowering: DataFusion's expression simplifier expands them into a conjunction of != comparisons first. A three-element list is expanded; a ten-element list arrives with negated: true and hits the lowering. So the bug is invisible in small examples and wrong in large ones, which is the worst shape for it to have.
Reproducer on a 12-row dataset with an idx column of 0..11:
ds.scan().filter("_rowid NOT IN (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)")
returns idx = [0..9]. The correct answer is [10, 11].
_rowaddr and _rowoffset go through the same arm and are affected identically.
Reproduced on main at ebba5814c.
TakeOperation::try_from_exprinrust/lance/src/dataset/scanner.rsmatchesExpr::InListwithout consultingnegated, so a filter like_rowid NOT IN (...)is planned as a take of exactly the listed ids. That is the complement of what the user asked for. The arm returns no remainder, so the predicate is dropped from the plan rather than merely ignored by the take.Small lists never reach the lowering: DataFusion's expression simplifier expands them into a conjunction of
!=comparisons first. A three-element list is expanded; a ten-element list arrives withnegated: trueand hits the lowering. So the bug is invisible in small examples and wrong in large ones, which is the worst shape for it to have.Reproducer on a 12-row dataset with an
idxcolumn of 0..11:returns
idx = [0..9]. The correct answer is[10, 11]._rowaddrand_rowoffsetgo through the same arm and are affected identically.Reproduced on
mainatebba5814c.