_rowid point lookup drops valid non-stable row IDs from fragments

[unconsolable]

Overview

Rowid point lookup incorrectly returns no rows for valid non-stable row IDs that belong to fragments after the first fragment.

When stable row IDs are disabled, Lance exposes _rowid values as encoded row addresses:

rowid = (fragment_id << 32) | row_offset

However, the lance-duckdb take path filters row IDs by comparing them with the dataset's total row count. Valid row addresses from later fragments are therefore treated as out of range and removed before calling Dataset::take_rows.

let max_row_id = if handle.dataset.manifest.uses_stable_row_ids() {
    handle.dataset.manifest.next_row_id
} else {
    handle
        .dataset
        .manifest
        .fragments
        .iter()
        .map(|fragment| fragment.num_rows().unwrap_or_default() as u64)
        .sum::<u64>()
};

Environment

• lance-duckdb commit: 63c2446
• Lance version: 9.0.0
• DuckDB platform: linux_amd64

Reproduction

Build the extension:

GEN=ninja DUCKDB_PLATFORM=linux_amd64 make release

Create a dataset with multiple fragments:

tmp_dir="$(mktemp -d /tmp/lance-rowid-take-repro.XXXXXX)"

./build/release/duckdb -unsigned -c "
LOAD 'build/release/extension/lance/lance.duckdb_extension';

COPY (
    SELECT range::BIGINT AS id
    FROM range(6)
)
TO '${tmp_dir}/data.lance'
(
    FORMAT lance,
    max_rows_per_file 2
);
"

Inspect the generated row IDs:

./build/release/duckdb -unsigned -c "
LOAD 'build/release/extension/lance/lance.duckdb_extension';

SELECT id, _rowid
FROM '${tmp_dir}/data.lance'
ORDER BY id;
"

Output:

id  _rowid
0   0
1   1
2   4294967296
3   4294967297
4   8589934592
5   8589934593

The values 4294967296 and 8589934592 are valid row addresses for the first rows of fragments 1 and 2.

Run a rowid point lookup:

./build/release/duckdb -unsigned -c "
LOAD 'build/release/extension/lance/lance.duckdb_extension';

SELECT id
FROM '${tmp_dir}/data.lance'
WHERE _rowid IN (4294967296, 8589934592)
ORDER BY id;
"

Expected result

2
4

Actual result

0 rows