Skip to content

feat: add blob take and BlobFile read APIs - #82

Open
MisterRaindrop wants to merge 5 commits into
lance-format:mainfrom
MisterRaindrop:feat/blob-take-api
Open

feat: add blob take and BlobFile read APIs#82
MisterRaindrop wants to merge 5 commits into
lance-format:mainfrom
MisterRaindrop:feat/blob-take-api

Conversation

@MisterRaindrop

Copy link
Copy Markdown

Builds on #81; the first two commits are that PR.

Adds the row-addressed blob API from #76: lance_dataset_take_blobs (by _rowid) and lance_dataset_take_blobs_by_indices (by row offset) fill a caller array of LanceBlobFile*, NULL for a null value; the handle has size, read, read_up_to, read_range, seek, tell and close. Reads copy into caller buffers and return 0 / -1 with the thread-local error set.

The handle wraps upstream BlobFile only, so it outlives the dataset. Take is all-or-nothing (out untouched on error), read rejects a buffer smaller than the remaining bytes instead of truncating, and an unknown column is an invalid argument rather than LANCE_ERR_INTERNAL. lance.hpp gets an RAII BlobFile and Dataset::take_blobs*() returning std::vector<std::optional<BlobFile>>.

Tests: sixteen test_blob_* cases (both entry points over every storage layout, with and without stable row ids, cursor semantics, bounds and overflow, handles outliving the dataset, NULL and invalid arguments with out untouched) plus the C and C++ smoke programs.

Open: read goes from the cursor to the end like the Rust and Java read(), and seek is not bounds-checked, like upstream. take_blobs_by_addresses and kind / uri accessors are not included.

Adds LanceBlobHandling and lance_scanner_set_blob_handling(), applied in
build_scanner() right after project(). Values outside the enum and calls
after the scan has started are rejected.
Two-fragment Blob v2 fixture (inline, packed, dedicated, empty and null
rows) and six tests. The C and C++ smoke programs get a blob dataset as a
third argument and check the setter too.
lance_dataset_take_blobs / _by_indices fill a caller array of LanceBlobFile*
(NULL for null values, untouched on error). The handle has size, read,
read_up_to, read_range, seek, tell and close, and wraps upstream BlobFile
only, so it outlives the dataset.
Reuses the fixture from the scanner blob handling PR, with a stable row id
switch.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The opaque LanceBlobFile wrapper closes the C-access gap documented in #76 while preserving upstream random and range reads instead of forcing large payloads into scan batches. Input validation happens before caller output is touched, handle ownership is independent of the dataset, and the multi-fragment coverage exercises inline, packed, dedicated, empty, null, bounds, and cursor behavior through Rust, C, and C++.

This is stacked on #81, whose current head exactly matches this branch’s first two commits; land #81 first, then this PR.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant