Skip to content

feature: support writing arrays of blob v2 values #815

Description

@Xuanwo

Support writing a sequence of blobs in one row from Spark, for example a clip containing multiple image frames. Lance core already supports List<blob>; the connector currently accepts blob encoding only on top-level BINARY columns.

With a configured Lance catalog and an existing namespace, the following SQL was tested on 8780cbdab386b40b9ba515264597a02fd027f11a (Spark 3.5 / Scala 2.13, Lance Java 11.0.0-beta.21):

CREATE TABLE lance_ns.default.list_blob_probe (frames ARRAY<BINARY>)
USING lance TBLPROPERTIES
('frames.lance.encoding'='blob', 'file_format_version'='2.2');

This fails with Blob column 'frames' must have BINARY type. Trying an element path instead silently leaves the array as ordinary binary data:

CREATE TABLE lance_ns.default.list_element_probe (frames ARRAY<BINARY>)
USING lance TBLPROPERTIES
('frames.item.lance.encoding'='blob', 'file_format_version'='2.2');
INSERT INTO lance_ns.default.list_element_probe VALUES (array(X'0102', X'0304'));

The second case succeeds, but the read schema remains array<binary> with no blob extension metadata. Both outcomes were asserted in local tests using the existing BaseBlobCreateTableTest setup.

SchemaConverter.addBlobMetadata only visits top-level fields and requires BinaryType. By contrast, Lance v10.0.0 already includes a list blob read/write test. Ordinary list<binary> is distinct from a list whose element carries the blob extension.

Provide a documented way to mark array elements as blob v2 and preserve that contract through create, append, and reads. Coverage should include multiple blobs per row, null elements, empty/null lists, and payloads stored outside the data file. Unsupported nested property paths should not silently appear to enable blob encoding.

Related to the blob v2 support work in #539. The array element writer metadata defect is tracked separately in #814 and also needs to be addressed for binary array input.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions