Support writing a sequence of blobs in one row from Spark, for example a clip containing multiple image frames. Lance core already supports List<blob>; the connector currently accepts blob encoding only on top-level BINARY columns.
With a configured Lance catalog and an existing namespace, the following SQL was tested on 8780cbdab386b40b9ba515264597a02fd027f11a (Spark 3.5 / Scala 2.13, Lance Java 11.0.0-beta.21):
CREATE TABLE lance_ns.default.list_blob_probe (frames ARRAY<BINARY>)
USING lance TBLPROPERTIES
('frames.lance.encoding'='blob', 'file_format_version'='2.2');
This fails with Blob column 'frames' must have BINARY type. Trying an element path instead silently leaves the array as ordinary binary data:
CREATE TABLE lance_ns.default.list_element_probe (frames ARRAY<BINARY>)
USING lance TBLPROPERTIES
('frames.item.lance.encoding'='blob', 'file_format_version'='2.2');
INSERT INTO lance_ns.default.list_element_probe VALUES (array(X'0102', X'0304'));
The second case succeeds, but the read schema remains array<binary> with no blob extension metadata. Both outcomes were asserted in local tests using the existing BaseBlobCreateTableTest setup.
SchemaConverter.addBlobMetadata only visits top-level fields and requires BinaryType. By contrast, Lance v10.0.0 already includes a list blob read/write test. Ordinary list<binary> is distinct from a list whose element carries the blob extension.
Provide a documented way to mark array elements as blob v2 and preserve that contract through create, append, and reads. Coverage should include multiple blobs per row, null elements, empty/null lists, and payloads stored outside the data file. Unsupported nested property paths should not silently appear to enable blob encoding.
Related to the blob v2 support work in #539. The array element writer metadata defect is tracked separately in #814 and also needs to be addressed for binary array input.
Support writing a sequence of blobs in one row from Spark, for example a clip containing multiple image frames. Lance core already supports
List<blob>; the connector currently accepts blob encoding only on top-levelBINARYcolumns.With a configured Lance catalog and an existing namespace, the following SQL was tested on
8780cbdab386b40b9ba515264597a02fd027f11a(Spark 3.5 / Scala 2.13, Lance Java11.0.0-beta.21):This fails with
Blob column 'frames' must have BINARY type. Trying an element path instead silently leaves the array as ordinary binary data:The second case succeeds, but the read schema remains
array<binary>with no blob extension metadata. Both outcomes were asserted in local tests using the existingBaseBlobCreateTableTestsetup.SchemaConverter.addBlobMetadataonly visits top-level fields and requiresBinaryType. By contrast, Lance v10.0.0 already includes a list blob read/write test. Ordinarylist<binary>is distinct from a list whose element carries the blob extension.Provide a documented way to mark array elements as blob v2 and preserve that contract through create, append, and reads. Coverage should include multiple blobs per row, null elements, empty/null lists, and payloads stored outside the data file. Unsupported nested property paths should not silently appear to enable blob encoding.
Related to the blob v2 support work in #539. The array element writer metadata defect is tracked separately in #814 and also needs to be addressed for binary array input.