Allow Spark to write external blob references directly, preserving the URI as an external reference instead of requiring callers to materialize the asset bytes or update the column later through another binding. This is useful when registering media that already exists in object storage.
The logical blob shape supports data, uri, position, and size, but the connector currently exposes only the binary-payload write path. External writes were explicitly deferred in #539; this issue tracks that follow-up.
With a configured Lance catalog and an existing namespace, this SQL was tested on 8780cbdab386b40b9ba515264597a02fd027f11a (Spark 3.5 / Scala 2.13, Lance Java 11.0.0-beta.21):
CREATE TABLE lance_ns.default.external_blob_probe
(content STRUCT<data:BINARY,uri:STRING,position:BIGINT,size:BIGINT>)
USING lance TBLPROPERTIES
('content.lance.encoding'='blob', 'file_format_version'='2.2');Creation fails with Blob column 'content' must have BINARY type. This rejection was asserted in a local test using the existing BaseBlobCreateTableTest setup.
There are also downstream restrictions:
Provide a documented external-reference input path for create and append, including whole-object URIs and explicit ranges. Preserve existing binary writes and delegate logical blob validation to Lance core where possible. Tests should verify that the stored descriptor remains external and that reading resolves the intended object or range.
Cross-bucket credential vending is a separate concern: accepting an external reference must not imply that access to the referenced location has been granted.
Allow Spark to write external blob references directly, preserving the URI as an external reference instead of requiring callers to materialize the asset bytes or update the column later through another binding. This is useful when registering media that already exists in object storage.
The logical blob shape supports
data,uri,position, andsize, but the connector currently exposes only the binary-payload write path. External writes were explicitly deferred in #539; this issue tracks that follow-up.With a configured Lance catalog and an existing namespace, this SQL was tested on
8780cbdab386b40b9ba515264597a02fd027f11a(Spark 3.5 / Scala 2.13, Lance Java11.0.0-beta.21):Creation fails with
Blob column 'content' must have BINARY type. This rejection was asserted in a local test using the existingBaseBlobCreateTableTestsetup.There are also downstream restrictions:
LanceWriteSchemaValidatoraccepts only binary input for blob v2 columns.BlobV2StructWriterpopulates onlydataand leavesuri,position, andsizenull.Provide a documented external-reference input path for create and append, including whole-object URIs and explicit ranges. Preserve existing binary writes and delegate logical blob validation to Lance core where possible. Tests should verify that the stored descriptor remains external and that reading resolves the intended object or range.
Cross-bucket credential vending is a separate concern: accepting an external reference must not imply that access to the referenced location has been granted.