Reading a Lance JSON column throws UnsupportedOperationException in Spark

[KnightChess]

Summary

Reading a Lance dataset that contains a JSON column fails at query time with:

Caused by: java.lang.UnsupportedOperationException
	at org.apache.spark.sql.vectorized.ArrowColumnVector$ArrowVectorAccessor.getBinary(ArrowColumnVector.java:253)
	at org.apache.spark.sql.vectorized.ArrowColumnVector.getBinary(ArrowColumnVector.java:117)
	at org.lance.spark.vectorized.LanceArrowColumnVector.getBinary(LanceArrowColumnVector.java:552)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:396)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:897)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:897)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:140)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:562)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1552)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:565)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Reproduction

1. Create a Lance dataset with a JSON column via PyArrow / lance-core.
2. Read it from Spark:

spark.read.format("lance").load(path).select("payload").show()

Expected: JSON text returned as a Spark string.
Actual: UnsupportedOperationException at task execution.

Suspected root cause

Lance JSON columns are exposed via Arrow as LargeBinary with the extension marker ARROW:extension:name = lance.json (matches lance-core's is_json_field in rust/lance-arrow/src/json.rs). At runtime, however, the data is delivered as a VarCharVector (Utf8).

LanceArrowUtils.convertArrowFieldType doesn't recognize the marker, so LargeBinary maps to BinaryType. Spark's codegen then dispatches to getBinary() on the VarCharVector accessor, which throws.

Could someone confirm whether this diagnosis is correct?

Proposed fix

Recognize the lance.json extension in convertArrowFieldType and return StringType:

1. JsonTypeUtil.java — new utility class (sibling of Float16Utils / LargeVarCharUtils):
   public static final String ARROW_EXTENSION_JSON = "lance.json";
   public static boolean isJsonArrowField(Field field) { /* metadata check */ }
2. LanceArrowUtils.scala — add a guarded case ahead of the generic LargeBinary branch:
   case _: ArrowType.LargeBinary if JsonTypeUtil.isJsonArrowField(field) => StringType
3. Also preserve the extension marker in augmentTypeMarkers so downstream consumers can identify the origin.
4. No change to LanceArrowColumnVector — once the schema is StringType, Spark dispatches to getUTF8String, which the fallback ArrowColumnVector handles correctly on the VarCharVector. The write path is untouched.

With this change, JSON columns become readable from Spark as strings — a simple SELECT payload FROM t returns the JSON text without error.

Ask for the community

1. Does the diagnosis above match what maintainers expect?
2. Any objection to the fix direction, or to introducing JsonTypeUtil as the home for JSON extension handling?

Happy to open a PR against main if the approach looks reasonable.