pushDownFilters=true can trigger ArrowBuf IndexOutOfBoundsException when reading sliced LargeUtf8 columns

[ziwenzhang]

Summary

When the following configuration is enabled:

SET spark.sql.catalog.lance.pushDownFilters = true;

reading a Lance LargeUtf8 column can fail with an ArrowBuf IndexOutOfBoundsException.

The issue occurs when filter pushdown allows Lance to perform filtering and offset operations internally, producing a sliced variable-width Arrow array that is subsequently imported into Arrow Java through the C Data interface.

The issue does not reproduce when spark.sql.catalog.lance.pushDownFilters is disabled because the filters remain in Spark and the affected sliced LargeUtf8 C Data import path is avoided.

This affects lance-spark modules using Arrow Java versions older than 18.3.0. The same low-level reproducer fails with Arrow Java 15.0.2 and passes with Arrow Java 18.3.0.

Minimal Spark SQL reproduction

The following test creates 61 rows:

• row 0 contains 522262 bytes and is skipped by OFFSET 1
• rows 1–59 contain 2654 bytes each
• row 60 contains 2052 bytes

The visible slice therefore contains:

59 * 2654 + 2052 = 158638 bytes

CREATE TABLE lance.default.large_var_char_sql_oob (
  id BIGINT,
  keep INT,
  score DOUBLE,
  text STRING
)
USING lance
TBLPROPERTIES (
  'text.arrow.large_var_char' = 'true'
);

INSERT INTO lance.default.large_var_char_sql_oob
SELECT
  id,
  1 AS keep,
  CAST(NULL AS DOUBLE) AS score,
  CASE
    WHEN id = 0 THEN repeat('p', 522262)
    WHEN id < 60 THEN repeat('x', 2654)
    ELSE repeat('y', 2052)
  END AS text
FROM range(0, 61, 1, 1);

SET spark.sql.catalog.lance.pushDownFilters = true;

EXPLAIN FORMATTED
SELECT
  COUNT(id),
  AVG(length(text))
FROM (
  SELECT id, text
  FROM lance.default.large_var_char_sql_oob
  WHERE keep = 1
    AND score IS NULL
    AND text IS NOT NULL
  OFFSET 1
) t;

SELECT
  COUNT(id),
  AVG(length(text))
FROM (
  SELECT id, text
  FROM lance.default.large_var_char_sql_oob
  WHERE keep = 1
    AND score IS NULL
    AND text IS NOT NULL
  OFFSET 1
) t;

The formatted plan includes the pushed filters, projection, and offset:

BatchScan ...
Output: [id, text]
offset: Optional[1]
whereConditions: Optional[...]

Note about LIMIT

The original workload used:

LIMIT 60 OFFSET 1

On Spark 3.5, lance-spark reports LIMIT as partially pushed through SupportsPushDownLimit.isPartiallyPushed(). Spark therefore does not attempt to push the OFFSET from a combined LIMIT/OFFSET node.

The minimal reproduction omits LIMIT. Because the input contains 61 rows, OFFSET 1 still produces exactly the same 60-row sliced array.

Observed behavior

With filter pushdown enabled:

SET spark.sql.catalog.lance.pushDownFilters = true;

the aggregation fails with:

java.lang.IndexOutOfBoundsException: index: 522262, length: 2654 (expected: range(0, 158638))
  at org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:701)
  at org.apache.arrow.memory.ArrowBuf.getBytes(ArrowBuf.java:728)
  at org.apache.arrow.vector.LargeVarCharVector.get(LargeVarCharVector.java:114)
  at org.lance.spark.vectorized.LargeVarCharAccessor.getUTF8String(LargeVarCharAccessor.java:32)
  at org.lance.spark.vectorized.LanceArrowColumnVector.getUTF8String(LanceArrowColumnVector.java:516)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1
      .hashAgg_doAggregateWithoutKey_0$(Unknown Source)

Expected behavior

The query should complete successfully with filter pushdown enabled and return:

count(id) = 60
avg(length(text)) = 158638 / 60

Reading a valid sliced LargeUtf8 array should not depend on whether its first visible value starts at byte offset zero.

Workaround

Disabling filter pushdown avoids the affected path:

SET spark.sql.catalog.lance.pushDownFilters = false;

With pushdown disabled, the filters remain in Spark and Lance does not return the same internally filtered and sliced LargeUtf8 array through the affected C Data import path.

This is only a workaround because it disables a connector optimization and may affect query performance.

Root cause

A sliced LargeUtf8 array can retain offsets based on the original value buffer:

firstOffset = 522262
lastOffset  = 680900

Arrow Java versions before 18.3.0 calculate the imported value-buffer capacity as:

lastOffset - firstOffset = 158638

However, LargeVarCharVector.get(0) uses the absolute offset stored in the offset buffer:

valueBuffer.getBytes(522262, ..., 2654)

This mixes two coordinate systems:

• the offset buffer contains absolute offsets into the original value buffer
• the imported value buffer capacity only covers the visible slice length

The resulting access starts at byte 522262 in a buffer whose imported capacity is only 158638.

This is the Arrow Java issue fixed by:

• Accessing values from imported 'C data interface' array can result in out of bounds reads apache/arrow-java#709
• GH-709: Correct length calculation of value buffers of variable-sized arrays apache/arrow-java#707

The upstream fix changed the variable-width value-buffer length calculation from:

lastOffset - firstOffset

to:

lastOffset

The fix was released in Arrow Java 18.3.0.

Version comparison

Verified locally:

Module	Arrow Java	Result
lance-spark-3.5_2.12	15.0.2	Reproduces the OOB with pushDownFilters=true
lance-spark-4.1_2.13	18.3.0	Equivalent C Data regression test passes

Based on the Arrow Java fix version, the following currently pinned modules may also be affected when filter pushdown produces a sliced variable-width array:

Spark module	Arrow Java
Spark 3.4	11.0.0
Spark 3.5	12.0.1
Spark 4.1	18.3.0

Spark 4.1 already uses Arrow Java 18.3.0.

Questions

Would upgrading the affected modules to Arrow Java 18.3.0 or newer be an acceptable fix, assuming compatibility with the corresponding Spark releases?

If so, should all Arrow Java artifacts be upgraded together to keep their versions aligned, including:

arrow-vector
arrow-c-data
arrow-memory-core
arrow-memory-netty
arrow-format
arrow-dataset

If upgrading Arrow is not feasible for some supported Spark versions, would the maintainers recommend normalizing or materializing sliced variable-width arrays inside lance-spark before they reach the affected Arrow Java C Data import path?

I would appreciate guidance on the preferred approach and would be happy to help prepare a fix and regression tests.

Environment

• lance-spark commit: 58a37dd
• Spark: 3.5.1
• Scala: 2.12
• Arrow Java: 15.0.2
• lance-core: 8.0.0
• Java: 17.0.11
• OS: macOS arm64