Feature: Push down nested struct subfield projections to Lance scanner for IO pruning

[yyzhao2025]

Feature: Push down nested struct subfield projections to Lance scanner for IO pruning

Background

Spark's nestedSchemaPruning optimizer can prune nested struct schemas so that only the requested sub-fields are passed to the data source. However, the current lance-spark connector does not exploit this information at the IO layer — it still passes top-level struct column names to the Lance scanner. As a result, Lance reads the entire struct column from disk, and Spark discards the unneeded sub-fields in memory after the scan. This means the IO benefit of Spark's nested schema pruning is entirely lost.

The existing ReadSchemaNestedStructWidening mechanism correctly handles the correctness aspect of nested schema pruning — it widens the pruned schema back to the full struct to prevent Arrow child ordinal mismatches between the pruned schema and the physical Arrow layout. But it does not address the performance aspect: physical IO pruning at the Lance scanner level.

This is particularly impactful for wide, deeply nested struct columns commonly found in data lake workloads (e.g. metadata, attributes, usage_metrics structs with dozens of fields), where reading the full struct can easily account for a large fraction of total scan IO.

Proposed Design

Add a nested subfield projection pushdown layer that bridges Spark's nested schema pruning to Lance's column-projection capability. The implementation spans three phases:

1. Planning phase (driver)

• Introduce ReadSchemaNestedColumnProjection that recursively walks the required schema and expands non-nullable struct fields into dot-separated leaf column paths (e.g. usage_metrics.token_total).
• Add projectedColumns to LanceScan and LanceInputPartition, carrying the leaf path list from driver to executor.
• Include projectedColumns in LanceScan.equals() / hashCode() to preserve correct scan identity for Spark's ReusedExchange mechanism.

2. Scan phase (executor — Lance scanner)

• In LanceFragmentScanner, pass the projectedColumns list directly to ScanOptions.columns() instead of the top-level field names. This tells Lance to read only the requested leaf columns.

3. Reconstruction phase (executor — columnar batch assembly)

• Introduce ProjectedStructColumnVector, a lightweight ColumnVector subclass that wraps individual child vectors and presents them as a complete struct to Spark's physical plan.
• In LanceFragmentColumnarBatchScanner, detect when a struct has projected children by checking whether any actual output field key starts with the struct path prefix, and reconstruct the struct view from leaf vectors.
• For sibling fields not included in the projection, fill with ConstantColumnVector nulls — consistent with the existing widening contract.

Correctness Guardrails

To avoid correctness regressions, the implementation follows conservative boundaries:

• Non-nullable structs only — Only push down sub-fields when the parent struct is non-nullable. Nullable structs are kept as top-level projections because reconstructing a nullable parent from child vectors loses the parent validity bitmap, which would break struct IS NULL / struct IS NOT NULL semantics and null-sensitive aggregations.

• Array / Map boundaries — Stop recursion at ArrayType and MapType. These types have more complex rep/def level semantics and are not handled in this initial implementation.

• Top-level parity — When the required schema equals the full schema for a struct (no pruning), the projected column list is identical to the top-level column name, so behavior is unchanged from the current code path.

Testing Strategy

• Unit tests for ReadSchemaNestedColumnProjection:

  • Deeply nested struct expansion
  • Nullable struct fallback to top-level projection
  • Array / Map boundary behavior
  • Missing field error handling
  • Scanner special field filtering (_fragid, _rowid, etc.)

• Unit tests for ProjectedStructColumnVector:

  • Child access by ordinal
  • Idempotent close() behavior

• Integration tests:

  • Nested struct aggregation consistency with and without nestedSchemaPruning
  • Nullable struct null semantics when projecting subfields
  • End-to-end verification that leaf-level projection actually reaches the Lance scanner

Relation to Roadmap

This fits under the "Read Performance" category in the 2026 Spark Lance Connector Roadmap, alongside other read path optimization items like "Better split planning" and "Optimizing aggregate pushdown".

I have a working implementation and can open a PR if there is interest.