Skip to content

Commit 1943db3

Browse files
committed
Read source arrays through a nanoarrow ArrowArrayView
Reading walked the ArrowArrays directly: hasNulls counted the bits of a validity bitmap by hand where a producer had left null_count at -1, and ListOffsets dispatched on the format string to read 32-bit ("+l") or 64-bit ("+L") offsets out of buffer 1, tracking the slot offset a sliced array carries. An ArrowArrayView does both. InputStream lays one out from the geometry schema once and points it at each batch as it arrives, so the readers take a view rather than a schema and array pair, hasNulls becomes a call to ArrowArrayViewComputeNullCount, and the offsets come from ArrowArrayViewListChildOffset, which picks its width from the view's storage type. That function indexes the offsets buffer directly, so listBegin still adds the view's own offset - the case that a new test_sliced_arrays pins for both geometry types, and which fails without it. Pointing the view at a batch also checks the batch over, which is new: its children and buffers must be the shape the schema promised and its offsets must stay inside the arrays they index. A stream that declares linestrings and then yields something else used to be read blindly through the declared schema; it now raises, with the reason. Verified bit-identical against the same golden arrays, with flat ru_maxrss across the stream lifecycles and rejected sources. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SqdWshmD4AUHhrqeMh86GS
1 parent 6d438ea commit 1943db3

2 files changed

Lines changed: 155 additions & 158 deletions

File tree

0 commit comments

Comments
 (0)