infer_arrow: the columnar boundary — pa.Table in, pa.Table out (TASK-60) - #47
Merged
Merged
Conversation
… pa.Table out (TASK-60) Ingest walks pyarrow buffers directly (address+size via the Python buffer API, validity/bool bitmaps, utf8 offsets, slice offsets honored — no arrow-rs dependency) into ColData lanes; emit builds one pa.Array.from_buffers per OUTPUT COLUMN from rust-built buffers. Zero per-value Python objects on either side. Strict-by-name column matching; named rejections for multi-chunk, missing columns, wrong dtypes, struct models, and the constant path. The differential contract (infer_rows == infer_arrow, every serving scenario + NULLs + 'many' LEFT null-extensions) is the test suite. Measured p50 (release): at n>=1024 infer_arrow beats the row path on every scenario (house_prices 5308us -> 2795us) and beats the handcrafted python twin on most (house 0.55x, fraud 0.80x, titanic/ store ~0.9-1.0x) — the first wins over the twin, exactly the boundary thesis. At n=64 the pyarrow API's fixed ~150us favors the row path (documented; the columnar-core wave addresses compute next). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ahrzb
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of the approved columnar roadmap (proposal). An additive fast lane: zero per-value Python objects on either boundary.
How
buffer.address/sizevia the Python buffer API — no arrow-rs dependency): validity + bool bitmaps (LSB), utf8/large_utf8 offsets, non-zero array offsets (slices) honored. Copy-first per the proposal; zero-copy is a measured follow-up.pa.Array.from_buffersper OUTPUT column from rust-built buffers (validity bitmap + data / offsets+bytes). Strings emit aslarge_string.combine_chunks(), missing column, wrong dtype → cast first, struct models → row path, constant queries). The differential —infer_rows == infer_arrowon every serving scenario, NULLs, andshape='many'LEFT null-extensions — is the test suite: parity by construction.Measured (p50, release)
At n≥1024 the columnar boundary beats the row path everywhere and beats the handcrafted python twin on most scenarios — the first twin wins (house_prices: 1.8× faster). At n=64 pyarrow's fixed API overhead (~150 µs) favors the row path; the columnar-core wave attacks compute next, with the batch-size scaling report.
🤖 Generated with Claude Code