Add nodes to convert between csp.Structs and Arrow RecordBatches#698
Merged
Conversation
Add core Arrow adapter library with: - ArrowTypeVisitor: visitArrowValueType maps arrow::Type to C++ types - ArrowFieldReader: per-column value extraction from Arrow arrays - ArrowFieldWriter: per-column value serialization to Arrow builders - CMake build for csp_arrow_adapter static library - Centralize Arrow link-target resolution in FindDepsArrowAdapter Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
Add RecordBatchToStruct and StructToRecordBatch converters for bulk-converting between Arrow RecordBatches and csp::Struct vectors. Support pluggable custom field readers/writers for extension types. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
Add ArrowCppNodes with record_batches_to_struct and struct_to_record_batches CppNode implementations. Make RecordBatchPullInputAdapter schema optional (lazy extraction from first batch). Register graph functions in arrow.py. Simplify arrowadapterimpl build to link against csp_arrow_adapter. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
Add tests for record_batches_to_struct and struct_to_record_batches covering all scalar types, nested structs, schema validation, and round-trip conversion. Add RecordBatch pull input adapter tests. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
force-pushed
the
ac/arrow_nodes_scalar_support
branch
from
8920496 to
4068d0c
Compare
arhamchopra
changed the title
robambalu
requested changes
Apr 16, 2026
- Remove unnecessary ticked() checks on single-input nodes - Remove extra blank line in START() - Use PyList_GET_ITEM for faster unchecked access in loop - Use typed TS_INPUT(vector<StructPtr>) instead of Generic for struct_to_record_batches - Use standard @csp.node decorator instead of _node_internal_use Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>
arhamchopra
force-pushed
the
ac/arrow_nodes_scalar_support
branch
from
c2b253a to
506f5aa
Compare
arhamchopra
commented
Apr 16, 2026
Collaborator
Author
There was a problem hiding this comment.
Addressed all review feedback in commit 506f5aa:
robambalu
requested review from
AdamGlustein,
alexddobkin,
ptomecek,
svatasoiu and
timkpaine
as code owners
robambalu
approved these changes
Apr 16, 2026
robambalu
approved these changes
Apr 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Arrow Batch Adapter: Scalar Type Support
Adds bidirectional conversion between Arrow
RecordBatchandcsp.Structfor all scalar types, with new CSP graph nodes for use in running graphs.Motivation
CSP lacked a general-purpose way to move data between Arrow RecordBatches and csp Structs. The parquet adapter had its own bespoke conversion logic, and there was no reusable building block for other Arrow-based integrations (IPC, Flight, custom adapters). This PR introduces a shared, extensible Arrow ↔ Struct conversion layer that any adapter can use.
What's New
Core C++ Library (
cpp/csp/adapters/arrow/)ArrowTypeVisitor— Generic Arrow type → C++ type dispatch using template metaprogramming. Eliminates repetitive switch statements across the codebase.ArrowFieldReader— Column-oriented extraction from Arrow arrays to CSP values. Supports scalar types, dictionary-encoded columns, enums, nested structs, and pluggable custom readers for extension types (e.g. numpy arrays).ArrowFieldWriter— Mirror of FieldReader for serialization. Converts CSP struct fields to Arrow arrays via builders.RecordBatchToStruct/StructToRecordBatch— Batch converters built on the readers/writers. Pre-allocate structs and use columnarreadAll()/writeAll()for cache-friendly bulk conversion.Python Graph Nodes (
csp/adapters/arrow.py,ArrowCppNodes.cpp)csp.adapters.arrow.record_batches_to_struct(data, cls, ...)— Convertsts[List[pa.RecordBatch]]→ts[List[Struct]]csp.adapters.arrow.struct_to_record_batches(data, cls, ...)— Convertsts[List[Struct]]→ts[List[pa.RecordBatch]]field_mapfor column ↔ field renaming, optionalmax_batch_sizefor chunked output.Build System
csp_arrow_adapterstatic library with centralized Arrow dependency resolution (FindDepsArrowAdapter.cmake).arrowadapterimplPython extension — now links againstcsp_arrow_adapterinstead of resolving Arrow directly.Supported Types
boolintfloatstrbytesdatetimedatetimetimedeltaEnumcsp.StructExtensibility
The converters accept pluggable custom readers/writers for types not covered by the built-in scalar support (e.g. numpy list columns). Custom readers are passed at construction time and excluded from auto-detection — no core library changes needed to add new types.
This partially resolves #286