Add support for lists to Arrow <-> csp.Struct conversion nodes by arhamchopra · Pull Request #701 · Point72/csp

arhamchopra · 2026-04-16T19:57:26Z

Arrow RecordBatch ↔ csp.Struct: Numpy Array Support

Summary

Extends the Arrow RecordBatch ↔ csp.Struct conversion (introduced in #698) to support numpy array fields — both 1D (Numpy1DArray[T]) and N-dimensional (NumpyNDArray[T]).

Struct fields annotated as numpy arrays are serialized to/from Arrow List<element> columns. For ND arrays, a companion List<Int64> dimensions column stores the shape, enabling lossless round-trip of arbitrary-dimensional data.

What's New

C++ Layer

ArrowNumpyListReader.h — Reads Arrow list columns into numpy arrays. Supports native types (double, int64, int32, bool), unicode strings, and temporal types (datetime64, timedelta64, date). Includes fast-path bulk memcpy for contiguous numeric data with no nulls.
ArrowNumpyListWriter.h — Writes numpy arrays into Arrow list columns. Handles C-contiguous and non-contiguous arrays, NaT sentinel round-tripping for temporal types, and UTF-8 ↔ char32_t conversion for strings.
ArrowCppNodes.cpp — Extended record_batches_to_struct and struct_to_record_batches CppNodes to detect numpy fields in the properties dict, create the appropriate readers/writers, and wire up reshape/shape callbacks for ND arrays.
NumpyConversions.h/.cpp — Moved numpyReshape, numpyShape, and npyTypeFromPyType utility functions here from ArrowCppNodes for better code organization.

Python Layer

csp/adapters/arrow.py — record_batches_to_struct and struct_to_record_batches graph functions now inspect struct metadata to separate numpy fields from scalar fields, resolve dimension column names for ND arrays, and pass everything through to the C++ nodes.

Element Types Supported

Python Type Annotation	Arrow Representation	Numpy dtype
`Numpy1DArray[float]`	`List<Float64>`	`float64`
`Numpy1DArray[int]`	`List<Int64>`	`int64`
`Numpy1DArray[bool]`	`List<Boolean>`	`bool`
`Numpy1DArray[str]`	`List<Utf8>`	`U<n>` (unicode)
`Numpy1DArray[datetime]`	`List<Timestamp(ns, UTC)>`	`datetime64[ns]`
`Numpy1DArray[timedelta]`	`List<Duration(ns)>`	`timedelta64[ns]`
`Numpy1DArray[date]`	`List<Date32>`	`datetime64[ns]`
`NumpyNDArray[T]`	`List<...>` + `List<Int64>` dims	same as 1D

Architecture

Python layer (arrow.py)
  │
  ├── Detects Numpy1DArray / NumpyNDArray fields from struct metadata
  ├── Separates scalar vs numpy field maps
  └── Passes numpy_fields, numpy_element_types, numpy_dimension_names to C++
        │
C++ CppNodes (ArrowCppNodes.cpp)
  │
  ├── Read path: creates NumpyArrayReader or NumpyNDArrayReader per field
  │     └── Reader dispatches by Arrow element type → readValue lambda
  │
  └── Write path: creates NumpyArrayWriter or NumpyNDArrayWriter per field
        └── Writer dispatches by numpy dtype → appropriate builder

Core arrow layer (ArrowFieldReader/Writer)
  │
  └── registerListFieldReader/WriterFactory() hooks let the numpy layer
      plug into the generic field creation path (used by parquet adapter)

This along with #698 resolve #286 .

Add ArrowNumpyListReader and ArrowNumpyListWriter that handle numpy ndarray fields as Arrow list-of-primitive columns. Update NumpyConversions.h with helpers for Arrow↔numpy type mapping. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

Update ArrowCppNodes to register custom numpy FieldReaders/Writers for ndarray struct fields. Update arrow.py graph functions to detect and handle numpy array columns in struct types. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

Add tests for numpy ndarray fields in record_batches_to_struct and struct_to_record_batches: 1D/2D arrays, various dtypes, round-trip conversion, edge cases. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

Tests expose two bugs: - StringListWriter uses PyArray_GETPTR1 which is incorrect for ndim>1 - TemporalListWriter writes NaT as raw int64 instead of Arrow null Also fixes pre-existing MiddleStruct -> InnerStruct typo that prevented the test file from being collected. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

- StringListWriter: use flat pointer arithmetic on C-contiguous data instead of PyArray_GETPTR1 which uses strides[0] only, causing OOB access for ndim > 1 arrays - makeStringListItemsWriter: same fix for the factory path - TemporalListWriter: check for NPY_DATETIME_NAT and emit Arrow null instead of writing raw INT64_MIN as a value (which overflows on scaling and loses NaT semantics) - DateListWriter: same NaT check before date conversion Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

These functions are pure numpy utilities, not CppNode-specific. Moving them to NumpyConversions.h/.cpp keeps ArrowCppNodes.cpp focused on node wiring and makes the utilities reusable. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

…tting - Make field_map Optional in both record_batches_to_struct and struct_to_record_batches; Python resolves None to identity mapping and always passes explicit map to C++ - Reorder record_batches_to_struct signature: schema before field_map (required param before optional) - Extract _extract_numpy_metadata utility to deduplicate field categorization logic - Remove C++ auto-detect branches in StructToRecordBatch and RecordBatchToStruct; both now require explicit fieldMap and throw on invalid field names - Remove resolveFieldName helper from RecordBatchToStruct - Formatting fixes: template return+name on same line (reader), single-line reinterpret_cast (writer), makeStringListItemsWriter signature, member alignment Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

…oral reader, NPY_TYPE specializations - Move timeUnitMultiplier to ArrowFieldReader.h as inline, remove duplicate - Unify makeDurationListReadValue into makeTemporalListReadValue with dtype param - Remove static dtype cache, capture dtype by lambda instead - Add NPY_TYPE<DateTime>, NPY_TYPE<TimeDelta>, NPY_TYPE<Date> specializations - Simplify npyTypeFromPyType: add temporal types to PartialSwitchCspType - Use npy_intp loop variable when iterating over npy_intp size Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

…optimize null dims - Extract ListFieldWriterBase with shared reserve/writeNull/finish/m_listBuilder - Add ArrowArrayType<T> trait (bool -> uint8_t), remove makeBoolListItemsWriter - NumpyNDArrayWriter: write null dims when shape matches previous row - NumpyNDArrayReader: reuse last-seen dims on null Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

The NPY_TYPE<DateTime/TimeDelta> specializations would cause empty_array() to create unitless datetime64/timedelta64 arrays via PyArray_SimpleNew, which newer pandas rejects. Use NPY_OBJECT for the empty (0-length) case to match prior behavior. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

AdamGlustein

All comments addressed for me, wait for @robambalu to look as well

robambalu · 2026-04-21T21:02:19Z

I wonder at what point we can remove / cleanup old parquet / arrow numpy processing code does this PR help us get there?
ie

csp/cpp/csp/python/adapters/parquetadapterimpl.cpp

Line 188 in 65d2a59

template< typename CspCType>

https://github.com/Point72/csp/blob/main/cpp/csp/adapters/parquet/DialectGenericListReaderInterface.h

https://github.com/Point72/csp/blob/main/cpp/csp/adapters/parquet/DialectGenericListWriterInterface.h

arhamchopra · 2026-04-21T21:13:10Z

I wonder at what point we can remove / cleanup old parquet / arrow numpy processing code does this PR help us get there? ie

csp/cpp/csp/python/adapters/parquetadapterimpl.cpp

Line 188 in 65d2a59

template< typename CspCType>

https://github.com/Point72/csp/blob/main/cpp/csp/adapters/parquet/DialectGenericListReaderInterface.h

https://github.com/Point72/csp/blob/main/cpp/csp/adapters/parquet/DialectGenericListWriterInterface.h

Yeah, so this PR adds support for lists in the (csp.Struct <-> Arrow RB) conversion and #698 added support for the scalar types. Once these both are merged we can remove a lot of the parquet conversion code and use the conversion logic in the arrow code (which is simpler)

Replace import_array1(false) with import_array() in a void* returning static initializer with AcquireGIL, matching NumpyConversions.cpp pattern. Remove unnecessary registerCallback wrapper. Fix formatting: createNumpyNDArrayReader call and one-liner else. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

arhamchopra added 6 commits April 16, 2026 15:46

Tests for numpy array Arrow record batch conversion

528d4c2

Add tests for numpy ndarray fields in record_batches_to_struct and struct_to_record_batches: 1D/2D arrays, various dtypes, round-trip conversion, edge cases. Signed-off-by: Arham Chopra <arham.chopra@cubistsystematic.com>

arhamchopra force-pushed the ac/arrow_nodes_numpy_support branch from bcd4e71 to e850962 Compare April 16, 2026 20:18

arhamchopra marked this pull request as ready for review April 16, 2026 23:30

arhamchopra requested review from AdamGlustein, alexddobkin, robambalu and svatasoiu as code owners April 16, 2026 23:30

AdamGlustein requested changes Apr 21, 2026

View reviewed changes

arhamchopra added 3 commits April 21, 2026 13:52

arhamchopra requested a review from AdamGlustein April 21, 2026 17:53

AdamGlustein reviewed Apr 21, 2026

View reviewed changes

Comment thread cpp/csp/python/adapters/ArrowNumpyListWriter.h Outdated

arhamchopra force-pushed the ac/arrow_nodes_numpy_support branch from 5d6d1e0 to 45624f2 Compare April 21, 2026 18:30

arhamchopra requested a review from AdamGlustein April 21, 2026 18:30

arhamchopra force-pushed the ac/arrow_nodes_numpy_support branch from 45624f2 to df9d2f2 Compare April 21, 2026 18:57

AdamGlustein previously approved these changes Apr 21, 2026

View reviewed changes

AdamGlustein reviewed Apr 21, 2026

View reviewed changes

Comment thread cpp/csp/python/adapters/ArrowCppNodes.cpp Outdated

Comment thread cpp/csp/python/adapters/ArrowCppNodes.cpp Outdated

Comment thread cpp/csp/python/adapters/ArrowCppNodes.cpp

Comment thread cpp/csp/python/adapters/ArrowCppNodes.cpp Outdated

arhamchopra mentioned this pull request Apr 21, 2026

Add numpy array support for struct→RecordBatch conversion #702

Closed

arhamchopra dismissed AdamGlustein’s stale review via 6168414 April 22, 2026 15:54

arhamchopra requested a review from AdamGlustein April 22, 2026 15:55

AdamGlustein approved these changes Apr 22, 2026

View reviewed changes

arhamchopra merged commit 72c224b into main Apr 22, 2026
25 checks passed

arhamchopra deleted the ac/arrow_nodes_numpy_support branch April 22, 2026 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for lists to Arrow <-> csp.Struct conversion nodes#701

Add support for lists to Arrow <-> csp.Struct conversion nodes#701
arhamchopra merged 11 commits into
mainfrom
ac/arrow_nodes_numpy_support

arhamchopra commented Apr 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdamGlustein left a comment

Uh oh!

robambalu commented Apr 21, 2026 •

edited

Loading

Uh oh!

arhamchopra commented Apr 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

arhamchopra commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Arrow RecordBatch ↔ csp.Struct: Numpy Array Support

Summary

What's New

C++ Layer

Python Layer

Element Types Supported

Architecture

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AdamGlustein left a comment

Choose a reason for hiding this comment

Uh oh!

robambalu commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arhamchopra commented Apr 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arhamchopra commented Apr 16, 2026 •

edited

Loading

robambalu commented Apr 21, 2026 •

edited

Loading