Context
PR #155 removed the per-element interface tax (lock + map lookup per Item() call) via the batch column access API (ItemColumn / itemColumn / item_column), and made ColumnFrame's hot paths column-major. Investigation data (llmdoc/memory/reflections/column-vs-row-parity-investigation.md): a typed []float64 scan (257ns) is now nearly identical to an ideal []any scan (262ns) — meaning the remaining gains from typed columns must come from eliminating boxing, and the prerequisite interface surface is in place.
Current state
- pine-go ColumnFrame columns are
map[string][]any — every element is a boxed interface pointing at a scattered heap object; pine-java uses Object[] (same shape). pine-cpp already has a typed Column hierarchy (DoubleColumn / Int64Column / BoolColumn / StringColumn / JsonColumn).
- Operators pay per-element type assertions on every scan (e.g.
toFloat64(raw[i]) in transform_normalize).
Proposal
Typed flat columns for Go/Java mirroring pine-cpp's Column model:
[]float64 / []int64 / []bool + presence bitmap + string arena
- Typed batch accessors on OperatorInput:
ItemColumnFloat64(field) ([]float64, ok) etc.; existing ItemColumn degrades to the boxing compatibility layer
- Type promotion rules on mixed writes should follow pine-cpp's write_item_field_locked (first-value type pick, promote to Json on mismatch / present-nil)
Payoffs
- Native operators drop boxing + per-element type assertions entirely
- Cache-friendly scans (contiguous primitive arrays)
- Flat columns can later map zero-copy into VM linear memory / arena (roadmap step 2 synergy)
References
- llmdoc/memory/decisions/perf-evolution-roadmap.md (step 1)
- llmdoc/memory/reflections/column-vs-row-parity-investigation.md
- Batch API contract: llmdoc/reference/operator-contract.md
Context
PR #155 removed the per-element interface tax (lock + map lookup per
Item()call) via the batch column access API (ItemColumn/itemColumn/item_column), and made ColumnFrame's hot paths column-major. Investigation data (llmdoc/memory/reflections/column-vs-row-parity-investigation.md): a typed[]float64scan (257ns) is now nearly identical to an ideal[]anyscan (262ns) — meaning the remaining gains from typed columns must come from eliminating boxing, and the prerequisite interface surface is in place.Current state
map[string][]any— every element is a boxed interface pointing at a scattered heap object; pine-java usesObject[](same shape). pine-cpp already has a typed Column hierarchy (DoubleColumn / Int64Column / BoolColumn / StringColumn / JsonColumn).toFloat64(raw[i])in transform_normalize).Proposal
Typed flat columns for Go/Java mirroring pine-cpp's Column model:
[]float64/[]int64/[]bool+ presence bitmap + string arenaItemColumnFloat64(field) ([]float64, ok)etc.; existingItemColumndegrades to the boxing compatibility layerPayoffs
References