Skip to content

feat(dataframe): batch column write API SetItemColumnFloat64 across three engines - #163

Merged
Liam0205 merged 2 commits into
masterfrom
feat/batch-column-write
Jul 8, 2026
Merged

feat(dataframe): batch column write API SetItemColumnFloat64 across three engines#163
Liam0205 merged 2 commits into
masterfrom
feat/batch-column-write

Conversation

@Liam0205

@Liam0205 Liam0205 commented Jul 8, 2026

Copy link
Copy Markdown
Owner

Closes #157.

What

Write-side counterpart of the batch column read API (#155) and typed columns (#156): operators hand a whole float64/double column to the frame in one call instead of N per-element SetItem records.

Engine API
pine-go OperatorOutput.SetItemColumnFloat64(field string, vals []float64)
pine-java OperatorOutput.setItemColumnDouble(String field, double[] vals)
pine-cpp OperatorOutput::set_item_column_double(const std::string&, std::vector<double>)

Semantics (identical across engines, pinned by tests)

  • Applied at stage 2b, after per-element item writes — a column write to the same field deterministically wins.
  • Length must equal the frame's item count (whole column or nothing); mismatch error message byte-identical across engines: SetItemColumnFloat64 "f" length N does not match item count M.
  • NaN/Inf batch validation produces the same first-error message as the per-element path: item[i] write: field "f": NaN/Inf is not a valid JSON value.
  • Column-store frames adopt the array as the column's backing storage (zero-copy, all slots present, typed-fast-path readable); row-store frames scatter per row in one lock window.
  • Counts as SetItem for OperatorType.ValidateOutput; folds into the item_writes debug snapshot ("final state" semantics preserved); data_parallel shard merges fold column writes into offset-adjusted per-element writes (shards cover windows and cannot adopt).

transform_normalize now emits its result column via the batch API in all three engines.

Why (profiling attribution, from #157)

Write side accounted for ~24% of transform-heavy allocations: per-element SetItem boxing (137MB), newColumnForValue rebuild churn (154MB), ItemWrite records (14MB). The >15% gate passed, so the typed pass-through write API is justified.

Benchmark (pine-go, StorageAB TransformHeavy 1000, count=5, vs master)

master this PR Δ
column ns/op ~0.92ms ~0.70ms -24%
column allocs/op 12,134 4,141 -66%
column B/op 641KB 572KB -11%

Row-store unaffected in allocs (15.1k, scatter path — boxing into maps is unavoidable there).

Verification

  • Three-engine unit tests green: Go full suite, Java 263 tests, C++ 225 cases / 110,245 assertions
  • New dedicated batch-write test files per engine: adopt/scatter both modes, length mismatch, NaN message parity, column-write-overrides-per-element, zero-copy adopt aliasing, row/column result parity
  • cross-validate sections 3/4/5: 95/95 exec parity + 95/95 column-store parity + 30/30 error parity (Go vs Java vs C++)
  • differential fuzz 120 rounds seed=42: zero divergence (row 73/0, column 47/0)
  • repo-root make all green (fmt-check, three-engine lint, three-engine tests, codegen-check)

…hree engines (#157)

Write-side counterpart of the batch column read API (#155) and typed
columns (#156): operators hand a whole float64/double column to the
frame in one call instead of N per-element SetItem records.

API (per engine):
- pine-go:   OperatorOutput.SetItemColumnFloat64(field, []float64)
- pine-java: OperatorOutput.setItemColumnDouble(field, double[])
- pine-cpp:  OperatorOutput::set_item_column_double(field, vector<double>)

Semantics (identical across engines, pinned by tests):
- Applied at stage 2b, AFTER per-element item writes — a column write
  to the same field deterministically wins.
- Length must equal the frame's item count (whole column or nothing);
  mismatch error message is byte-identical across engines.
- NaN/Inf batch validation produces the same first-error message as
  the per-element path (item[i] write: field "f": NaN/Inf ...).
- Column-store frames ADOPT the array as the column's backing storage
  (zero-copy, all slots present); row-store frames scatter per row in
  one lock window.
- Counts as SetItem for OperatorType.ValidateOutput; folds into the
  item_writes debug snapshot; data_parallel shard merges fold column
  writes into offset-adjusted per-element writes.

transform_normalize now writes its result column via the batch API in
all three engines.

pine-go transform-heavy 1000 A/B (vs master): column 0.92ms → 0.70ms
(-24%), 12.1k → 4.1k allocs/op (-66%), bytes -11%; row-store also
drops 15.1k → didn't regress (scatter path, boxing unavoidable).

Verification: three-engine unit tests green (Go all, Java 263, C++
225/110245 assertions); cross-validate sections 3/4/5 (95/95 exec +
column-store parity, 30/30 error parity, Go vs Java vs C++);
differential fuzz 120 rounds seed=42 zero divergence (row 73/0,
column 47/0).
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔍 PR 审查

项目 结果
结论 ✅ APPROVE
审查截止 8f79cb93c9994c075f0b06dcf6992f7c52ac8a2f

三引擎批量列写 API 实现一致、语义对齐、测试覆盖充分,未发现阻塞或重要问题。逐一核对了写入路径、并行合并、debug 快照、类型校验四处消费点,跨引擎行为可对齐。

✅ 已核对的关键点

  • 消费点全覆盖 (pine-go)GetColumnWritesApplyOutput(行/列存)、mergeOutputs(并行)、snapshotOutputValidateOutputItemWriteMapReset 六处均已处理,无遗漏。
  • C++ move-after-use 顺序安全ColumnFrame::apply_output 通过可变 column_writes()cw.valsstd::move 消费,而 engine.cpp 中执行顺序为 snapshot_output(L927) → validate_output_against_type(L929) → apply_output(L957),快照与校验都在移动之前读取,不存在 use-after-move。
  • NaN/Inf 检查等价:Go math.IsNaN || math.IsInf、Java Double.isNaN || isInfinite、C++ !std::isfinite,三者对 NaN/Inf 判定一致,首个错误消息 byte 级对齐。
  • 零拷贝 adopt:三引擎列存路径均直接移动/接管底层数组(Go adoptFloat64Column、Java DoubleColumn.adopt、C++ TypedColumn::adopt),C++ 测试 通过原始指针别名验证了真正的零拷贝。
  • DAG 无需改动:调度器的读写危险排序基于计划期静态声明的输出字段,transform_normalize 输出字段不变,运行期改用列写不影响 hazard 边,下游读者仍被正确排序。
  • 顺序语义:列写统一在 stage 2b(逐元素写之后)应用,同字段列写覆盖逐元素写;set_column 前的长度校验(column_frame.cpp L355-L360)保证了 ColumnStorerow_count 不变式。
  • 校验先于变更:行存 scatter 路径先整列扫描 NaN 再写入,单个列写做到全有或全无,不会留下半写状态。

🟢 小问题 (0)

无。transform_normalizerng==0 时依赖新建数组零值初始化,与旧的显式 0.0 写入等价,行为无变化。

- operator-contract: add SetItemColumnFloat64 batch write contract
  (three-engine method table, stage-2b ordering, whole-column-or-nothing
  length check, NaN batch validation message parity, adopt vs scatter
  ownership transfer, ValidateOutput/snapshot/parallel-merge folding)
- column-vs-row reflection: add phase-3 implementation record with
  profiling gate result (~24% write-side allocations), benchmark
  (column allocs -66%), and lessons (any-boxing tax in Go batch paths,
  six-consumer checklist for new write record kinds, CI hang triage)
- perf-evolution-roadmap: mark write-side batch API landed; read+write
  batch API loop closed
- index: sync entries
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔍 PR 增量审查

项目 结果
结论 ✅ APPROVE
审查范围 8f79cb9..7c45018
审查截止 7c45018c80a8cf94aee2a2951a3d1188a2cbe66a

增量部分为纯文档提交(llmdoc/),记录批量列写 API(issue #157 / PR #163)。代码改动已在上一轮审查(截止 8f79cb9)中 APPROVE。本轮逐条核对文档声明与代码实现,事实准确,无阻塞或重要问题。

✅ 已核对的文档事实(4 处均与代码一致)

🟢 小问题 (0)

无。文档三处(index.md / perf-evolution-roadmap.md / column-vs-row-parity-investigation.md)互相引用的契约位置指向 operator-contract.md,链接自洽。

@Liam0205
Liam0205 merged commit a72b621 into master Jul 8, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: batch column write API (SetItemColumn) — profile first, then decide

1 participant