feat(dataframe): batch column write API SetItemColumnFloat64 across three engines - #163
Merged
Conversation
…hree engines (#157) Write-side counterpart of the batch column read API (#155) and typed columns (#156): operators hand a whole float64/double column to the frame in one call instead of N per-element SetItem records. API (per engine): - pine-go: OperatorOutput.SetItemColumnFloat64(field, []float64) - pine-java: OperatorOutput.setItemColumnDouble(field, double[]) - pine-cpp: OperatorOutput::set_item_column_double(field, vector<double>) Semantics (identical across engines, pinned by tests): - Applied at stage 2b, AFTER per-element item writes — a column write to the same field deterministically wins. - Length must equal the frame's item count (whole column or nothing); mismatch error message is byte-identical across engines. - NaN/Inf batch validation produces the same first-error message as the per-element path (item[i] write: field "f": NaN/Inf ...). - Column-store frames ADOPT the array as the column's backing storage (zero-copy, all slots present); row-store frames scatter per row in one lock window. - Counts as SetItem for OperatorType.ValidateOutput; folds into the item_writes debug snapshot; data_parallel shard merges fold column writes into offset-adjusted per-element writes. transform_normalize now writes its result column via the batch API in all three engines. pine-go transform-heavy 1000 A/B (vs master): column 0.92ms → 0.70ms (-24%), 12.1k → 4.1k allocs/op (-66%), bytes -11%; row-store also drops 15.1k → didn't regress (scatter path, boxing unavoidable). Verification: three-engine unit tests green (Go all, Java 263, C++ 225/110245 assertions); cross-validate sections 3/4/5 (95/95 exec + column-store parity, 30/30 error parity, Go vs Java vs C++); differential fuzz 120 rounds seed=42 zero divergence (row 73/0, column 47/0).
Contributor
🔍 PR 审查
三引擎批量列写 API 实现一致、语义对齐、测试覆盖充分,未发现阻塞或重要问题。逐一核对了写入路径、并行合并、debug 快照、类型校验四处消费点,跨引擎行为可对齐。
|
- operator-contract: add SetItemColumnFloat64 batch write contract (three-engine method table, stage-2b ordering, whole-column-or-nothing length check, NaN batch validation message parity, adopt vs scatter ownership transfer, ValidateOutput/snapshot/parallel-merge folding) - column-vs-row reflection: add phase-3 implementation record with profiling gate result (~24% write-side allocations), benchmark (column allocs -66%), and lessons (any-boxing tax in Go batch paths, six-consumer checklist for new write record kinds, CI hang triage) - perf-evolution-roadmap: mark write-side batch API landed; read+write batch API loop closed - index: sync entries
Contributor
🔍 PR 增量审查
增量部分为纯文档提交(
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #157.
What
Write-side counterpart of the batch column read API (#155) and typed columns (#156): operators hand a whole float64/double column to the frame in one call instead of N per-element
SetItemrecords.OperatorOutput.SetItemColumnFloat64(field string, vals []float64)OperatorOutput.setItemColumnDouble(String field, double[] vals)OperatorOutput::set_item_column_double(const std::string&, std::vector<double>)Semantics (identical across engines, pinned by tests)
SetItemColumnFloat64 "f" length N does not match item count M.item[i] write: field "f": NaN/Inf is not a valid JSON value.SetItemforOperatorType.ValidateOutput; folds into theitem_writesdebug snapshot ("final state" semantics preserved);data_parallelshard merges fold column writes into offset-adjusted per-element writes (shards cover windows and cannot adopt).transform_normalizenow emits its result column via the batch API in all three engines.Why (profiling attribution, from #157)
Write side accounted for ~24% of transform-heavy allocations: per-element
SetItemboxing (137MB),newColumnForValuerebuild churn (154MB), ItemWrite records (14MB). The >15% gate passed, so the typed pass-through write API is justified.Benchmark (pine-go, StorageAB TransformHeavy 1000, count=5, vs master)
Row-store unaffected in allocs (15.1k, scatter path — boxing into maps is unavoidable there).
Verification
make allgreen (fmt-check, three-engine lint, three-engine tests, codegen-check)