Problem
Commit a281c0dd (replace Arrow CSV/JSON IO with native reader framework) changed insert_primary_keys from direct typed array access to virtual dispatch via get_elem(), adding per-vertex overhead.
Root Cause
In vertex_table.h:268-287, insert_primary_keys now goes through IContextColumn::get_elem() for every vertex:
template <typename PK_T>
std::vector<vid_t> insert_primary_keys(
const std::shared_ptr<IContextColumn>& pk_col) {
size_t row_num = pk_col->size();
std::vector<vid_t> vids;
vids.resize(row_num);
for (size_t j = 0; j < row_num; ++j) {
auto oid = pk_col->get_elem(j); // ← virtual dispatch + Value construction
if (NEUG_UNLIKELY(indexer_->get_index(oid, vids[j]))) {
...
}
vids[j] = insert_vertex_pk(oid, 0, is_string);
}
return vids;
}get_elem(j) is a virtual call that constructs a Value object for every row. For int64 PKs (the common LDBC case), this means a virtual call + Value::CreateValue<int64_t>() per vertex.
v0.1.3 comparison
The old Arrow-based code directly accessed the typed array:
auto casted_array = std::static_pointer_cast<arrow_array_t>(primary_key_column);
for (size_t j = 0; j < row_num; ++j) {
auto oid = execution::Value::CreateValue<PK_T>(casted_array->Value(j));
...
}No virtual dispatch — direct memory read from the Arrow array.
Same pattern as #694
This is the same class of overhead that was already fixed for set_properties_from_context_column in #694 (b0905a1). The fix pattern is identical: cast to ValueColumn<PK_T> and access data() directly.
Impact
Estimated ~500s of the remaining ~4108s regression on SF1000.
Proposed Fix
Cast pk_col to ValueColumn<PK_T> and access data() directly, falling back to get_elem() only for non-standard column types:
auto value_col = std::dynamic_pointer_cast<ValueColumn<PK_T>>(pk_col);
if (value_col) {
const auto& data = value_col->data();
for (size_t j = 0; j < row_num; ++j) {
auto oid = Value::CreateValue<PK_T>(data[j]);
// ...
}
} else {
// fallback to get_elem()
}Benchmark
v0.1.3: 15166.20s
main (b0905a1): 19273.97s (+4108s remaining)
References
Problem
Commit
a281c0dd(replace Arrow CSV/JSON IO with native reader framework) changedinsert_primary_keysfrom direct typed array access to virtual dispatch viaget_elem(), adding per-vertex overhead.Root Cause
In
vertex_table.h:268-287,insert_primary_keysnow goes throughIContextColumn::get_elem()for every vertex:get_elem(j)is a virtual call that constructs aValueobject for every row. For int64 PKs (the common LDBC case), this means a virtual call +Value::CreateValue<int64_t>()per vertex.v0.1.3 comparison
The old Arrow-based code directly accessed the typed array:
No virtual dispatch — direct memory read from the Arrow array.
Same pattern as #694
This is the same class of overhead that was already fixed for
set_properties_from_context_columnin #694 (b0905a1). The fix pattern is identical: cast toValueColumn<PK_T>and accessdata()directly.Impact
Estimated ~500s of the remaining ~4108s regression on SF1000.
Proposed Fix
Cast
pk_coltoValueColumn<PK_T>and accessdata()directly, falling back toget_elem()only for non-standard column types:Benchmark
References
a281c0dd1d467cc2ca9a584dd90fad9ecfd34242include/neug/storages/graph/vertex_table.h