Load performance regression: parse_endpoint_column get_elem() virtual dispatch overhead

[zhanglei1949]

Problem

Commit a281c0dd (replace Arrow CSV/JSON IO with native reader framework) changed parse_endpoint_column from direct typed array access to virtual dispatch via get_elem(), adding per-edge overhead for both src and dst endpoint lookups.

Root Cause

In edge_table.cc:184-190, parse_endpoint_column now goes through IContextColumn::get_elem() for every edge endpoint:

static void parse_endpoint_column(const IndexerType& indexer,
                                  const std::shared_ptr<IContextColumn>& col,
                                  std::vector<vid_t>& lids) {
  for (size_t i = 0; i < col->size(); ++i) {
    auto val = col->get_elem(i);        // ← virtual dispatch + Value construction
    auto vid = indexer.get_index(val);  // ← Value-based index lookup
    lids.push_back(vid);
  }
}

This is called twice per edge (src + dst), so the overhead is 2 × edge_count virtual dispatches + Value constructions.

v0.1.3 comparison

The old Arrow-based code dispatched on Arrow type and directly accessed the typed array:

if (array->type()->Equals(arrow::int64())) {
    auto casted = std::static_pointer_cast<arrow::Int64Array>(array);
    for (int64_t i = 0; i < casted->length(); ++i) {
        auto vid = indexer.get_index(execution::Value::INT64(casted->Value(i)));
        lids.push_back(vid);
    }
}

No virtual dispatch per element — direct memory read from the Arrow array.

Same pattern as #694

This is the same class of overhead that was already fixed for set_properties_from_context_column in #694 (b0905a1). The fix pattern is identical: cast to ValueColumn<T> and access data() directly.

Impact

Estimated ~500-1000s of the remaining ~4108s regression on SF1000. Impact is higher than vertex PK path because edges have two endpoint lookups per row.

Proposed Fix

Cast col to ValueColumn<T> based on the indexer's key type and access data() directly:

// Dispatch based on column type, then access data() directly
if (auto vc = std::dynamic_pointer_cast<ValueColumn<int64_t>>(col)) {
    const auto& data = vc->data();
    for (size_t i = 0; i < data.size(); ++i) {
        auto vid = indexer.get_index(Value::INT64(data[i]));
        lids.push_back(vid);
    }
} else if (auto vc = std::dynamic_pointer_cast<ValueColumn<std::string>>(col)) {
    // handle string PKs
} else {
    // fallback to get_elem()
}

Benchmark

v0.1.3:          15166.20s
main (b0905a1):  19273.97s  (+4108s remaining)

References

• Introducing commit: a281c0dd1d467cc2ca9a584dd90fad9ecfd34242
• Related fix pattern: refactor: Avoid unnecessary value cast cost when setting properties from context column #694 (b0905a1)
• Related issues: Load performance regression: count_rows() causes full duplicate CSV parse #702, Load performance regression: per-field intermediate std::string heap allocation in CSV parsing #703, Load performance regression: insert_primary_keys get_elem() virtual dispatch overhead #704
• Affected files: src/storages/graph/edge_table.cc

[github-actions]

🤖 Auto-triage: linked as sub-issue of #221 — Improve & Extend COPY FROM: schema, typing, performance, options (confidence: high).

Reason: The issue is a performance regression in the COPY/LOAD FROM ingestion path (parse_endpoint_column) caused by virtual dispatch overhead, which falls squarely under #221's scope of COPY FROM performance optimization.

Auto-applied: type → Task; labels → executor

If this is wrong, please unlink and re-assign — your correction improves this bot.