Skip to content

[Feature] Optimize bulk load performance for CSV COPY inputs #735

Description

@zhanglei1949

Is your feature request related to a problem? Please describe.
Currently, bulk loading vertices/edges from CSV via COPY reads all batches into memory before insertion, and the bulk load process runs single-threaded. This leads to high memory usage and limits load throughput for large CSV datasets.

Describe the solution you'd like

  • Fuse DataSource and BatchInsert so batches are consumed as they are produced, avoiding buffering the entire input in memory.
  • Parallelize the bulk loading process across multiple threads to improve load throughput.

Describe alternatives you've considered
Keeping the current single-threaded, fully-buffered load path, which is simpler but does not scale for large datasets.

Additional context
Related to #734, which implements this optimization by refactoring csv_read_function, batch_insert_edge/batch_insert_vertex, loader_utils, and related storage/graph interfaces.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestexecutorExecution engineperformancePerformance issue or improvementstoreStorage layer

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions