Is your feature request related to a problem? Please describe.
Currently, bulk loading vertices/edges from CSV via COPY reads all batches into memory before insertion, and the bulk load process runs single-threaded. This leads to high memory usage and limits load throughput for large CSV datasets.
Describe the solution you'd like
- Fuse
DataSource and BatchInsert so batches are consumed as they are produced, avoiding buffering the entire input in memory.
- Parallelize the bulk loading process across multiple threads to improve load throughput.
Describe alternatives you've considered
Keeping the current single-threaded, fully-buffered load path, which is simpler but does not scale for large datasets.
Additional context
Related to #734, which implements this optimization by refactoring csv_read_function, batch_insert_edge/batch_insert_vertex, loader_utils, and related storage/graph interfaces.
Is your feature request related to a problem? Please describe.
Currently, bulk loading vertices/edges from CSV via
COPYreads all batches into memory before insertion, and the bulk load process runs single-threaded. This leads to high memory usage and limits load throughput for large CSV datasets.Describe the solution you'd like
DataSourceandBatchInsertso batches are consumed as they are produced, avoiding buffering the entire input in memory.Describe alternatives you've considered
Keeping the current single-threaded, fully-buffered load path, which is simpler but does not scale for large datasets.
Additional context
Related to #734, which implements this optimization by refactoring
csv_read_function,batch_insert_edge/batch_insert_vertex,loader_utils, and related storage/graph interfaces.