Problem
When loading LDBC SNB SF1000, checkpoint dump is spending a very long time after compaction:
00:50:49 Compaction completed
00:50:49 Creating checkpoint
02:59:27 Checkpoint manifest saved
So dump took about 2h9m.
Code inspection shows PropertyGraph::Dump() expands storage before persisting:
- vertex tables are expanded to
v_size + 25%
- unbundled edge property tables are expanded to about
e_size + 20%
- expanded capacities are then dumped as checkpoint files
For SF1000 this is especially expensive because COMMENT has about 2.3B rows, POST has about 340M rows, and edges are about 19B total. The extra reserve capacity becomes extra MD5 scanning and extra checkpoint I/O.
Current behavior
Dump writes capacity-sized files instead of compact logical data plus post-dump reserve.
Relevant areas:
PropertyGraph::Dump()
VertexTable::EnsureCapacity()
EdgeTable::EnsureCapacity()
MMapContainer::Dump()
- string column dump path
Expected direction
Do not expand the live graph before checkpoint dump.
Instead:
- dump logical/live data first
- finalize checkpoint files with a 125% reserve where the format safely supports it
- for structured formats like CSR/indexer/timestamp, reserve must be reflected in metadata/sentinel initialization, not only raw file
ftruncate
Acceptance criteria
- Checkpoint no longer calls pre-dump
EnsureCapacity(v_size + 25%)
- Checkpoint files still reopen correctly
- Reopened DB can continue inserting after checkpoint
- SF1000 checkpoint writes significantly less data
Problem
When loading LDBC SNB SF1000, checkpoint dump is spending a very long time after compaction:
00:50:49Compaction completed00:50:49Creating checkpoint02:59:27Checkpoint manifest savedSo dump took about 2h9m.
Code inspection shows
PropertyGraph::Dump()expands storage before persisting:v_size + 25%e_size + 20%For SF1000 this is especially expensive because COMMENT has about 2.3B rows, POST has about 340M rows, and edges are about 19B total. The extra reserve capacity becomes extra MD5 scanning and extra checkpoint I/O.
Current behavior
Dump writes capacity-sized files instead of compact logical data plus post-dump reserve.
Relevant areas:
PropertyGraph::Dump()VertexTable::EnsureCapacity()EdgeTable::EnsureCapacity()MMapContainer::Dump()Expected direction
Do not expand the live graph before checkpoint dump.
Instead:
ftruncateAcceptance criteria
EnsureCapacity(v_size + 25%)