Problem
Checkpoint currently runs a full graph compaction before dumping:
graph.Compact(MAX_TIMESTAMP);
graph.Dump(ckp, reopen);
On LDBC SNB SF1000 this standalone compaction took about 52 minutes before dump even started.
This duplicates work because dump immediately scans the same large vertex/edge structures again to write checkpoint files. For SF1000, the expensive parts include:
- out/in CSR compaction for every edge table
- timestamp reset for mutable CSR entries
- optional neighbor sorting for
sort_key_for_nbr edge tables
- very large edges such as HASCREATOR, HASMEMBER, LIKES, HASTAG, REPLYOF
Current behavior
Checkpoint has two full passes:
PropertyGraph::Compact()
- module/CSR/column dump
This is costly for tens of billions of edges.
Expected direction
Fold checkpoint compaction into dump.
During checkpoint dump:
- filter deleted edges/vertices
- reset persisted timestamps
- apply required neighbor sorting
- write compact checkpoint data directly
Keep non-checkpoint compaction paths unchanged for background/manual compaction.
Acceptance criteria
NeugDB::createCheckpoint() no longer calls standalone graph.Compact(MAX_TIMESTAMP)
- loader checkpoint path also avoids separate
graph_.Compact(...)
- checkpoint output remains equivalent after reopen
- existing manual/background compaction behavior remains intact
- tests cover deleted edge filtering, timestamp reset, and sorted-neighbor checkpoint persistence
Problem
Checkpoint currently runs a full graph compaction before dumping:
graph.Compact(MAX_TIMESTAMP); graph.Dump(ckp, reopen);On LDBC SNB SF1000 this standalone compaction took about 52 minutes before dump even started.
This duplicates work because dump immediately scans the same large vertex/edge structures again to write checkpoint files. For SF1000, the expensive parts include:
sort_key_for_nbredge tablesCurrent behavior
Checkpoint has two full passes:
PropertyGraph::Compact()This is costly for tens of billions of edges.
Expected direction
Fold checkpoint compaction into dump.
During checkpoint dump:
Keep non-checkpoint compaction paths unchanged for background/manual compaction.
Acceptance criteria
NeugDB::createCheckpoint()no longer calls standalonegraph.Compact(MAX_TIMESTAMP)graph_.Compact(...)