Skip to content

Checkpoint performs standalone full compaction before dump, duplicating scan work on large graphs #701

Description

@zhanglei1949

Problem

Checkpoint currently runs a full graph compaction before dumping:

graph.Compact(MAX_TIMESTAMP);
graph.Dump(ckp, reopen);

On LDBC SNB SF1000 this standalone compaction took about 52 minutes before dump even started.

This duplicates work because dump immediately scans the same large vertex/edge structures again to write checkpoint files. For SF1000, the expensive parts include:

  • out/in CSR compaction for every edge table
  • timestamp reset for mutable CSR entries
  • optional neighbor sorting for sort_key_for_nbr edge tables
  • very large edges such as HASCREATOR, HASMEMBER, LIKES, HASTAG, REPLYOF

Current behavior

Checkpoint has two full passes:

  1. PropertyGraph::Compact()
  2. module/CSR/column dump

This is costly for tens of billions of edges.

Expected direction

Fold checkpoint compaction into dump.

During checkpoint dump:

  • filter deleted edges/vertices
  • reset persisted timestamps
  • apply required neighbor sorting
  • write compact checkpoint data directly

Keep non-checkpoint compaction paths unchanged for background/manual compaction.

Acceptance criteria

  • NeugDB::createCheckpoint() no longer calls standalone graph.Compact(MAX_TIMESTAMP)
  • loader checkpoint path also avoids separate graph_.Compact(...)
  • checkpoint output remains equivalent after reopen
  • existing manual/background compaction behavior remains intact
  • tests cover deleted edge filtering, timestamp reset, and sorted-neighbor checkpoint persistence

Metadata

Metadata

Assignees

No one assigned

    Labels

    storeStorage layer

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions