Skip to content

Checkpoint dump expands storage before persisting, causing excessive SF1000 I/O #700

Description

@zhanglei1949

Problem

When loading LDBC SNB SF1000, checkpoint dump is spending a very long time after compaction:

  • 00:50:49 Compaction completed
  • 00:50:49 Creating checkpoint
  • 02:59:27 Checkpoint manifest saved

So dump took about 2h9m.

Code inspection shows PropertyGraph::Dump() expands storage before persisting:

  • vertex tables are expanded to v_size + 25%
  • unbundled edge property tables are expanded to about e_size + 20%
  • expanded capacities are then dumped as checkpoint files

For SF1000 this is especially expensive because COMMENT has about 2.3B rows, POST has about 340M rows, and edges are about 19B total. The extra reserve capacity becomes extra MD5 scanning and extra checkpoint I/O.

Current behavior

Dump writes capacity-sized files instead of compact logical data plus post-dump reserve.

Relevant areas:

  • PropertyGraph::Dump()
  • VertexTable::EnsureCapacity()
  • EdgeTable::EnsureCapacity()
  • MMapContainer::Dump()
  • string column dump path

Expected direction

Do not expand the live graph before checkpoint dump.

Instead:

  1. dump logical/live data first
  2. finalize checkpoint files with a 125% reserve where the format safely supports it
  3. for structured formats like CSR/indexer/timestamp, reserve must be reflected in metadata/sentinel initialization, not only raw file ftruncate

Acceptance criteria

  • Checkpoint no longer calls pre-dump EnsureCapacity(v_size + 25%)
  • Checkpoint files still reopen correctly
  • Reopened DB can continue inserting after checkpoint
  • SF1000 checkpoint writes significantly less data

Metadata

Metadata

Assignees

No one assigned

    Labels

    storeStorage layer

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions