Skip to content

issue #592: exclude BLink from estimatedInMemoryBytes() to fix spurious back-pressure - #594

Merged
eolivelli merged 2 commits into
masterfrom
issue-592-is-blink-snapshot-overcounts-segment-mem
May 19, 2026
Merged

eolivelli merged 2 commits into
masterfrom
issue-592-is-blink-snapshot-overcounts-segment-mem

Conversation

@eolivelli

Copy link
Copy Markdown
Owner

Fixes #592.

Root Cause

VectorSegment.estimatedInMemoryBytes() included onDiskPkToNode.getUsedMemory() — a snapshot of the BLink pk-to-ordinal tree's loaded-page memory taken once at segment registration (when all BLink pages are fully in-memory after a checkpoint flush). The MemoryManager independently evicts BLink pages afterward but the snapshot in onDiskSegmentsEstimatedMemoryBytes is never decremented. This causes a ~128 B/vector permanent phantom cost that makes waitForMemoryPressureRelief() stall all apply workers at ~14M vectors under default settings (33% × 8 GiB = 2.64 GiB limit).

Changes

  • VectorSegment.java: Removed p2n.getUsedMemory() from estimatedInMemoryBytes(). Updated Javadoc to explain that BLink is intentionally excluded (managed by MemoryManager independently; including it was double-counting that caused issue [IS] BLink snapshot in estimatedInMemoryBytes() permanently over-counts segment memory, causing spurious back-pressure at scale #592).
  • PersistentVectorStore.java: Updated 6 comments that mentioned BLink being included in the on-disk segment estimate (onDiskSegmentsEstimatedMemoryBytes field, estimatedMemoryUsageBytes(), getLiveShardMemoryBytes(), getOnDiskSegmentsEstimatedMemoryBytes(), registerSegmentMemoryEstimate(), sumOnDiskSegmentMemoryBytesByIterationForTesting()).

Tests

  • PersistentVectorStoreEstimatedMemoryCounterTest#testOnDiskCounterExcludesBLinkPageCache (new): inserts 400 vectors, checkpoints, and asserts that getOnDiskSegmentsEstimatedMemoryBytes() is ≥ raw array lower bound (12 B/vector) and ≤ 48 B/vector — a threshold comfortably below the BLink contribution of 128 B/vector. If BLink is re-introduced, this test fails immediately.
  • PersistentVectorStoreEstimatedMemoryCounterTest: all 8 existing tests still pass; updated class/method Javadoc to reflect that the "clean lifecycle point" caveat is now moot (BLink is no longer counted so no drift can occur).
  • PersistentVectorStoreBackpressureLivelockTest: both existing tests pass unchanged; updated calibration comment in testIngestionCompletesUnderMemoryPressure to reflect the corrected on-disk estimate (~12 B/vector instead of ~120 B/vector).

Impact

Per-vector on-disk estimate: ~144 B → ~16 B (for BIGINT PKs). The global back-pressure limit is now accurate across the full 50M-vector range without requiring indexing.memory.vector.percentage workaround bumps.

🤖 Implemented by the pr-worker agent.

eolivelli added 2 commits May 19, 2026 07:23
… to fix spurious back-pressure

The BLink pk-to-ordinal tree was included in the on-disk segment memory
estimate (getUsedMemory() snapshot at registration time), but the
MemoryManager independently evicts BLink pages after registration without
updating the counter.  This caused onDiskSegmentsEstimatedMemoryBytes to
remain permanently at the peak value (~128 B/vector phantom cost), making
waitForMemoryPressureRelief() stall all apply workers at ~14M vectors — far
below the true memory limit.

Fix (Option A from the issue): remove BLink.getUsedMemory() from
VectorSegment.estimatedInMemoryBytes().  The BLink page budget is already
bounded by the MemoryManager's index-page replacement policy, so including
it in the global back-pressure counter is double-counting by design.

Per-vector on-disk estimate drops from ~144 B to ~16 B (BIGINT PKs),
making the estimator accurate across the full 50M-vector range without
any artificial limit bumps.

Add regression test testOnDiskCounterExcludesBLinkPageCache that verifies
the per-vector cost is <= 48 B after checkpoint (well below BLink's 128 B).
@eolivelli
eolivelli merged commit 9fc654d into master May 19, 2026
8 checks passed
@eolivelli
eolivelli deleted the issue-592-is-blink-snapshot-overcounts-segment-mem branch May 19, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[IS] BLink snapshot in estimatedInMemoryBytes() permanently over-counts segment memory, causing spurious back-pressure at scale

1 participant