Repository navigation
issue #592: exclude BLink from estimatedInMemoryBytes() to fix spurious back-pressure - #594
Merged
eolivelli merged 2 commits intoMay 19, 2026
Conversation
… to fix spurious back-pressure The BLink pk-to-ordinal tree was included in the on-disk segment memory estimate (getUsedMemory() snapshot at registration time), but the MemoryManager independently evicts BLink pages after registration without updating the counter. This caused onDiskSegmentsEstimatedMemoryBytes to remain permanently at the peak value (~128 B/vector phantom cost), making waitForMemoryPressureRelief() stall all apply workers at ~14M vectors — far below the true memory limit. Fix (Option A from the issue): remove BLink.getUsedMemory() from VectorSegment.estimatedInMemoryBytes(). The BLink page budget is already bounded by the MemoryManager's index-page replacement policy, so including it in the global back-pressure counter is double-counting by design. Per-vector on-disk estimate drops from ~144 B to ~16 B (BIGINT PKs), making the estimator accurate across the full 50M-vector range without any artificial limit bumps. Add regression test testOnDiskCounterExcludesBLinkPageCache that verifies the per-vector cost is <= 48 B after checkpoint (well below BLink's 128 B).
…doc with key widths
This was referenced May 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #592.
Root Cause
VectorSegment.estimatedInMemoryBytes()includedonDiskPkToNode.getUsedMemory()— a snapshot of the BLink pk-to-ordinal tree's loaded-page memory taken once at segment registration (when all BLink pages are fully in-memory after a checkpoint flush). The MemoryManager independently evicts BLink pages afterward but the snapshot inonDiskSegmentsEstimatedMemoryBytesis never decremented. This causes a ~128 B/vector permanent phantom cost that makeswaitForMemoryPressureRelief()stall all apply workers at ~14M vectors under default settings (33% × 8 GiB = 2.64 GiB limit).Changes
VectorSegment.java: Removedp2n.getUsedMemory()fromestimatedInMemoryBytes(). Updated Javadoc to explain that BLink is intentionally excluded (managed by MemoryManager independently; including it was double-counting that caused issue [IS] BLink snapshot in estimatedInMemoryBytes() permanently over-counts segment memory, causing spurious back-pressure at scale #592).PersistentVectorStore.java: Updated 6 comments that mentioned BLink being included in the on-disk segment estimate (onDiskSegmentsEstimatedMemoryBytesfield,estimatedMemoryUsageBytes(),getLiveShardMemoryBytes(),getOnDiskSegmentsEstimatedMemoryBytes(),registerSegmentMemoryEstimate(),sumOnDiskSegmentMemoryBytesByIterationForTesting()).Tests
PersistentVectorStoreEstimatedMemoryCounterTest#testOnDiskCounterExcludesBLinkPageCache(new): inserts 400 vectors, checkpoints, and asserts thatgetOnDiskSegmentsEstimatedMemoryBytes()is ≥ raw array lower bound (12 B/vector) and ≤ 48 B/vector — a threshold comfortably below the BLink contribution of 128 B/vector. If BLink is re-introduced, this test fails immediately.PersistentVectorStoreEstimatedMemoryCounterTest: all 8 existing tests still pass; updated class/method Javadoc to reflect that the "clean lifecycle point" caveat is now moot (BLink is no longer counted so no drift can occur).PersistentVectorStoreBackpressureLivelockTest: both existing tests pass unchanged; updated calibration comment intestIngestionCompletesUnderMemoryPressureto reflect the corrected on-disk estimate (~12 B/vector instead of ~120 B/vector).Impact
Per-vector on-disk estimate: ~144 B → ~16 B (for BIGINT PKs). The global back-pressure limit is now accurate across the full 50M-vector range without requiring
indexing.memory.vector.percentageworkaround bumps.🤖 Implemented by the
pr-workeragent.