Repository navigation
issue #643: bound IS-local compaction cycle duration via maxOutputNodes cap - #644
Merged
eolivelli merged 2 commits intoMay 23, 2026
Merged
Conversation
…es cap Add `vector.index.compaction.maxOutputNodes`: a per-cycle cap on the sum of live-vector counts across the inputs a single IS-local compaction cycle merges. Bounds per-cycle wall-clock time so cycle duration tracks the per-cycle work budget rather than the total table size. Root cause (confirmed via jvector + HerdDB source inspection): jvector's `OnDiskGraphIndexCompactor.buildBatches` emits `max(40, srcNodes/128)` batches per source segment with no scan of non-input on-disk segments, so total batches scale linearly with the selected input set. The IS-local picker sorts smallest-first by `estimatedSizeBytes` and caps by `maxBytes`/`maxCount`/`maxInputs` — none of these bounds the total live-vector count of the output. As small segments are consumed by earlier cycles and merged into medium segments, the "smallest N" candidates shift upward over time, and each successive cycle merges progressively larger inputs — the 200M-row BIGANN run in the issue shows 3× growth in cycle duration without any change in throughput. The new cap is applied AFTER the trigger / smallest-first / micro-segment / graduation-cap selection (never changes whether a cycle compacts), is bypassed by the micro-segment fast path (those merges are already bounded per input), is tier-scaled (2×/4×/8× at 100/300/500 segments) like `maxInputs` so a flat cap cannot starve the tailer, and always keeps at least 2 inputs for a meaningful merge. Default is `0` (disabled) — purely additive. Tests: `VectorIndexCompactorMaxOutputNodesTest` pins all branches (disabled-by-default, caps at node budget, keeps ≥ 2 when the first two already exceed the cap, no-op short-circuits, micro-segment fast-path bypass, interaction with `maxInputs`, tier-scaling math including overflow safety, and fire/no-fire invariance). Existing `VectorIndexCompactorChooseTest` exercises the 8-arg overload to pin backward compatibility. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…l trade-off Address pr-reviewer follow-ups for #644: 1. New PersistentVectorStoreCompactionMaxOutputNodesTest pins the public API surface — default (0, disabled), positive set, explicit-0 disabling, and the Math.max(0L, ...) clamp on negatives / Long.MIN_VALUE — so the property → engine → store wiring stays correct. Mirrors PersistentVectorStoreCompactionTargetBytesTest. 2. Javadoc on the 9-arg chooseSegmentsToMerge and VECTOR.md now spell out that the cap is on liveCount, while jvector's batch loop scales with maxOrdinal (which includes tombstoned ordinals). For segments with liveCount << maxOrdinal the wall-clock cap underestimates the real work; deletion-heavy workloads should size maxOutputNodes with the expected tombstone fraction in mind. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #643.
Root cause
Confirmed by reading both jvector's
OnDiskGraphIndexCompactor.buildBatchesand HerdDB'sVectorIndexCompactor.chooseSegmentsToMerge:buildBatcheswalks eachsources[s]at level 0 and emitsmax(40, srcNodes/128)batches per source. Total batches = sum across the merge inputs only. There is no scan of non-input on-disk segments — the issue's investigation point TEST CI #3 (super-linear neighbourhood expansion) is refuted. Investigation point TEST CI on better concurrency model #2 is confirmed: batch count grows ~linearly with the selected node count.estimatedSizeBytesand caps bymaxBytes/maxCount/maxInputs. None of these bounds the total live-vector count of the output. As small segments are consumed and merged into medium segments, the "smallest N" candidates shift upward over time, until each input is close to the graduation cap (8 GiB). Cycle wall-clock time grows with the table size — exactly as the 200M-row BIGANN run in the issue shows (3× growth in cycle duration, stable throughput).Summary
vector.index.compaction.maxOutputNodes: per-cycle cap on the sum ofVectorSegment.size()(live-vector count = output node count) across the selected inputs. Default0(disabled — purely additive).microsegment.max.nodesper input).maxInputs([k3s-bench] PQRetrainer.extractVectorsSequential: 2+ hour stall due to sequential random I/O over remote file server #587) so a flat cap cannot starve the tailer at high segment counts.computeTieredMaxOutputNodeshelper symmetric tocomputeTieredMaxInputs, with the same overflow guard.IndexingServerConfiguration→IndexingServiceEngine→PersistentVectorStorewith setter/getter and an INFO log line update.Tests
VectorIndexCompactorMaxOutputNodesTest(new, 17 cases): pins disabled-by-default, caps at the configured node budget, keeps ≥ 2 when the first two inputs already exceed the cap, all no-op short-circuit branches (disabled / single-segment / already-fits), micro-segment fast-path bypass, interaction withmaxInputs(smaller cap wins both directions), tier-scaling math at every threshold, base-0-stays-0, overflow safety, fire/no-fire invariance (cap never changes whether a cycle compacts), and the input-list-must-not-be-mutated contract.VectorIndexCompactorChooseTest(23 cases) exercises the 8-arg overload that now delegates to the 9-arg one — pins backward compatibility.-Dmaven.repo.local=$MAVEN_REPO.spotless:check apache-rat:check spotbugs:check install -Pci -DskipTests) passes onherddb-indexing-serviceand all its transitive ancestors.Operator guidance
The cap is opt-in: operators who observe cycle duration growing with table size (the symptom from this issue) can set
vector.index.compaction.maxOutputNodesto a positive value. A reasonable starting point ismaxLiveGraphSize × maxCount(e.g. 2,000,000 formaxLiveGraphSize=100000andmaxCount=20— the configuration in the issue). The picker logs at INFO when the cap actually bites, mirroringcapInputs, so the post-fix steady state is easy to verify.🤖 Implemented by the
pr-workeragent.