Skip to content

issue #643: bound IS-local compaction cycle duration via maxOutputNodes cap - #644

Merged
eolivelli merged 2 commits into
masterfrom
issue-643-is-compaction-cycle-duration-grows-with
May 23, 2026
Merged

eolivelli merged 2 commits into
masterfrom
issue-643-is-compaction-cycle-duration-grows-with

Conversation

@eolivelli

Copy link
Copy Markdown
Owner

Fixes #643.

Root cause

Confirmed by reading both jvector's OnDiskGraphIndexCompactor.buildBatches and HerdDB's VectorIndexCompactor.chooseSegmentsToMerge:

  1. jvector batch cost is purely per-input. buildBatches walks each sources[s] at level 0 and emits max(40, srcNodes/128) batches per source. Total batches = sum across the merge inputs only. There is no scan of non-input on-disk segments — the issue's investigation point TEST CI #3 (super-linear neighbourhood expansion) is refuted. Investigation point TEST CI on better concurrency model #2 is confirmed: batch count grows ~linearly with the selected node count.
  2. The IS-local picker has no output-node cap. It sorts smallest-first by estimatedSizeBytes and caps by maxBytes / maxCount / maxInputs. None of these bounds the total live-vector count of the output. As small segments are consumed and merged into medium segments, the "smallest N" candidates shift upward over time, until each input is close to the graduation cap (8 GiB). Cycle wall-clock time grows with the table size — exactly as the 200M-row BIGANN run in the issue shows (3× growth in cycle duration, stable throughput).

Summary

  • Add vector.index.compaction.maxOutputNodes: per-cycle cap on the sum of VectorSegment.size() (live-vector count = output node count) across the selected inputs. Default 0 (disabled — purely additive).
  • Applied AFTER the trigger / smallest-first / micro-segment / graduation-cap selection logic, so it never changes the fire/no-fire decision.
  • Bypassed by the micro-segment fast path (those merges are already bounded by microsegment.max.nodes per input).
  • Tier-scaled (2×/4×/8× at 100/300/500 segments) like maxInputs ([k3s-bench] PQRetrainer.extractVectorsSequential: 2+ hour stall due to sequential random I/O over remote file server #587) so a flat cap cannot starve the tailer at high segment counts.
  • Always keeps at least 2 inputs for a meaningful merge — even an aggressively low cap still produces monotonic progress (segment count decreases by one per cycle).
  • New computeTieredMaxOutputNodes helper symmetric to computeTieredMaxInputs, with the same overflow guard.
  • Wired through IndexingServerConfiguration → IndexingServiceEngine → PersistentVectorStore with setter/getter and an INFO log line update.
  • VECTOR.md updated with the new policy paragraph and configuration-table row (present-tense, no change-history per the project doc convention).

Tests

  • VectorIndexCompactorMaxOutputNodesTest (new, 17 cases): pins disabled-by-default, caps at the configured node budget, keeps ≥ 2 when the first two inputs already exceed the cap, all no-op short-circuit branches (disabled / single-segment / already-fits), micro-segment fast-path bypass, interaction with maxInputs (smaller cap wins both directions), tier-scaling math at every threshold, base-0-stays-0, overflow safety, fire/no-fire invariance (cap never changes whether a cycle compacts), and the input-list-must-not-be-mutated contract.
  • Existing VectorIndexCompactorChooseTest (23 cases) exercises the 8-arg overload that now delegates to the 9-arg one — pins backward compatibility.
  • All 40 tests pass locally with -Dmaven.repo.local=$MAVEN_REPO.
  • Pre-PR validation (spotless:check apache-rat:check spotbugs:check install -Pci -DskipTests) passes on herddb-indexing-service and all its transitive ancestors.

Operator guidance

The cap is opt-in: operators who observe cycle duration growing with table size (the symptom from this issue) can set vector.index.compaction.maxOutputNodes to a positive value. A reasonable starting point is maxLiveGraphSize × maxCount (e.g. 2,000,000 for maxLiveGraphSize=100000 and maxCount=20 — the configuration in the issue). The picker logs at INFO when the cap actually bites, mirroring capInputs, so the post-fix steady state is easy to verify.

🤖 Implemented by the pr-worker agent.

eolivelli and others added 2 commits May 22, 2026 23:51
…es cap

Add `vector.index.compaction.maxOutputNodes`: a per-cycle cap on the sum
of live-vector counts across the inputs a single IS-local compaction
cycle merges. Bounds per-cycle wall-clock time so cycle duration tracks
the per-cycle work budget rather than the total table size.

Root cause (confirmed via jvector + HerdDB source inspection): jvector's
`OnDiskGraphIndexCompactor.buildBatches` emits `max(40, srcNodes/128)`
batches per source segment with no scan of non-input on-disk segments,
so total batches scale linearly with the selected input set. The IS-local
picker sorts smallest-first by `estimatedSizeBytes` and caps by
`maxBytes`/`maxCount`/`maxInputs` — none of these bounds the total
live-vector count of the output. As small segments are consumed by
earlier cycles and merged into medium segments, the "smallest N"
candidates shift upward over time, and each successive cycle merges
progressively larger inputs — the 200M-row BIGANN run in the issue
shows 3× growth in cycle duration without any change in throughput.

The new cap is applied AFTER the trigger / smallest-first / micro-segment
/ graduation-cap selection (never changes whether a cycle compacts), is
bypassed by the micro-segment fast path (those merges are already
bounded per input), is tier-scaled (2×/4×/8× at 100/300/500 segments)
like `maxInputs` so a flat cap cannot starve the tailer, and always
keeps at least 2 inputs for a meaningful merge. Default is `0`
(disabled) — purely additive.

Tests: `VectorIndexCompactorMaxOutputNodesTest` pins all branches
(disabled-by-default, caps at node budget, keeps ≥ 2 when the first
two already exceed the cap, no-op short-circuits, micro-segment
fast-path bypass, interaction with `maxInputs`, tier-scaling math
including overflow safety, and fire/no-fire invariance). Existing
`VectorIndexCompactorChooseTest` exercises the 8-arg overload to pin
backward compatibility.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…l trade-off

Address pr-reviewer follow-ups for #644:

1. New PersistentVectorStoreCompactionMaxOutputNodesTest pins the public
   API surface — default (0, disabled), positive set, explicit-0
   disabling, and the Math.max(0L, ...) clamp on negatives /
   Long.MIN_VALUE — so the property → engine → store wiring stays
   correct. Mirrors PersistentVectorStoreCompactionTargetBytesTest.

2. Javadoc on the 9-arg chooseSegmentsToMerge and VECTOR.md now spell
   out that the cap is on liveCount, while jvector's batch loop scales
   with maxOrdinal (which includes tombstoned ordinals). For segments
   with liveCount << maxOrdinal the wall-clock cap underestimates the
   real work; deletion-heavy workloads should size maxOutputNodes with
   the expected tombstone fraction in mind.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@eolivelli
eolivelli merged commit f2ab182 into master May 23, 2026
8 checks passed
@eolivelli
eolivelli deleted the issue-643-is-compaction-cycle-duration-grows-with branch May 23, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[IS compaction] Cycle duration grows with table size rather than input segment size — investigate OnDiskGraphIndexCompactor batch cost

1 participant