Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions VECTOR.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ The `IndexingServerConfiguration` class provides typed access to all IndexingSer
| Vector compaction min bytes | `vector.index.compaction.minBytes` | `268435456` | Minimum total size of compaction candidates before firing (256 MB) |
| Vector compaction max bytes | `vector.index.compaction.maxBytes` | `1073741824` | Hard cap on bytes read per compaction run (1 GB) |
| Vector compaction target bytes | `vector.index.compaction.target.bytes` | `8589934592` | Per-segment graduation cap: segments ≥ this size are filtered out of the IS-local picker and never re-merged. Mirrors `indexoptimizer.merge.target.bytes` so the IS-local path converges to the same steady-state segment size as the optimizer pod (8 GiB). Set to `Long.MAX_VALUE` to disable. |
| Vector compaction max output nodes | `vector.index.compaction.maxOutputNodes` | `0` | Per-cycle cap on the sum of live-vector counts across the selected merge inputs (= live-node count of the merged output). Bounds per-cycle wall-clock time so cycle duration tracks the work budget rather than the table size. Applied after the trigger / smallest-first / micro-segment / graduation selection logic; always keeps ≥ 2 inputs for a meaningful merge; bypassed by the micro-segment fast path; tier-scaled (2×/4×/8× at 100/300/500 segments) so a flat cap cannot starve the tailer. `0` disables the cap (default). |
| Vector compaction retention | `vector.index.compaction.retentionMs` | `600000` | Retention deadline for old segment files after a compaction swap (10 min) |
| Storage type | `indexing.storage.type` | `file` | `file` (persistent) or `memory` (testing) |
| Memory multiplier | `indexing.vector.memoryMultiplier` | `5.0` | Multiplier for memory estimation |
Expand Down Expand Up @@ -604,6 +605,8 @@ Large accumulations of small on-disk segments slow queries (search fans out acro

**Policy.** `VectorIndexCompactor.chooseSegmentsToMerge` picks the smallest-first subset of candidates up to `vector.index.compaction.maxBytes` (default 1 GB), firing only when both a minimum count is met and the combined input size crosses `vector.index.compaction.minBytes` (default 256 MB). Compaction is throttled by `vector.index.compaction.intervalMs` (default 5 min) — independent of the checkpoint driver's `indexing.compaction.interval`.

**Per-cycle output-node cap.** After the trigger / smallest-first / micro-segment / graduation-cap selection runs, the picker additionally caps the sum of `VectorSegment.size()` (live-vector count per input, which equals the live-node count contributed to the merged output) across the chosen inputs at `vector.index.compaction.maxOutputNodes` (default `0`, disabled). The cap bounds per-cycle wall-clock time: jvector's `OnDiskGraphIndexCompactor.buildBatches` emits `max(40, srcNodes/128)` batches per source segment, so wall-clock time tracks the total live-node count of the merge inputs. Without the cap, each successive cycle progressively merges larger segments as the "smallest N" candidates shift upward over time, and cycle duration grows with the table size rather than with the work budget. The cap is applied AFTER the fire/no-fire trigger (never changes whether a cycle compacts), is bypassed by the micro-segment fast path (those merges are bounded by `vector.index.compaction.microsegment.max.nodes` per input by construction), is tier-scaled (2×/4×/8× at 100/300/500 segments) like `maxInputs` so the per-cycle drain rate keeps pace with the backlog, and always keeps at least 2 inputs so an aggressively low cap still produces monotonic progress (segment count decreases by one per cycle). The cap is on **live** node count, but jvector's batch loop iterates over the on-disk graph including tombstoned ordinals, so the wall-clock bound is approximate on segments where `liveCount << maxOrdinal` — workloads dominated by deletes should size the cap with the expected tombstone fraction in mind. Set to a positive value (a reasonable starting point is `maxLiveGraphSize × maxCount`) when per-cycle stalls are observed to grow with table size.

**Graduation cap.** Before any of the selection logic runs, the picker filters out every segment whose `estimatedSizeBytes` is at or above `vector.index.compaction.target.bytes` (default 8 GiB) — segments that have crossed the cap are treated as "graduated" and are never merged again. This mirrors the optimizer pod's `AggressivePolicy` (`indexoptimizer.merge.target.bytes`, same 8 GiB default), so the IS-local path converges to the same steady-state segment size whether or not the external optimizer is enabled. Once at most one segment remains below the cap, neither the byte nor the count trigger fires on the filtered set and compaction stops running for that index until ingest produces fresh sub-target segments. Operators can spot the steady state in the IS log: each cycle that finds nothing to merge emits one INFO line (`compaction in steady state — nothing to merge…`) with the total / graduated / sub-target counts and the trigger thresholds. If the index is also at or above `vector.index.compaction.backpressure.segments` while compaction has nothing to do — typically every segment has graduated — the IS instead emits a WARNING line tagged `BACKPRESSURE DEADLOCK`, naming the operator knobs to raise (`target.bytes`, `backpressure.segments`, cluster size). Set `target.bytes` to `Long.MAX_VALUE` to disable the cap and restore the pre-#631 smallest-first-with-no-graduation behaviour.

**Live-PK filter.** `VectorIndexCompactor.buildAuthorityMap` drops from the merged output:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,40 @@ public final class IndexingServerConfiguration {
public static final long PROPERTY_VECTOR_INDEX_COMPACTION_TARGET_BYTES_DEFAULT =
8L * 1024L * 1024L * 1024L; // 8 GiB — matches indexoptimizer.merge.target.bytes

/**
* Per-cycle cap on the sum of live-vector counts across the selected merge
* inputs (issue #643). Bounds the per-cycle wall-clock time of the IS-local
* compaction picker: jvector's {@code OnDiskGraphIndexCompactor.buildBatches}
* emits {@code max(40, srcNodes/128)} batches per source segment, so wall-clock
* time tracks the total live-node count of the merge inputs. Without this
* cap, each successive cycle progressively merges larger segments as the
* "smallest 20" candidates shift upward over time — cycle duration grows
* with the table size rather than with the per-cycle work budget (see issue
* for evidence on a 200M-row BIGANN run).
*
* <p>Applied AFTER the trigger / smallest-first / micro-segment / graduation
* filtering (it never changes whether a cycle compacts, only the size of the
* merged output). The cap is also tier-scaled (2×/4×/8× at 100/300/500
* segments) when tiered compaction is enabled, so the per-cycle drain rate
* rises with the backlog and a flat cap cannot starve the tailer. The cap
* never applies to the micro-segment fast path (those merges are bounded
* by {@link #PROPERTY_VECTOR_INDEX_COMPACTION_MICROSEGMENT_MAX_NODES} per
* input by construction).
*
* <p><b>Always keeps at least 2 inputs.</b> Even when the first two
* candidates already exceed the cap, the picker keeps both so the cycle
* still performs a meaningful merge.
*
* <p>Default is {@code 0} (disabled — preserves the pre-#643 unbounded
* output behaviour). Set explicitly when an operator observes per-cycle
* stalls growing with table size; a reasonable starting value is
* {@code maxLiveGraphSize × maxCount} (e.g. 2,000,000 for a configuration
* with {@code maxLiveGraphSize=100000} and {@code maxCount=20}).
*/
public static final String PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES =
"vector.index.compaction.maxOutputNodes";
public static final long PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES_DEFAULT = 0L;

/**
* How long old segment files remain on-disk after a compaction swap
* before the reaper may physically delete them. Also gated by
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -807,6 +807,11 @@ public void start() throws Exception {
final long vectorCompactionTargetBytes = config.getLong(
IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TARGET_BYTES,
IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TARGET_BYTES_DEFAULT);
// Per-cycle output-node cap for the IS-local picker (issue #643).
final long vectorCompactionMaxOutputNodes = config.getLong(
IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES,
IndexingServerConfiguration
.PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES_DEFAULT);
final boolean vectorCompactionTieredEnabled = config.getBoolean(
IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TIERED_ENABLED,
IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TIERED_ENABLED_DEFAULT);
Expand All @@ -833,15 +838,16 @@ public void start() throws Exception {
+ "backpressureMaxWaitMs={2}, localKickFraction={3},"
+ " localEnabledWithOptimizer={4},"
+ " microSegmentMaxNodes={5}, maxInputs={6},"
+ " maxInputBytes={7}, targetBytes={8}",
+ " maxInputBytes={7}, targetBytes={8}, maxOutputNodes={9}",
new Object[]{vectorCompactionTieredEnabled, vectorCompactionBackpressureSegments,
vectorCompactionBackpressureMaxWaitMs,
vectorCompactionLocalKickFraction,
vectorCompactionLocalEnabledWithOptimizer,
vectorCompactionMicroSegmentMaxNodes,
vectorCompactionMaxInputs,
vectorCompactionMaxInputBytes,
vectorCompactionTargetBytes});
vectorCompactionTargetBytes,
vectorCompactionMaxOutputNodes});
// Async IO pipeline for FusedPQ search (issue #547).
// Config key takes precedence over the system property.
final boolean vectorSearchAsyncPipelineEnabled = config.getBoolean(
Expand Down Expand Up @@ -962,6 +968,10 @@ public void start() throws Exception {
// compaction path converges to the same steady-state segment
// size whether or not the external optimizer is enabled.
store.setCompactionTargetBytes(vectorCompactionTargetBytes);
// Issue #643: per-cycle output-node cap bounds per-cycle
// wall-clock time so it tracks the work budget rather than
// the table size. Default 0 (disabled).
store.setCompactionMaxOutputNodes(vectorCompactionMaxOutputNodes);
store.setCompactionBackpressureThreshold(vectorCompactionBackpressureSegments);
store.setCompactionBackpressureMaxWaitMs(vectorCompactionBackpressureMaxWaitMs);
store.setLocalCompactionKickFraction(vectorCompactionLocalKickFraction);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,18 @@ private static ForkJoinPool createCheckpointPool() {
*/
private volatile long vectorIndexCompactionTargetBytes =
8L * 1024L * 1024L * 1024L;
/**
* Per-cycle cap on the sum of live-vector counts across the selected merge
* inputs (issue #643). Bounds per-cycle wall-clock time so cycle duration
* tracks the per-cycle work budget rather than the table size. When tiered
* compaction is enabled this base value is tier-scaled (2×/4×/8×) per cycle
* by {@link VectorIndexCompactor#computeTieredMaxOutputNodes} so the
* per-cycle drain rate keeps pace with the backlog and the cap cannot
* starve the tailer. {@code 0} (or negative) disables the cap — the
* default, so the cap is purely additive. Pre-normalised by
* {@link #setCompactionMaxOutputNodes}.
*/
private volatile long vectorIndexCompactionMaxOutputNodes = 0L;
private volatile long vectorIndexCompactionRetentionMs = 10L * 60_000L;

/**
Expand Down Expand Up @@ -2324,6 +2336,36 @@ public long getCompactionTargetBytes() {
return vectorIndexCompactionTargetBytes;
}

/**
* Sets the per-cycle output-node cap (issue #643). Bounds the sum of
* live-vector counts across the inputs a single compaction cycle merges,
* so per-cycle wall-clock time tracks the per-cycle work budget rather
* than the table size. The cap is applied AFTER the trigger / smallest-first
* / micro-segment / graduation-cap selection logic (never changes whether
* a cycle compacts) and is tier-scaled by
* {@link VectorIndexCompactor#computeTieredMaxOutputNodes} when tiered
* compaction is enabled.
*
* <p>The picker always keeps at least 2 inputs even when the first two
* candidates already exceed the cap, so an aggressively low value still
* produces monotonic progress (segment count decreases by one per cycle).
*
* @param maxOutputNodes positive cap on summed input
* {@link VectorSegment#size()}; values {@code <= 0} disable the cap
* (the default). Can be called at any time to re-tune a running store.
*/
public void setCompactionMaxOutputNodes(long maxOutputNodes) {
this.vectorIndexCompactionMaxOutputNodes = Math.max(0L, maxOutputNodes);
}

/**
* Returns the current per-cycle output-node cap (issue #643); {@code 0}
* means the cap is disabled.
*/
public long getCompactionMaxOutputNodes() {
return vectorIndexCompactionMaxOutputNodes;
}

/**
* Sets the segment-count back-pressure threshold (issue #354).
* {@link #addVectorInternal} will block when {@code segments.size()}
Expand Down Expand Up @@ -2880,6 +2922,11 @@ public void runCompactionCycle() throws InterruptedException {
// byte/count caps so the per-cycle drain rate rises with the backlog
// — that is what keeps the flat cap from starving the tailer.
int effectiveMaxInputs = vectorIndexCompactionMaxInputs;
// Issue #643: the output-node cap is tier-scaled too, same starvation
// argument as maxInputs — per-cycle work budget rises with the
// backlog so a flat cap cannot let the segment count grow toward
// back-pressure.
long effectiveMaxOutputNodes = vectorIndexCompactionMaxOutputNodes;
if (vectorIndexCompactionTieredEnabled) {
int tier = VectorIndexCompactor.tieredMultiplier(snapshot.size());
if (tier > 1) {
Expand All @@ -2889,13 +2936,16 @@ public void runCompactionCycle() throws InterruptedException {
snapshot.size(), vectorIndexCompactionMaxCount);
effectiveMaxInputs = VectorIndexCompactor.computeTieredMaxInputs(
snapshot.size(), vectorIndexCompactionMaxInputs);
effectiveMaxOutputNodes = VectorIndexCompactor.computeTieredMaxOutputNodes(
snapshot.size(), vectorIndexCompactionMaxOutputNodes);
LOGGER.log(Level.INFO,
"vector store {0}: tiered compaction — {1} segments → "
+ "effective maxBytes={2} ({3}×), effective maxCount={4} ({3}×),"
+ " effective maxInputs={5} ({3}×)",
+ " effective maxInputs={5} ({3}×),"
+ " effective maxOutputNodes={6} ({3}×)",
new Object[]{indexName, snapshot.size(),
effectiveMaxBytes, tier, effectiveMaxCount,
effectiveMaxInputs});
effectiveMaxInputs, effectiveMaxOutputNodes});
}
}

Expand All @@ -2913,7 +2963,12 @@ public void runCompactionCycle() throws InterruptedException {
// of every selection step so the policy converges to a fixed
// point once every segment crosses the cap — parity with the
// optimizer pod's AggressivePolicy.
vectorIndexCompactionTargetBytes);
vectorIndexCompactionTargetBytes,
// Issue #643: per-cycle cap on the sum of input liveCount
// (= output node count). Tier-scaled above. Bounds per-cycle
// wall-clock time so cycle duration tracks the work budget
// rather than the table size.
effectiveMaxOutputNodes);

// Issue #602: apply the per-cycle download budget cap AFTER candidate
// selection. Tiered scaling can push effectiveMaxBytes to 8 GiB; with
Expand Down
Loading
Loading