diff --git a/VECTOR.md b/VECTOR.md index 0bf6c88f..dd0e87d3 100644 --- a/VECTOR.md +++ b/VECTOR.md @@ -190,6 +190,7 @@ The `IndexingServerConfiguration` class provides typed access to all IndexingSer | Vector compaction min bytes | `vector.index.compaction.minBytes` | `268435456` | Minimum total size of compaction candidates before firing (256 MB) | | Vector compaction max bytes | `vector.index.compaction.maxBytes` | `1073741824` | Hard cap on bytes read per compaction run (1 GB) | | Vector compaction target bytes | `vector.index.compaction.target.bytes` | `8589934592` | Per-segment graduation cap: segments ≥ this size are filtered out of the IS-local picker and never re-merged. Mirrors `indexoptimizer.merge.target.bytes` so the IS-local path converges to the same steady-state segment size as the optimizer pod (8 GiB). Set to `Long.MAX_VALUE` to disable. | +| Vector compaction max output nodes | `vector.index.compaction.maxOutputNodes` | `0` | Per-cycle cap on the sum of live-vector counts across the selected merge inputs (= live-node count of the merged output). Bounds per-cycle wall-clock time so cycle duration tracks the work budget rather than the table size. Applied after the trigger / smallest-first / micro-segment / graduation selection logic; always keeps ≥ 2 inputs for a meaningful merge; bypassed by the micro-segment fast path; tier-scaled (2×/4×/8× at 100/300/500 segments) so a flat cap cannot starve the tailer. `0` disables the cap (default). | | Vector compaction retention | `vector.index.compaction.retentionMs` | `600000` | Retention deadline for old segment files after a compaction swap (10 min) | | Storage type | `indexing.storage.type` | `file` | `file` (persistent) or `memory` (testing) | | Memory multiplier | `indexing.vector.memoryMultiplier` | `5.0` | Multiplier for memory estimation | @@ -604,6 +605,8 @@ Large accumulations of small on-disk segments slow queries (search fans out acro **Policy.** `VectorIndexCompactor.chooseSegmentsToMerge` picks the smallest-first subset of candidates up to `vector.index.compaction.maxBytes` (default 1 GB), firing only when both a minimum count is met and the combined input size crosses `vector.index.compaction.minBytes` (default 256 MB). Compaction is throttled by `vector.index.compaction.intervalMs` (default 5 min) — independent of the checkpoint driver's `indexing.compaction.interval`. +**Per-cycle output-node cap.** After the trigger / smallest-first / micro-segment / graduation-cap selection runs, the picker additionally caps the sum of `VectorSegment.size()` (live-vector count per input, which equals the live-node count contributed to the merged output) across the chosen inputs at `vector.index.compaction.maxOutputNodes` (default `0`, disabled). The cap bounds per-cycle wall-clock time: jvector's `OnDiskGraphIndexCompactor.buildBatches` emits `max(40, srcNodes/128)` batches per source segment, so wall-clock time tracks the total live-node count of the merge inputs. Without the cap, each successive cycle progressively merges larger segments as the "smallest N" candidates shift upward over time, and cycle duration grows with the table size rather than with the work budget. The cap is applied AFTER the fire/no-fire trigger (never changes whether a cycle compacts), is bypassed by the micro-segment fast path (those merges are bounded by `vector.index.compaction.microsegment.max.nodes` per input by construction), is tier-scaled (2×/4×/8× at 100/300/500 segments) like `maxInputs` so the per-cycle drain rate keeps pace with the backlog, and always keeps at least 2 inputs so an aggressively low cap still produces monotonic progress (segment count decreases by one per cycle). The cap is on **live** node count, but jvector's batch loop iterates over the on-disk graph including tombstoned ordinals, so the wall-clock bound is approximate on segments where `liveCount << maxOrdinal` — workloads dominated by deletes should size the cap with the expected tombstone fraction in mind. Set to a positive value (a reasonable starting point is `maxLiveGraphSize × maxCount`) when per-cycle stalls are observed to grow with table size. + **Graduation cap.** Before any of the selection logic runs, the picker filters out every segment whose `estimatedSizeBytes` is at or above `vector.index.compaction.target.bytes` (default 8 GiB) — segments that have crossed the cap are treated as "graduated" and are never merged again. This mirrors the optimizer pod's `AggressivePolicy` (`indexoptimizer.merge.target.bytes`, same 8 GiB default), so the IS-local path converges to the same steady-state segment size whether or not the external optimizer is enabled. Once at most one segment remains below the cap, neither the byte nor the count trigger fires on the filtered set and compaction stops running for that index until ingest produces fresh sub-target segments. Operators can spot the steady state in the IS log: each cycle that finds nothing to merge emits one INFO line (`compaction in steady state — nothing to merge…`) with the total / graduated / sub-target counts and the trigger thresholds. If the index is also at or above `vector.index.compaction.backpressure.segments` while compaction has nothing to do — typically every segment has graduated — the IS instead emits a WARNING line tagged `BACKPRESSURE DEADLOCK`, naming the operator knobs to raise (`target.bytes`, `backpressure.segments`, cluster size). Set `target.bytes` to `Long.MAX_VALUE` to disable the cap and restore the pre-#631 smallest-first-with-no-graduation behaviour. **Live-PK filter.** `VectorIndexCompactor.buildAuthorityMap` drops from the merged output: diff --git a/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServerConfiguration.java b/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServerConfiguration.java index 9090a051..6914029a 100644 --- a/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServerConfiguration.java +++ b/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServerConfiguration.java @@ -357,6 +357,40 @@ public final class IndexingServerConfiguration { public static final long PROPERTY_VECTOR_INDEX_COMPACTION_TARGET_BYTES_DEFAULT = 8L * 1024L * 1024L * 1024L; // 8 GiB — matches indexoptimizer.merge.target.bytes + /** + * Per-cycle cap on the sum of live-vector counts across the selected merge + * inputs (issue #643). Bounds the per-cycle wall-clock time of the IS-local + * compaction picker: jvector's {@code OnDiskGraphIndexCompactor.buildBatches} + * emits {@code max(40, srcNodes/128)} batches per source segment, so wall-clock + * time tracks the total live-node count of the merge inputs. Without this + * cap, each successive cycle progressively merges larger segments as the + * "smallest 20" candidates shift upward over time — cycle duration grows + * with the table size rather than with the per-cycle work budget (see issue + * for evidence on a 200M-row BIGANN run). + * + *
Applied AFTER the trigger / smallest-first / micro-segment / graduation + * filtering (it never changes whether a cycle compacts, only the size of the + * merged output). The cap is also tier-scaled (2×/4×/8× at 100/300/500 + * segments) when tiered compaction is enabled, so the per-cycle drain rate + * rises with the backlog and a flat cap cannot starve the tailer. The cap + * never applies to the micro-segment fast path (those merges are bounded + * by {@link #PROPERTY_VECTOR_INDEX_COMPACTION_MICROSEGMENT_MAX_NODES} per + * input by construction). + * + *
Always keeps at least 2 inputs. Even when the first two + * candidates already exceed the cap, the picker keeps both so the cycle + * still performs a meaningful merge. + * + *
Default is {@code 0} (disabled — preserves the pre-#643 unbounded + * output behaviour). Set explicitly when an operator observes per-cycle + * stalls growing with table size; a reasonable starting value is + * {@code maxLiveGraphSize × maxCount} (e.g. 2,000,000 for a configuration + * with {@code maxLiveGraphSize=100000} and {@code maxCount=20}). + */ + public static final String PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES = + "vector.index.compaction.maxOutputNodes"; + public static final long PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES_DEFAULT = 0L; + /** * How long old segment files remain on-disk after a compaction swap * before the reaper may physically delete them. Also gated by diff --git a/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServiceEngine.java b/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServiceEngine.java index 98cfcdff..f9660f66 100644 --- a/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServiceEngine.java +++ b/herddb-indexing-service/src/main/java/herddb/indexing/IndexingServiceEngine.java @@ -807,6 +807,11 @@ public void start() throws Exception { final long vectorCompactionTargetBytes = config.getLong( IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TARGET_BYTES, IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TARGET_BYTES_DEFAULT); + // Per-cycle output-node cap for the IS-local picker (issue #643). + final long vectorCompactionMaxOutputNodes = config.getLong( + IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES, + IndexingServerConfiguration + .PROPERTY_VECTOR_INDEX_COMPACTION_MAX_OUTPUT_NODES_DEFAULT); final boolean vectorCompactionTieredEnabled = config.getBoolean( IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TIERED_ENABLED, IndexingServerConfiguration.PROPERTY_VECTOR_INDEX_COMPACTION_TIERED_ENABLED_DEFAULT); @@ -833,7 +838,7 @@ public void start() throws Exception { + "backpressureMaxWaitMs={2}, localKickFraction={3}," + " localEnabledWithOptimizer={4}," + " microSegmentMaxNodes={5}, maxInputs={6}," - + " maxInputBytes={7}, targetBytes={8}", + + " maxInputBytes={7}, targetBytes={8}, maxOutputNodes={9}", new Object[]{vectorCompactionTieredEnabled, vectorCompactionBackpressureSegments, vectorCompactionBackpressureMaxWaitMs, vectorCompactionLocalKickFraction, @@ -841,7 +846,8 @@ public void start() throws Exception { vectorCompactionMicroSegmentMaxNodes, vectorCompactionMaxInputs, vectorCompactionMaxInputBytes, - vectorCompactionTargetBytes}); + vectorCompactionTargetBytes, + vectorCompactionMaxOutputNodes}); // Async IO pipeline for FusedPQ search (issue #547). // Config key takes precedence over the system property. final boolean vectorSearchAsyncPipelineEnabled = config.getBoolean( @@ -962,6 +968,10 @@ public void start() throws Exception { // compaction path converges to the same steady-state segment // size whether or not the external optimizer is enabled. store.setCompactionTargetBytes(vectorCompactionTargetBytes); + // Issue #643: per-cycle output-node cap bounds per-cycle + // wall-clock time so it tracks the work budget rather than + // the table size. Default 0 (disabled). + store.setCompactionMaxOutputNodes(vectorCompactionMaxOutputNodes); store.setCompactionBackpressureThreshold(vectorCompactionBackpressureSegments); store.setCompactionBackpressureMaxWaitMs(vectorCompactionBackpressureMaxWaitMs); store.setLocalCompactionKickFraction(vectorCompactionLocalKickFraction); diff --git a/herddb-indexing-service/src/main/java/herddb/indexing/vector/PersistentVectorStore.java b/herddb-indexing-service/src/main/java/herddb/indexing/vector/PersistentVectorStore.java index 1a4cb6be..927002be 100644 --- a/herddb-indexing-service/src/main/java/herddb/indexing/vector/PersistentVectorStore.java +++ b/herddb-indexing-service/src/main/java/herddb/indexing/vector/PersistentVectorStore.java @@ -654,6 +654,18 @@ private static ForkJoinPool createCheckpointPool() { */ private volatile long vectorIndexCompactionTargetBytes = 8L * 1024L * 1024L * 1024L; + /** + * Per-cycle cap on the sum of live-vector counts across the selected merge + * inputs (issue #643). Bounds per-cycle wall-clock time so cycle duration + * tracks the per-cycle work budget rather than the table size. When tiered + * compaction is enabled this base value is tier-scaled (2×/4×/8×) per cycle + * by {@link VectorIndexCompactor#computeTieredMaxOutputNodes} so the + * per-cycle drain rate keeps pace with the backlog and the cap cannot + * starve the tailer. {@code 0} (or negative) disables the cap — the + * default, so the cap is purely additive. Pre-normalised by + * {@link #setCompactionMaxOutputNodes}. + */ + private volatile long vectorIndexCompactionMaxOutputNodes = 0L; private volatile long vectorIndexCompactionRetentionMs = 10L * 60_000L; /** @@ -2324,6 +2336,36 @@ public long getCompactionTargetBytes() { return vectorIndexCompactionTargetBytes; } + /** + * Sets the per-cycle output-node cap (issue #643). Bounds the sum of + * live-vector counts across the inputs a single compaction cycle merges, + * so per-cycle wall-clock time tracks the per-cycle work budget rather + * than the table size. The cap is applied AFTER the trigger / smallest-first + * / micro-segment / graduation-cap selection logic (never changes whether + * a cycle compacts) and is tier-scaled by + * {@link VectorIndexCompactor#computeTieredMaxOutputNodes} when tiered + * compaction is enabled. + * + *
The picker always keeps at least 2 inputs even when the first two
+ * candidates already exceed the cap, so an aggressively low value still
+ * produces monotonic progress (segment count decreases by one per cycle).
+ *
+ * @param maxOutputNodes positive cap on summed input
+ * {@link VectorSegment#size()}; values {@code <= 0} disable the cap
+ * (the default). Can be called at any time to re-tune a running store.
+ */
+ public void setCompactionMaxOutputNodes(long maxOutputNodes) {
+ this.vectorIndexCompactionMaxOutputNodes = Math.max(0L, maxOutputNodes);
+ }
+
+ /**
+ * Returns the current per-cycle output-node cap (issue #643); {@code 0}
+ * means the cap is disabled.
+ */
+ public long getCompactionMaxOutputNodes() {
+ return vectorIndexCompactionMaxOutputNodes;
+ }
+
/**
* Sets the segment-count back-pressure threshold (issue #354).
* {@link #addVectorInternal} will block when {@code segments.size()}
@@ -2880,6 +2922,11 @@ public void runCompactionCycle() throws InterruptedException {
// byte/count caps so the per-cycle drain rate rises with the backlog
// — that is what keeps the flat cap from starving the tailer.
int effectiveMaxInputs = vectorIndexCompactionMaxInputs;
+ // Issue #643: the output-node cap is tier-scaled too, same starvation
+ // argument as maxInputs — per-cycle work budget rises with the
+ // backlog so a flat cap cannot let the segment count grow toward
+ // back-pressure.
+ long effectiveMaxOutputNodes = vectorIndexCompactionMaxOutputNodes;
if (vectorIndexCompactionTieredEnabled) {
int tier = VectorIndexCompactor.tieredMultiplier(snapshot.size());
if (tier > 1) {
@@ -2889,13 +2936,16 @@ public void runCompactionCycle() throws InterruptedException {
snapshot.size(), vectorIndexCompactionMaxCount);
effectiveMaxInputs = VectorIndexCompactor.computeTieredMaxInputs(
snapshot.size(), vectorIndexCompactionMaxInputs);
+ effectiveMaxOutputNodes = VectorIndexCompactor.computeTieredMaxOutputNodes(
+ snapshot.size(), vectorIndexCompactionMaxOutputNodes);
LOGGER.log(Level.INFO,
"vector store {0}: tiered compaction — {1} segments → "
+ "effective maxBytes={2} ({3}×), effective maxCount={4} ({3}×),"
- + " effective maxInputs={5} ({3}×)",
+ + " effective maxInputs={5} ({3}×),"
+ + " effective maxOutputNodes={6} ({3}×)",
new Object[]{indexName, snapshot.size(),
effectiveMaxBytes, tier, effectiveMaxCount,
- effectiveMaxInputs});
+ effectiveMaxInputs, effectiveMaxOutputNodes});
}
}
@@ -2913,7 +2963,12 @@ public void runCompactionCycle() throws InterruptedException {
// of every selection step so the policy converges to a fixed
// point once every segment crosses the cap — parity with the
// optimizer pod's AggressivePolicy.
- vectorIndexCompactionTargetBytes);
+ vectorIndexCompactionTargetBytes,
+ // Issue #643: per-cycle cap on the sum of input liveCount
+ // (= output node count). Tier-scaled above. Bounds per-cycle
+ // wall-clock time so cycle duration tracks the work budget
+ // rather than the table size.
+ effectiveMaxOutputNodes);
// Issue #602: apply the per-cycle download budget cap AFTER candidate
// selection. Tiered scaling can push effectiveMaxBytes to 8 GiB; with
diff --git a/herddb-indexing-service/src/main/java/herddb/indexing/vector/VectorIndexCompactor.java b/herddb-indexing-service/src/main/java/herddb/indexing/vector/VectorIndexCompactor.java
index dd294f52..ec30d3bc 100644
--- a/herddb-indexing-service/src/main/java/herddb/indexing/vector/VectorIndexCompactor.java
+++ b/herddb-indexing-service/src/main/java/herddb/indexing/vector/VectorIndexCompactor.java
@@ -416,7 +416,80 @@ static List Package-private for unit tests.
+ */
+ static List The graduation-cap filter (issue #631) runs first, exactly as before:
+ * segments whose {@link VectorSegment#estimatedSizeBytes} is at or above
+ * {@code targetMaxBytes} are excluded from every later step. See the
+ * paragraphs below for the unchanged graduation-cap semantics.
+ *
+ * Per-cycle output-node cap (issue #643). After the byte
+ * accumulation loop has selected the smallest-first picked set, this method
+ * additionally truncates {@code picked} so that the sum of
+ * {@link VectorSegment#size()} (the live-vector count of each input, which
+ * equals the live-node count contributed to the merged output) stays at or
+ * below {@code maxOutputNodes}. The cap is applied after the
+ * fire/no-fire trigger and before {@link #capInputs}, on the normal
+ * selection only — the micro-segment fast path is deliberately bypassed
+ * (mirroring the {@code maxInputs} cap, see issue #570). Without this cap,
+ * each compaction cycle's wall-clock time grew with the table size rather
+ * than with the per-cycle work budget: as small segments were consumed and
+ * merged into medium segments, the "smallest 20" candidates shifted upward
+ * over time, until each input was close to the graduation cap. jvector's
+ * {@code OnDiskGraphIndexCompactor.buildBatches} emits
+ * {@code max(40, srcNodes/128)} batches per source segment, so wall-clock
+ * time tracks the total live-node count of the merge inputs. Capping that
+ * sum bounds cycle duration to roughly the time required to merge a fixed
+ * number of vectors regardless of how large the index has grown.
+ *
+ * Approximate on heavily-tombstoned segments. The cap is on
+ * {@link VectorSegment#size()} (live count), but jvector's batch loop
+ * iterates over the on-disk graph including tombstoned ordinals (work
+ * scales with {@code maxOrdinal}, not {@code liveCount}). For a segment
+ * where most PKs have been tombstoned, {@code liveCount} can be much
+ * smaller than {@code maxOrdinal} — the wall-clock cap under-estimates
+ * the actual per-cycle work in that case. This is intentional: heavily
+ * tombstoned segments are exactly the ones whose merge most aggressively
+ * reclaims storage (the rebuild drops every tombstoned ordinal), so
+ * letting the picker over-pick them is the right trade-off. Operators
+ * with workloads dominated by deletes should size {@code maxOutputNodes}
+ * with the expected tombstone fraction in mind.
+ *
+ * Always keeps at least 2 inputs. Even when the first two
+ * candidates by ascending byte size already exceed {@code maxOutputNodes},
+ * the cap retains both so the cycle still performs a meaningful merge (a
+ * one-input "merge" is degenerate and would defeat the policy). Operators
+ * who configure an aggressively low cap therefore still get monotonic
+ * progress: the segment count decreases by one per cycle, while per-cycle
+ * cost stays bounded by the two-segment minimum.
+ *
+ * Disabled by default. Set {@code maxOutputNodes <= 0} to disable
+ * the cap and restore the pre-#643 unbounded-output behaviour; this is the
+ * default so the new cap is purely additive.
+ *
+ * The pre-existing graduation-cap docs follow.
*
* Before any of the trigger / smallest-first / micro-segment / input-count
* logic runs, {@code candidates} is filtered to retain only those whose
@@ -457,6 +530,10 @@ static List The input list is expected to be sorted smallest-bytes-first by the
+ * caller, so the prefix kept after truncation continues to maximise the
+ * contraction ratio (consume the smallest segments first).
+ *
+ * Always keeps at least 2 inputs. Even when the first two
+ * segments already exceed the cap, the helper keeps both — a single-input
+ * "merge" is degenerate, so a flat cap that could reduce {@code picked} to
+ * one segment would defeat the policy. Operators who configure an
+ * aggressively low cap therefore still get monotonic progress: the segment
+ * count decreases by one per cycle, while per-cycle cost stays bounded by
+ * the two-segment minimum.
+ *
+ * Logs at INFO when the cap actually bites, mirroring {@link #capInputs}.
+ *
+ * Package-private for unit tests.
+ */
+ static List Tier-scaling the cap is what makes it starvation-safe — the same
+ * argument as for {@link #computeTieredMaxInputs}. The per-cycle drain rate
+ * rises with the backlog (2×/4×/8× at 100/300/500 segments), mirroring
+ * {@link #computeTieredMaxCount}, so the cap can never let the segment
+ * count grow unboundedly toward the back-pressure threshold while still
+ * bounding worst-case per-cycle wall-clock time at each tier.
+ *
+ * A disabled cap ({@code baseMaxOutputNodes <= 0}) is returned unchanged
+ * so it stays disabled. The result is capped at {@link Long#MAX_VALUE} to
+ * avoid overflow.
+ *
+ * Package-private for unit tests.
+ */
+ static long computeTieredMaxOutputNodes(int totalSegmentCount, long baseMaxOutputNodes) {
+ if (baseMaxOutputNodes <= 0L) {
+ return baseMaxOutputNodes;
+ }
+ int multiplier = tieredMultiplier(totalSegmentCount);
+ if (multiplier == 1) {
+ return baseMaxOutputNodes;
+ }
+ // Guard against overflow.
+ if (baseMaxOutputNodes > Long.MAX_VALUE / multiplier) {
+ return Long.MAX_VALUE;
+ }
+ return baseMaxOutputNodes * multiplier;
+ }
+
/**
* Populates the supplied {@link CompactionAuthorityMap} with the per-PK
* authority decisions used by the live-PK filter during a compaction
diff --git a/herddb-indexing-service/src/test/java/herddb/indexing/vector/PersistentVectorStoreCompactionMaxOutputNodesTest.java b/herddb-indexing-service/src/test/java/herddb/indexing/vector/PersistentVectorStoreCompactionMaxOutputNodesTest.java
new file mode 100644
index 00000000..0fa14931
--- /dev/null
+++ b/herddb-indexing-service/src/test/java/herddb/indexing/vector/PersistentVectorStoreCompactionMaxOutputNodesTest.java
@@ -0,0 +1,135 @@
+/*
+ Licensed to Diennea S.r.l. under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. Diennea S.r.l. licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ */
+package herddb.indexing.vector;
+
+import static org.junit.Assert.assertEquals;
+import herddb.core.MemoryManager;
+import herddb.file.FileDataStorageManager;
+import java.nio.file.Path;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+/**
+ * Setter/getter wiring tests for the per-cycle output-node cap (issue #643).
+ *
+ * The cap field {@code vectorIndexCompactionMaxOutputNodes} is what
+ * {@link PersistentVectorStore#runCompactionCycle} passes to
+ * {@link VectorIndexCompactor#chooseSegmentsToMerge}'s 9-arg overload. The
+ * policy-level behaviour is covered by
+ * {@link VectorIndexCompactorMaxOutputNodesTest}; this test pins the public
+ * API surface — default (disabled), positive set, clamp on negative — so the
+ * wiring through {@link herddb.indexing.IndexingServiceEngine} (which reads
+ * the config key {@code vector.index.compaction.maxOutputNodes} and calls
+ * {@link PersistentVectorStore#setCompactionMaxOutputNodes(long)}) stays
+ * correct.
+ */
+public class PersistentVectorStoreCompactionMaxOutputNodesTest {
+
+ private static final String TABLE_SPACE = "tstblspace";
+ private static final String INDEX_NAME = "mon";
+ private static final String INDEX_UUID = "mon_idx_uuid";
+
+ @Rule
+ public TemporaryFolder tmpFolder = new TemporaryFolder();
+
+ private PersistentVectorStore createStore(Path tmpDir, FileDataStorageManager dsm) {
+ MemoryManager mm = new MemoryManager(64 * 1024 * 1024, 0, 1024 * 1024, 1024 * 1024);
+ return new PersistentVectorStore(INDEX_NAME, "testtable", TABLE_SPACE,
+ "vector_col", INDEX_UUID, tmpDir, dsm, mm,
+ 16, 100, 1.2f, 1.4f, true, 2_000_000_000L, 0, Long.MAX_VALUE);
+ }
+
+ @Test
+ public void defaultIsDisabled() throws Exception {
+ Path baseDir = tmpFolder.newFolder("data").toPath();
+ Path tmpDir = tmpFolder.newFolder("tmp").toPath();
+ FileDataStorageManager dsm = new FileDataStorageManager(baseDir);
+ dsm.initTablespace(TABLE_SPACE);
+ try (PersistentVectorStore store = createStore(tmpDir, dsm)) {
+ assertEquals("default must be 0 (cap disabled — purely additive on upgrade)",
+ 0L, store.getCompactionMaxOutputNodes());
+ }
+ }
+
+ @Test
+ public void positiveValuePassesThrough() throws Exception {
+ Path baseDir = tmpFolder.newFolder("data").toPath();
+ Path tmpDir = tmpFolder.newFolder("tmp").toPath();
+ FileDataStorageManager dsm = new FileDataStorageManager(baseDir);
+ dsm.initTablespace(TABLE_SPACE);
+ try (PersistentVectorStore store = createStore(tmpDir, dsm)) {
+ // Typical operator-recommended value: maxLiveGraphSize × maxCount
+ // (2,000,000 for maxLiveGraphSize=100,000 and maxCount=20).
+ store.setCompactionMaxOutputNodes(2_000_000L);
+ assertEquals(2_000_000L, store.getCompactionMaxOutputNodes());
+
+ // Also re-tune to a different value, simulating a live operator change.
+ store.setCompactionMaxOutputNodes(500_000L);
+ assertEquals(500_000L, store.getCompactionMaxOutputNodes());
+
+ // Long.MAX_VALUE passes through unchanged (a positive value).
+ store.setCompactionMaxOutputNodes(Long.MAX_VALUE);
+ assertEquals(Long.MAX_VALUE, store.getCompactionMaxOutputNodes());
+ }
+ }
+
+ @Test
+ public void zeroExplicitlyDisablesTheCap() throws Exception {
+ Path baseDir = tmpFolder.newFolder("data").toPath();
+ Path tmpDir = tmpFolder.newFolder("tmp").toPath();
+ FileDataStorageManager dsm = new FileDataStorageManager(baseDir);
+ dsm.initTablespace(TABLE_SPACE);
+ try (PersistentVectorStore store = createStore(tmpDir, dsm)) {
+ // Start enabled, then disable via 0.
+ store.setCompactionMaxOutputNodes(123_456_789L);
+ assertEquals(123_456_789L, store.getCompactionMaxOutputNodes());
+
+ store.setCompactionMaxOutputNodes(0L);
+ assertEquals("0 must disable the cap", 0L,
+ store.getCompactionMaxOutputNodes());
+ }
+ }
+
+ @Test
+ public void negativeValueClampsToZero() throws Exception {
+ Path baseDir = tmpFolder.newFolder("data").toPath();
+ Path tmpDir = tmpFolder.newFolder("tmp").toPath();
+ FileDataStorageManager dsm = new FileDataStorageManager(baseDir);
+ dsm.initTablespace(TABLE_SPACE);
+ try (PersistentVectorStore store = createStore(tmpDir, dsm)) {
+ // Negative values are not meaningful for a node count; the setter
+ // must clamp to 0 (disabled) rather than store a nonsense value
+ // that the picker would then have to defensively re-check.
+ store.setCompactionMaxOutputNodes(-5L);
+ assertEquals("negative must clamp to 0 (disabled)",
+ 0L, store.getCompactionMaxOutputNodes());
+
+ store.setCompactionMaxOutputNodes(Long.MIN_VALUE);
+ assertEquals("Long.MIN_VALUE must clamp to 0 (disabled)",
+ 0L, store.getCompactionMaxOutputNodes());
+
+ // After a clamp, a subsequent positive set must still work
+ // (defensive: pin that the clamp does not poison the field).
+ store.setCompactionMaxOutputNodes(1_000_000L);
+ assertEquals(1_000_000L, store.getCompactionMaxOutputNodes());
+ }
+ }
+}
diff --git a/herddb-indexing-service/src/test/java/herddb/indexing/vector/VectorIndexCompactorMaxOutputNodesTest.java b/herddb-indexing-service/src/test/java/herddb/indexing/vector/VectorIndexCompactorMaxOutputNodesTest.java
new file mode 100644
index 00000000..37868d7b
--- /dev/null
+++ b/herddb-indexing-service/src/test/java/herddb/indexing/vector/VectorIndexCompactorMaxOutputNodesTest.java
@@ -0,0 +1,356 @@
+/*
+ Licensed to Diennea S.r.l. under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. Diennea S.r.l. licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/
+
+package herddb.indexing.vector;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+import java.util.ArrayList;
+import java.util.List;
+import org.junit.Test;
+
+/**
+ * Policy tests for the per-cycle output-node cap added to
+ * {@link VectorIndexCompactor#chooseSegmentsToMerge} in issue #643.
+ *
+ * The cap bounds the sum of {@link VectorSegment#size()} (live-vector count
+ * per input, which equals the live-node count contributed to the merged output)
+ * across the inputs of a single cycle, so per-cycle wall-clock time tracks the
+ * per-cycle work budget rather than the table size. The cap is applied AFTER
+ * the trigger / smallest-first / micro-segment / graduation-cap selection
+ * logic, never changes whether a cycle compacts, and always keeps at least 2
+ * inputs for a meaningful merge. Bypassed by the micro-segment fast path.
+ */
+public class VectorIndexCompactorMaxOutputNodesTest {
+
+ /**
+ * Helper: build a candidate with {@code sizeBytes} bytes and
+ * {@code liveNodes} live vectors. Both are independently set so a test
+ * can configure each candidate's contribution to the smallest-first
+ * byte sort (sizeBytes) and the output-node cap (liveNodes).
+ */
+ private static VectorSegment seg(int id, long sizeBytes, int liveNodes) {
+ VectorSegment s = new VectorSegment(id);
+ s.estimatedSizeBytes = sizeBytes;
+ s.liveCount.set(liveNodes);
+ return s;
+ }
+
+ // -------------------------------------------------------------------------
+ // The cap is disabled by default
+ // -------------------------------------------------------------------------
+
+ @Test
+ public void disabledByDefaultPicksFullSelection() {
+ // 20 segments of 200k live each. With maxOutputNodes=0 (disabled) the
+ // legacy behaviour kicks in and all 20 are picked (no other cap fires).
+ List