Repository navigation
issue #640: expose real-time compaction progress via describe-index / engine-stats - #641
Merged
eolivelli merged 2 commits intoMay 22, 2026
Merged
Conversation
… engine-stats
Streaming compactions on the IS side ran for minutes without surfacing any
progress: describe-index showed the previous cycle's terminal "100% done"
values for the whole new cycle, and compaction_phase read as "idle" even
while jvector's OnDiskGraphIndexCompactor was writing 13k+ batches to S3.
Wire a CompactionProgressListener into OnDiskGraphIndexCompactor.compact()
from VectorIndexCompactor.rebuildSegmentStreaming so the per-batch counter
that the IS log already prints ("Compaction I/O progress: X/Y batches") is
pushed into new atomic counters on PersistentVectorStore. Flip a new
streaming-active flag around the call so getCompactionPhase() reports a
new "compacting-graph" value during the batch loop. Add a single
beginCompactionCycle / endCompactionCycle pair on PVS that resets the
four progress counters at cycle start (the staleness root cause), bumps
a monotonic cycle id, captures input segment / vector count, and starts
the elapsed-ms clock. Both runCompactionCycle and the checkpoint Phase B
path now go through these helpers, so the legacy in-line resets in
doCheckpointFusedPQThreePhase are replaced.
Surface the new state through DescribeIndexResponse (7 new proto fields)
and aggregate compaction_running + compaction_phase across all loaded
stores in GetEngineStats so a single top-level call reveals whether any
index on the IS is currently compacting.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1. Hoist endCompactionCycle() into a single top-level try/finally around the post-Phase-A body of doCheckpointFusedPQThreePhase. Removes three open-coded calls (Phase B catch, Phase C-prep catch, Stage 2 finally) that left a coverage hole: a throw from warmUpNewSegmentsBeforePublish, the Stage-1 deletes apply, or phaseCPostDeletesApplyHook would leave compactionStartedNanos non-zero so isCompactionRunning() reported true indefinitely — exactly the stale-state symptom #640 is fixing. 2. Swap the CompactionProgressListener lambda's set order to `done` before `total`, so a racing describe-index reader sees a strictly non-decreasing perceived progress percent on a new level boundary. 3. Throttle the watcher loops in IndexingServiceEngineStatsCompactionTest (Thread.sleep(2)) and StreamingCompactionISLiveProgressTest (Thread.sleep(1)). Unthrottled spins on a single-vCPU CI runner can starve the cycle thread; the engine-stats watcher in particular issues an RPC every iteration and would otherwise generate tens of thousands of calls per cycle. 4. Add PersistentVectorStoreCompactionCycleResetTest#runningFlagClearsOnPhaseCHookThrow — injects a RuntimeException via setPhaseCPostDeletesApplyHookForTest, confirms it propagates out of checkpoint(), and asserts isCompactionRunning() == false afterwards. This is the regression gate for the unprotected stretch the review identified. Verified: all 64 indexing-service compaction-related tests pass, DirectMultipleConcurrentUpdatesSuite hammer suite (NoIndexes / WithNonUniqueIndexes / WithUniqueIndexes) green on two consecutive runs, pre-PR validation (spotless / apache-rat / spotbugs) clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #640.
Root cause
Streaming compactions on the IS side ran for minutes without surfacing any
progress:
describe-index --jsonshowed the previous cycle's terminal"100% done" values for the whole new cycle, and
compaction_phaseread as"idle"even while jvector'sOnDiskGraphIndexCompactorwas writing 13k+batches to S3.
Two gaps in the IS code path were responsible:
VectorIndexCompactor.rebuildSegmentStreamingcalledcompactor.compact(graphTemp)without aCompactionProgressListener,so jvector's
runBatchesWithBackpressurecounter (the one that the ISlog already prints as
"Compaction I/O progress: X/Y batches written to disk") never reached the IS counters.writingGraphActive/uploadingActivewere only flipped by thesurrounding
writeStreamingCompactedSegment, which runs AFTER thebatch loop — and the four legacy compaction counters were reset
only inside
doCheckpointFusedPQThreePhase, never on the streamingcompaction path.
Changes
PersistentVectorStore:compactionBatchesDone,compactionBatchesTotal,compactionCycleId,compactionInputSegmentCount,compactionInputVectorCount,compactionStartedNanos, plus acompactionStreamingActiveflag forphase tracking.
beginCompactionCycle(inputSegmentCount, inputVectorCount)/endCompactionCycle()helpers that own per-cycle reset semantics,cycle-id bump and elapsed-ms tracking. Both
runCompactionCycleandthe checkpoint Phase B path now go through these helpers; the legacy
in-line counter reset in
doCheckpointFusedPQThreePhaseis gone.VectorIndexCompactor.rebuildSegmentStreamingnow wraps thecompactor.compact(...)call inincrementCompactionStreamingActive()/
decrementCompactionStreamingActive()and passes aCompactionProgressListenerthat pushes(completed, total)fromevery
runBatchesWithBackpressuretick into the new batch counters.getCompactionPhase()returns a new"compacting-graph"value whilestreaming-active is non-zero (priority:
uploading-segment>compacting-graph>writing-graph>idle). The progresspercent falls back to batch-based progress in this phase.
DescribeIndexResponse(tags 32–38):compaction_batches_done/total,compaction_cycle_id,compaction_input_segment_count/vector_count,compaction_elapsed_ms,compaction_running.GetEngineStatsaggregatescompaction_running(any) andcompaction_phase(most advanced) across all loadedPersistentVectorStores, so a single top-level call reveals whetherany index on the IS is currently compacting.
indexing-admin describe-index --jsonemits the seven new fields.Tests
PersistentVectorStoreCompactionCycleResetTest— unit-levelcoverage of
beginCompactionCycle's per-cycle reset, monotonic cycleid, idempotent
endCompactionCycle, negative-clamp, and the newcompacting-graphphase string.StreamingCompactionISLiveProgressTest— drives a realrunCompactionCycle()throughVectorIndexCompactor.rebuildSegmentStreaming;a watcher thread asserts that
getCompactionPhase()reportscompacting-graph(NOTidle— the issue's central symptom),batches counters move from 0 to non-zero, cycle id bumps and elapsed-ms
reads 0 once the cycle ends.
IndexingServiceDiagnosticsGrpcCompactionFieldsTest— gRPC-levelround-trip: registers a real PVS, runs a streaming cycle, asserts the
new seven fields propagate end-to-end through
IndexingServiceImpl.describeIndexto the proto response and thatengine-statsreflects the post-cycle idle state.IndexingServiceEngineStatsCompactionTest— verifies theaggregated
compaction_running=trueflips during a cycle and back tofalseafterwards, and that any non-idlecompaction_phaseobservedis one of the recognised values.
IndexingServiceDiagnosticsGrpcTestaugmented with assertions for thenew defaults (0/false) on an in-memory store and for the new
compaction_running/compaction_phasekeys on engine-stats.All 44 compaction-related tests in
herddb-indexing-servicepasslocally. Pre-PR validation (
mvn spotless:check apache-rat:check spotbugs:check install -Pci -DskipTests) is green.🤖 Implemented by the `pr-worker` agent.