Skip to content

issue #640: expose real-time compaction progress via describe-index / engine-stats - #641

Merged
eolivelli merged 2 commits into
masterfrom
issue-640-is-describe-index-compaction-fields-are
May 22, 2026
Merged

eolivelli merged 2 commits into
masterfrom
issue-640-is-describe-index-compaction-fields-are

Conversation

@eolivelli

Copy link
Copy Markdown
Owner

Fixes #640.

Root cause

Streaming compactions on the IS side ran for minutes without surfacing any
progress: describe-index --json showed the previous cycle's terminal
"100% done" values for the whole new cycle, and compaction_phase read as
"idle" even while jvector's OnDiskGraphIndexCompactor was writing 13k+
batches to S3.

Two gaps in the IS code path were responsible:

  1. VectorIndexCompactor.rebuildSegmentStreaming called
    compactor.compact(graphTemp) without a CompactionProgressListener,
    so jvector's runBatchesWithBackpressure counter (the one that the IS
    log already prints as "Compaction I/O progress: X/Y batches written to disk") never reached the IS counters.
  2. Phase tracking was blind during the batch loop —
    writingGraphActive / uploadingActive were only flipped by the
    surrounding writeStreamingCompactedSegment, which runs AFTER the
    batch loop — and the four legacy compaction counters were reset
    only inside doCheckpointFusedPQThreePhase, never on the streaming
    compaction path.

Changes

  • New atomic counters on PersistentVectorStore: compactionBatchesDone,
    compactionBatchesTotal, compactionCycleId,
    compactionInputSegmentCount, compactionInputVectorCount,
    compactionStartedNanos, plus a compactionStreamingActive flag for
    phase tracking.
  • beginCompactionCycle(inputSegmentCount, inputVectorCount) /
    endCompactionCycle() helpers that own per-cycle reset semantics,
    cycle-id bump and elapsed-ms tracking. Both runCompactionCycle and
    the checkpoint Phase B path now go through these helpers; the legacy
    in-line counter reset in doCheckpointFusedPQThreePhase is gone.
  • VectorIndexCompactor.rebuildSegmentStreaming now wraps the
    compactor.compact(...) call in incrementCompactionStreamingActive()
    / decrementCompactionStreamingActive() and passes a
    CompactionProgressListener that pushes (completed, total) from
    every runBatchesWithBackpressure tick into the new batch counters.
  • getCompactionPhase() returns a new "compacting-graph" value while
    streaming-active is non-zero (priority: uploading-segment >
    compacting-graph > writing-graph > idle). The progress
    percent falls back to batch-based progress in this phase.
  • 7 new fields on DescribeIndexResponse (tags 32–38):
    compaction_batches_done/total, compaction_cycle_id,
    compaction_input_segment_count/vector_count,
    compaction_elapsed_ms, compaction_running.
  • GetEngineStats aggregates compaction_running (any) and
    compaction_phase (most advanced) across all loaded
    PersistentVectorStores, so a single top-level call reveals whether
    any index on the IS is currently compacting.
  • indexing-admin describe-index --json emits the seven new fields.

Tests

  • PersistentVectorStoreCompactionCycleResetTest — unit-level
    coverage of beginCompactionCycle's per-cycle reset, monotonic cycle
    id, idempotent endCompactionCycle, negative-clamp, and the new
    compacting-graph phase string.
  • StreamingCompactionISLiveProgressTest — drives a real
    runCompactionCycle() through VectorIndexCompactor.rebuildSegmentStreaming;
    a watcher thread asserts that getCompactionPhase() reports
    compacting-graph (NOT idle — the issue's central symptom),
    batches counters move from 0 to non-zero, cycle id bumps and elapsed-ms
    reads 0 once the cycle ends.
  • IndexingServiceDiagnosticsGrpcCompactionFieldsTest — gRPC-level
    round-trip: registers a real PVS, runs a streaming cycle, asserts the
    new seven fields propagate end-to-end through
    IndexingServiceImpl.describeIndex to the proto response and that
    engine-stats reflects the post-cycle idle state.
  • IndexingServiceEngineStatsCompactionTest — verifies the
    aggregated compaction_running=true flips during a cycle and back to
    false afterwards, and that any non-idle compaction_phase observed
    is one of the recognised values.
  • IndexingServiceDiagnosticsGrpcTest augmented with assertions for the
    new defaults (0/false) on an in-memory store and for the new
    compaction_running/compaction_phase keys on engine-stats.

All 44 compaction-related tests in herddb-indexing-service pass
locally. Pre-PR validation (mvn spotless:check apache-rat:check spotbugs:check install -Pci -DskipTests) is green.

🤖 Implemented by the `pr-worker` agent.

eolivelli and others added 2 commits May 22, 2026 09:59
… engine-stats

Streaming compactions on the IS side ran for minutes without surfacing any
progress: describe-index showed the previous cycle's terminal "100% done"
values for the whole new cycle, and compaction_phase read as "idle" even
while jvector's OnDiskGraphIndexCompactor was writing 13k+ batches to S3.

Wire a CompactionProgressListener into OnDiskGraphIndexCompactor.compact()
from VectorIndexCompactor.rebuildSegmentStreaming so the per-batch counter
that the IS log already prints ("Compaction I/O progress: X/Y batches") is
pushed into new atomic counters on PersistentVectorStore. Flip a new
streaming-active flag around the call so getCompactionPhase() reports a
new "compacting-graph" value during the batch loop. Add a single
beginCompactionCycle / endCompactionCycle pair on PVS that resets the
four progress counters at cycle start (the staleness root cause), bumps
a monotonic cycle id, captures input segment / vector count, and starts
the elapsed-ms clock. Both runCompactionCycle and the checkpoint Phase B
path now go through these helpers, so the legacy in-line resets in
doCheckpointFusedPQThreePhase are replaced.

Surface the new state through DescribeIndexResponse (7 new proto fields)
and aggregate compaction_running + compaction_phase across all loaded
stores in GetEngineStats so a single top-level call reveals whether any
index on the IS is currently compacting.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1. Hoist endCompactionCycle() into a single top-level try/finally around
   the post-Phase-A body of doCheckpointFusedPQThreePhase. Removes three
   open-coded calls (Phase B catch, Phase C-prep catch, Stage 2 finally)
   that left a coverage hole: a throw from warmUpNewSegmentsBeforePublish,
   the Stage-1 deletes apply, or phaseCPostDeletesApplyHook would leave
   compactionStartedNanos non-zero so isCompactionRunning() reported
   true indefinitely — exactly the stale-state symptom #640 is fixing.

2. Swap the CompactionProgressListener lambda's set order to
   `done` before `total`, so a racing describe-index reader sees a
   strictly non-decreasing perceived progress percent on a new
   level boundary.

3. Throttle the watcher loops in
   IndexingServiceEngineStatsCompactionTest (Thread.sleep(2)) and
   StreamingCompactionISLiveProgressTest (Thread.sleep(1)). Unthrottled
   spins on a single-vCPU CI runner can starve the cycle thread; the
   engine-stats watcher in particular issues an RPC every iteration
   and would otherwise generate tens of thousands of calls per cycle.

4. Add PersistentVectorStoreCompactionCycleResetTest#runningFlagClearsOnPhaseCHookThrow
   — injects a RuntimeException via setPhaseCPostDeletesApplyHookForTest,
   confirms it propagates out of checkpoint(), and asserts
   isCompactionRunning() == false afterwards. This is the regression
   gate for the unprotected stretch the review identified.

Verified: all 64 indexing-service compaction-related tests pass,
DirectMultipleConcurrentUpdatesSuite hammer suite (NoIndexes / WithNonUniqueIndexes / WithUniqueIndexes) green on two consecutive runs, pre-PR validation (spotless / apache-rat / spotbugs) clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@eolivelli
eolivelli merged commit 6624c7e into master May 22, 2026
8 checks passed
@eolivelli
eolivelli deleted the issue-640-is-describe-index-compaction-fields-are branch May 22, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[IS] describe-index compaction fields are stale — real-time Phase B I/O progress not exposed via API

1 participant