Skip to content

issue #616: Phase B checkpoint guard — pre-track multipart files before upload - #618

Merged
eolivelli merged 2 commits into
masterfrom
issue-616-is-add-segment-inventory-and-candidate-s
May 20, 2026
Merged

eolivelli merged 2 commits into
masterfrom
issue-616-is-add-segment-inventory-and-candidate-s

Conversation

@eolivelli

Copy link
Copy Markdown
Owner

Fixes #616 (item 4 only — the checkpoint Phase B guard).

Changes

  • herddb-indexing-service/.../PersistentVectorStore.java (writeShardAsFusedPQSegment):
    • Move both trackProvisionalMultipartFile(segUuid, "graph") and trackProvisionalMultipartFile(segUuid, "map") calls to BEFORE the corresponding writeMultipartIndexFile() call. Pre-fix they ran only after a successful upload, so a mid-flight failure (e.g. Netty OutOfDirectMemoryError while writing block 0) left the partial multipart file unrecorded in provisionalMultipartFiles — the subsequent rollbackProvisionalArtefacts() then had no record of it and could not delete it, leaking partial blocks in remote storage. Those orphans poisoned every later compaction cycle with S3 NoSuchKeyException, which is the failure mode the issue's reproduction describes.
    • Wrap each writeMultipartIndexFile() call in a narrow catch (IOException | RuntimeException e) that logs SEVERE with the failing segUuid, segmentId and shard live-vector count before rethrowing. The outer Phase B catch at doCheckpointFusedPQThreePhase previously logged only "Phase B exception" with no shard identity, so under PHASE_B_SEGMENT_PARALLELISM > 1 operators could not tell which upload aborted. Inline comments explain the broad-RuntimeException tradeoff (Netty wraps allocation failures into unchecked exceptions, DataStorageManagerException is a RuntimeException).

The pre-track is correct under all branches: on Phase B + Phase C-prep success the tracker is cleared to null WITHOUT deletion (existing behaviour); on partial-write failure the rollback's deleteMultipartIndexFile call is a no-op against a missing key and the rollback path already swallows DataStorageManagerException / RuntimeException from such deletes.

Scope note

The issue describes four related improvements (segment inventory log, candidate-selection trace, automatic exclusion of READ_IO-failed segments, checkpoint Phase B guard). This PR addresses only item 4 — the root-cause checkpoint guard. The other three observability items belong in a separate follow-up PR; mixing them would inflate review surface for a fix that is itself surgical.

Tests

  • Issue616PhaseBPartialUploadRollbackTest (new) — injects a MemoryDataStorageManager subclass whose writeMultipartIndexFile performs the underlying put AND THEN throws IOException, simulating "block N committed to remote storage but block N+1 aborts". The test:
    1. Drives a baseline successful checkpoint and snapshots the DSM's multipart-file set.
    2. Arms the DSM to fail on the next upload, then triggers a multi-segment Phase B checkpoint.
    3. Asserts Phase B failed (thrown or consecutiveCheckpointFailures bumped), the DSM's multipart-file set is unchanged across the failing checkpoint (no orphan leak — fails pre-fix), no half-segment leaked into this.segments, the SEVERE log names the failing segUuid, and at least one deleteMultipartIndexFile call was issued during rollback.
    4. Verified to catch the regression: with the production change reverted the test fails with Orphaned: [tstblspace/testidx_testtable_..._seg1/graph] — exactly the leak the fix closes.

Test plan

  • mvn -pl herddb-indexing-service -Dtest='Issue616PhaseBPartialUploadRollbackTest' test — new regression test, passes (and fails pre-fix as expected).
  • mvn -pl herddb-indexing-service -Dtest='PhaseCPrepRollbackSafetyTest,PersistentVectorStoreCheckpointPhaseBDeleteRecallTest,PersistentVectorStoreCheckpointPhaseBLocalScopeTest,PersistentVectorStoreCheckpointPoolTest,PersistentVectorStoreConcurrentCheckpointTest,PersistentVectorStorePhaseCDeletePkRaceTest,Issue235FrozenSearchDuringCheckpointTest,Issue256CheckpointFormatTest,Issue256PhaseCShardTeardownTest' test — 18 tests across the existing Phase B / Phase C / concurrent-checkpoint regression set, all pass.
  • mvn -B spotless:apply -DskipTests then mvn -B spotless:check apache-rat:check spotbugs:check install -DskipTests -Pci — green.

The herddb-core DirectMultipleConcurrentUpdatesSuite and the herddb-utils BLink hammer are deliberately not run: they exercise core DML / primary-key index / file DSM paths, not the vector-index Phase B checkpoint code path touched here. Coverage for the vector-index concurrent-checkpoint path comes from PersistentVectorStoreConcurrentCheckpointTest in the regression batch above.

🤖 Implemented by the pr-worker agent.

eolivelli and others added 2 commits May 20, 2026 18:10
…re upload

Move trackProvisionalMultipartFile() calls in writeShardAsFusedPQSegment
to BEFORE the corresponding writeMultipartIndexFile() call so that
mid-flight upload failures (e.g. OutOfDirectMemoryError from Netty's
direct-buffer pool while writing block 0) leave the partial multipart
file registered in provisionalMultipartFiles. The outer Phase B catch
arm's rollbackProvisionalArtefacts() then deletes the partial blocks
instead of leaking them indefinitely — which previously poisoned every
subsequent compaction cycle with NoSuchKeyException on the orphan
(issue #616 reproduction).

Also wrap each writeMultipartIndexFile() call in a narrow catch that
logs SEVERE with the failing segUuid, segmentId and vector count before
rethrowing. With PHASE_B_SEGMENT_PARALLELISM > 1 the outer
"Phase B exception" log line previously hid shard identity; the new
log names the victim segment so operators can identify which upload
aborted from the IS log stream.

The pre-track is correct under all branches: on Phase B + Phase C-prep
success the tracker is cleared to null without deletion (existing
behaviour at line 5760); on partial-write failure the rollback's
deleteMultipartIndexFile call is a no-op against a missing key and the
rollback path already swallows DataStorageManagerException /
RuntimeException from such deletes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Addresses the pr-reviewer findings on PR #618:

1. Broaden the inner SEVERE catch arms in writeShardAsFusedPQSegment from
   `catch (IOException | RuntimeException)` to `catch (Throwable)`. Netty's
   OutOfDirectMemoryError extends OutOfMemoryError extends Error — the exact
   failure mode cited in the issue's reproduction — and was previously
   skipped by the diagnostic log. We rethrow the original Throwable
   unchanged so the JVM's Error-handling semantics are preserved.

2. Broaden the outer Phase B catch in doCheckpointFusedPQThreePhase from
   `catch (IOException | RuntimeException)` to `catch (IOException |
   RuntimeException | Error)`. In the parallel Phase B path,
   awaitAllOrFirstFailure already wraps non-IOException/non-RuntimeException
   Throwables into an IOException, so the existing catch sufficed; but in
   the SERIAL path (shardTasks.size() == 1) an Error propagated directly,
   bypassing rollbackProvisionalArtefacts() and recoverFromPhaseBFailure().
   Now Error drives the full rollback pipeline in both paths.

3. Add clarifying comments at the two sibling track-after-success sites the
   reviewer flagged:
     - writeStreamingCompactedSegment (compaction code path):
       provisionalMultipartFiles is null during compaction; orphan cleanup
       runs via VectorIndexCompactor.CompactionException.orphanPaths +
       runCompactionCycle's pendingDeletes pipeline — pre-track would
       double-clean.
     - writeOneSegmentData (reachable only via buildSegmentsInParallel,
       which has no in-tree caller — dormant code): documented so a future
       re-enablement adopts the same pre-track + Throwable-catch layout as
       the patched live path.

4. Tighten Issue616PhaseBPartialUploadRollbackTest:
     - Replace the "threw OR counter bumped" assertion with two separate
       assertions requiring both — under the fix both must always hold.
     - Drop the trivially-true assertion that the SEVERE log does not
       contain "Phase B: shard segment" (the success line is logged at
       INFO and the captor only records SEVERE+).
     - Add a new phaseBPartialUploadUnderErrorRolledBack test case that
       throws OutOfMemoryError (mirroring Netty's OutOfDirectMemoryError
       class hierarchy) and asserts the diagnostic SEVERE log fires, the
       orphan is cleaned up, and the segment registry is untouched. This
       case fails pre-fix on both the Throwable-catch and outer-Error
       gaps from items 1 and 2 above.

5. Move uploadingActive.incrementAndGet() inside the try block so a
   hypothetical throw from trackProvisionalMultipartFile cannot leak the
   active-upload counter (minor defensive cleanup).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@eolivelli
eolivelli merged commit db98e5d into master May 20, 2026
8 checks passed
@eolivelli
eolivelli deleted the issue-616-is-add-segment-inventory-and-candidate-s branch May 20, 2026 17:13
eolivelli added a commit that referenced this pull request May 20, 2026
…es not leak orphan multipart blocks (#624)

Fixes #621.

Symmetric compaction-side fix for the bug class that #616 / #618 fixed
on the checkpoint side. Netty surfaces direct-buffer allocation failures
as `OutOfDirectMemoryError extends OutOfMemoryError extends Error`.
Several catch arms on the compaction code path were narrower than the
`Error` class hierarchy, so a Netty OOM mid multipart-upload during
compaction:

1. Bypassed the orphan-list wrap into `CompactionException`.
2. The outer `runCompactionCycle` catch arms never saw the orphans →
partial multipart blocks leaked in remote storage indefinitely.
3. The `Error` propagated up to `vectorIndexCompactionLoop`'s `catch
(RuntimeException)` safety net — also too narrow — and killed the
compaction daemon thread, stopping compaction until IS restart.

## Changes

Production-only (no new tests per direction):

- **`VectorIndexCompactor.rebuildSegmentStreaming`** — broaden four
catch arms to include `Error`:
  - `OnDiskGraphIndexCompactor` constructor + `compact()` (line ~1301)
  - `writeStreamingCompactedMapFile` (line ~1315)
- `writeStreamingCompactedSegment` (line ~1325) — **the main case from
the issue**: a Netty OOM mid multipart-write now produces the
orphan-list wrap that `runCompactionCycle`'s `catch
(CompactionException)` arm drains into `pendingDeletes`
- `preloadCompactedSegment` (line ~1367) — post-upload, with orphans
known
- **`VectorIndexCompactor.rebuildSegmentLegacy`** — broaden the matching
three catch arms (`builder.cleanup`, `writeSyntheticShard`,
`preloadCompactedSegment`) for symmetry with the streaming path.
- **`PersistentVectorStore.vectorIndexCompactionLoop`** — broaden the
safety-net catch from `RuntimeException` to `RuntimeException | Error`.
This is the daemon-thread lifecycle catch (analogous to the outer Phase
B broadening in #618). The catch already logs SEVERE, bumps
`compactionConsecutiveFailures`, and lets the loop's `computeBackoffMs`
back-off keep a persistent `Error` source from spinning the thread. We
do NOT rethrow `Error` here — it would end the only retry path the
compaction subsystem has.

Each broadened arm carries an inline comment per CLAUDE.md guidance
explaining why the broad `Error` catch is required. No `Error` is ever
silently swallowed: every arm either rethrows wrapped as
`CompactionException` (per-step catches) or rethrows nothing while
logging SEVERE + bumping the failure counter (the daemon safety net).

## Tests

Skipped per user direction. The regression batch below was run to
confirm nothing existing broke.

## Test plan
- [x] `mvn -pl herddb-indexing-service
-Dtest='Issue354TieredCompactionTest,Issue285SegmentCountCompactionTest,Issue570MicroSegmentCompactionTest,Issue587CompactionInputCapTest,EagerDownloadCompactionTest,Issue535DropSegmentByUuidRaceTest,Issue256CompactionReleaseTest,CompactionDeprecateInputsFailureSafetyTest,Issue616PhaseBPartialUploadRollbackTest,RemoteSegmentGraphMergerStreamingTest,RemoteSegmentGraphMergerUploadFailureTest,RemoteSegmentGraphMergerCorruptInputTest'
test` — 49 tests across the existing compaction + Phase B regression
set, all pass.
- [x] `mvn -B spotless:apply -DskipTests` then `mvn -B spotless:check
apache-rat:check spotbugs:check install -DskipTests -Pci` — green.

🤖 Implemented by the `pr-worker` agent.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[IS] Add segment inventory and candidate-selection logging to compaction cycle

1 participant