Repository navigation
issue #616: Phase B checkpoint guard — pre-track multipart files before upload - #618
Merged
eolivelli merged 2 commits intoMay 20, 2026
Merged
Conversation
…re upload Move trackProvisionalMultipartFile() calls in writeShardAsFusedPQSegment to BEFORE the corresponding writeMultipartIndexFile() call so that mid-flight upload failures (e.g. OutOfDirectMemoryError from Netty's direct-buffer pool while writing block 0) leave the partial multipart file registered in provisionalMultipartFiles. The outer Phase B catch arm's rollbackProvisionalArtefacts() then deletes the partial blocks instead of leaking them indefinitely — which previously poisoned every subsequent compaction cycle with NoSuchKeyException on the orphan (issue #616 reproduction). Also wrap each writeMultipartIndexFile() call in a narrow catch that logs SEVERE with the failing segUuid, segmentId and vector count before rethrowing. With PHASE_B_SEGMENT_PARALLELISM > 1 the outer "Phase B exception" log line previously hid shard identity; the new log names the victim segment so operators can identify which upload aborted from the IS log stream. The pre-track is correct under all branches: on Phase B + Phase C-prep success the tracker is cleared to null without deletion (existing behaviour at line 5760); on partial-write failure the rollback's deleteMultipartIndexFile call is a no-op against a missing key and the rollback path already swallows DataStorageManagerException / RuntimeException from such deletes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Addresses the pr-reviewer findings on PR #618: 1. Broaden the inner SEVERE catch arms in writeShardAsFusedPQSegment from `catch (IOException | RuntimeException)` to `catch (Throwable)`. Netty's OutOfDirectMemoryError extends OutOfMemoryError extends Error — the exact failure mode cited in the issue's reproduction — and was previously skipped by the diagnostic log. We rethrow the original Throwable unchanged so the JVM's Error-handling semantics are preserved. 2. Broaden the outer Phase B catch in doCheckpointFusedPQThreePhase from `catch (IOException | RuntimeException)` to `catch (IOException | RuntimeException | Error)`. In the parallel Phase B path, awaitAllOrFirstFailure already wraps non-IOException/non-RuntimeException Throwables into an IOException, so the existing catch sufficed; but in the SERIAL path (shardTasks.size() == 1) an Error propagated directly, bypassing rollbackProvisionalArtefacts() and recoverFromPhaseBFailure(). Now Error drives the full rollback pipeline in both paths. 3. Add clarifying comments at the two sibling track-after-success sites the reviewer flagged: - writeStreamingCompactedSegment (compaction code path): provisionalMultipartFiles is null during compaction; orphan cleanup runs via VectorIndexCompactor.CompactionException.orphanPaths + runCompactionCycle's pendingDeletes pipeline — pre-track would double-clean. - writeOneSegmentData (reachable only via buildSegmentsInParallel, which has no in-tree caller — dormant code): documented so a future re-enablement adopts the same pre-track + Throwable-catch layout as the patched live path. 4. Tighten Issue616PhaseBPartialUploadRollbackTest: - Replace the "threw OR counter bumped" assertion with two separate assertions requiring both — under the fix both must always hold. - Drop the trivially-true assertion that the SEVERE log does not contain "Phase B: shard segment" (the success line is logged at INFO and the captor only records SEVERE+). - Add a new phaseBPartialUploadUnderErrorRolledBack test case that throws OutOfMemoryError (mirroring Netty's OutOfDirectMemoryError class hierarchy) and asserts the diagnostic SEVERE log fires, the orphan is cleaned up, and the segment registry is untouched. This case fails pre-fix on both the Throwable-catch and outer-Error gaps from items 1 and 2 above. 5. Move uploadingActive.incrementAndGet() inside the try block so a hypothetical throw from trackProvisionalMultipartFile cannot leak the active-upload counter (minor defensive cleanup). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced May 20, 2026
eolivelli
added a commit
that referenced
this pull request
May 20, 2026
…es not leak orphan multipart blocks (#624) Fixes #621. Symmetric compaction-side fix for the bug class that #616 / #618 fixed on the checkpoint side. Netty surfaces direct-buffer allocation failures as `OutOfDirectMemoryError extends OutOfMemoryError extends Error`. Several catch arms on the compaction code path were narrower than the `Error` class hierarchy, so a Netty OOM mid multipart-upload during compaction: 1. Bypassed the orphan-list wrap into `CompactionException`. 2. The outer `runCompactionCycle` catch arms never saw the orphans → partial multipart blocks leaked in remote storage indefinitely. 3. The `Error` propagated up to `vectorIndexCompactionLoop`'s `catch (RuntimeException)` safety net — also too narrow — and killed the compaction daemon thread, stopping compaction until IS restart. ## Changes Production-only (no new tests per direction): - **`VectorIndexCompactor.rebuildSegmentStreaming`** — broaden four catch arms to include `Error`: - `OnDiskGraphIndexCompactor` constructor + `compact()` (line ~1301) - `writeStreamingCompactedMapFile` (line ~1315) - `writeStreamingCompactedSegment` (line ~1325) — **the main case from the issue**: a Netty OOM mid multipart-write now produces the orphan-list wrap that `runCompactionCycle`'s `catch (CompactionException)` arm drains into `pendingDeletes` - `preloadCompactedSegment` (line ~1367) — post-upload, with orphans known - **`VectorIndexCompactor.rebuildSegmentLegacy`** — broaden the matching three catch arms (`builder.cleanup`, `writeSyntheticShard`, `preloadCompactedSegment`) for symmetry with the streaming path. - **`PersistentVectorStore.vectorIndexCompactionLoop`** — broaden the safety-net catch from `RuntimeException` to `RuntimeException | Error`. This is the daemon-thread lifecycle catch (analogous to the outer Phase B broadening in #618). The catch already logs SEVERE, bumps `compactionConsecutiveFailures`, and lets the loop's `computeBackoffMs` back-off keep a persistent `Error` source from spinning the thread. We do NOT rethrow `Error` here — it would end the only retry path the compaction subsystem has. Each broadened arm carries an inline comment per CLAUDE.md guidance explaining why the broad `Error` catch is required. No `Error` is ever silently swallowed: every arm either rethrows wrapped as `CompactionException` (per-step catches) or rethrows nothing while logging SEVERE + bumping the failure counter (the daemon safety net). ## Tests Skipped per user direction. The regression batch below was run to confirm nothing existing broke. ## Test plan - [x] `mvn -pl herddb-indexing-service -Dtest='Issue354TieredCompactionTest,Issue285SegmentCountCompactionTest,Issue570MicroSegmentCompactionTest,Issue587CompactionInputCapTest,EagerDownloadCompactionTest,Issue535DropSegmentByUuidRaceTest,Issue256CompactionReleaseTest,CompactionDeprecateInputsFailureSafetyTest,Issue616PhaseBPartialUploadRollbackTest,RemoteSegmentGraphMergerStreamingTest,RemoteSegmentGraphMergerUploadFailureTest,RemoteSegmentGraphMergerCorruptInputTest' test` — 49 tests across the existing compaction + Phase B regression set, all pass. - [x] `mvn -B spotless:apply -DskipTests` then `mvn -B spotless:check apache-rat:check spotbugs:check install -DskipTests -Pci` — green. 🤖 Implemented by the `pr-worker` agent. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #616 (item 4 only — the checkpoint Phase B guard).
Changes
herddb-indexing-service/.../PersistentVectorStore.java(writeShardAsFusedPQSegment):trackProvisionalMultipartFile(segUuid, "graph")andtrackProvisionalMultipartFile(segUuid, "map")calls to BEFORE the correspondingwriteMultipartIndexFile()call. Pre-fix they ran only after a successful upload, so a mid-flight failure (e.g. NettyOutOfDirectMemoryErrorwhile writing block 0) left the partial multipart file unrecorded inprovisionalMultipartFiles— the subsequentrollbackProvisionalArtefacts()then had no record of it and could not delete it, leaking partial blocks in remote storage. Those orphans poisoned every later compaction cycle withS3 NoSuchKeyException, which is the failure mode the issue's reproduction describes.writeMultipartIndexFile()call in a narrowcatch (IOException | RuntimeException e)that logs SEVERE with the failingsegUuid,segmentIdand shard live-vector count before rethrowing. The outer Phase B catch atdoCheckpointFusedPQThreePhasepreviously logged only "Phase B exception" with no shard identity, so underPHASE_B_SEGMENT_PARALLELISM > 1operators could not tell which upload aborted. Inline comments explain the broad-RuntimeException tradeoff (Netty wraps allocation failures into unchecked exceptions,DataStorageManagerExceptionis aRuntimeException).The pre-track is correct under all branches: on Phase B + Phase C-prep success the tracker is cleared to
nullWITHOUT deletion (existing behaviour); on partial-write failure the rollback'sdeleteMultipartIndexFilecall is a no-op against a missing key and the rollback path already swallowsDataStorageManagerException/RuntimeExceptionfrom such deletes.Scope note
The issue describes four related improvements (segment inventory log, candidate-selection trace, automatic exclusion of READ_IO-failed segments, checkpoint Phase B guard). This PR addresses only item 4 — the root-cause checkpoint guard. The other three observability items belong in a separate follow-up PR; mixing them would inflate review surface for a fix that is itself surgical.
Tests
Issue616PhaseBPartialUploadRollbackTest(new) — injects aMemoryDataStorageManagersubclass whosewriteMultipartIndexFileperforms the underlying put AND THEN throwsIOException, simulating "block N committed to remote storage but block N+1 aborts". The test:consecutiveCheckpointFailuresbumped), the DSM's multipart-file set is unchanged across the failing checkpoint (no orphan leak — fails pre-fix), no half-segment leaked intothis.segments, the SEVERE log names the failingsegUuid, and at least onedeleteMultipartIndexFilecall was issued during rollback.Orphaned: [tstblspace/testidx_testtable_..._seg1/graph]— exactly the leak the fix closes.Test plan
mvn -pl herddb-indexing-service -Dtest='Issue616PhaseBPartialUploadRollbackTest' test— new regression test, passes (and fails pre-fix as expected).mvn -pl herddb-indexing-service -Dtest='PhaseCPrepRollbackSafetyTest,PersistentVectorStoreCheckpointPhaseBDeleteRecallTest,PersistentVectorStoreCheckpointPhaseBLocalScopeTest,PersistentVectorStoreCheckpointPoolTest,PersistentVectorStoreConcurrentCheckpointTest,PersistentVectorStorePhaseCDeletePkRaceTest,Issue235FrozenSearchDuringCheckpointTest,Issue256CheckpointFormatTest,Issue256PhaseCShardTeardownTest' test— 18 tests across the existing Phase B / Phase C / concurrent-checkpoint regression set, all pass.mvn -B spotless:apply -DskipTeststhenmvn -B spotless:check apache-rat:check spotbugs:check install -DskipTests -Pci— green.The
herddb-coreDirectMultipleConcurrentUpdatesSuiteand theherddb-utilsBLink hammer are deliberately not run: they exercise core DML / primary-key index / file DSM paths, not the vector-index Phase B checkpoint code path touched here. Coverage for the vector-index concurrent-checkpoint path comes fromPersistentVectorStoreConcurrentCheckpointTestin the regression batch above.🤖 Implemented by the
pr-workeragent.