Repository navigation
issue #638: direct-write to S3/MinIO via S3TransferManager during compaction - #639
Merged
eolivelli merged 3 commits intoMay 21, 2026
Merged
Conversation
…paction, bypassing file server Introduce a direct S3 upload path in RemoteFileDataStorageManager that bypasses the gRPC file-server during checkpoint/compaction writes when a direct ObjectStorage is available. Files are stored as a single S3 object at the logical path suffixed with ".bulk", using S3TransferManager (real S3 Multipart Upload with CRT-pipelined parts) for maximum throughput. Legacy per-block files remain readable via an isBulkLayout() probe-and- fallback mechanism with a ConcurrentHashMap probe cache to avoid per-read HEAD round-trips. An independent backpressure semaphore (separate from the gRPC client's inflight-write budget) bounds concurrent direct uploads. Both IndexingServer and IndexOptimizerMain wire the enableDirectUpload() call after setDirectObjectStorage(), controlled by new config keys: indexing.s3.direct.write.enabled (default: true) indexing.remote.file.client.max.inflight.direct.write.bytes (default: 512 MiB) indexoptimizer.s3.direct.write.enabled (default: true) indexoptimizer.remote.file.client.max.inflight.direct.write.bytes (default: 512 MiB) New tests: direct-upload unit tests, backpressure concurrency tests, legacy-fallback tests, IS/optimizer wiring tests, and a MinIO Testcontainers integration test for end-to-end S3 multipart upload.
- bulkLayoutCache: cache only positive results to prevent write/probe race from permanently pinning a logical path as "not bulk" for the JVM lifetime. A false probe result now returns without touching the cache so a concurrent writer's put(TRUE) is never overwritten by a stale probe. - enableDirectUpload: add idempotency guard so repeated calls with the same cap are no-ops (prevents discarding an in-use semaphore). - reserveDirectInflightUploadBytes: replace unbounded acquireUninterruptibly with tryAcquire(10 min, MINUTES) to surface stuck CRT futures as IOException instead of blocking forever. - Reduce HEAD probe and bulk delete timeouts from 60 s to 15 s to avoid stalling vector-search reads for a full minute on slow S3 backends. - Flip PROPERTY_S3_DIRECT_WRITE_ENABLED_DEFAULT from true to false in both IndexingServerConfiguration and OptimizerConfiguration. Operators opt in explicitly so no IS pod silently changes its on-disk segment format on upgrade. - Backpressure test: replace Thread.sleep(200L) "confirm B is blocked" with active wait on heldCount()==1 (proves B has NOT called uploadFile yet, i.e. is genuinely blocked at the semaphore). Raise 1 s busy-wait ceilings to 10 s for slow CI runners. - Tests: rename perBlockOnlyFileIsDiscoverable to perBlockOnlyFileIsNotDiscoverableWithoutGrpc. Replace tautological assertTrue(true) in deleteLegacyOnlyFileSucceeds with real post-conditions. Tighten S3DirectBulkUploadIT list assertion to assertEquals(1, size()). Add enableDirectUploadIsIdempotentForSameCap and cacheRaceWriteThenProbeDoesNotPinNegativeResult tests. - Update config tests for default=false in both IS and optimizer wiring tests.
- ensureBulkLocalCacheFile: download to a unique UUID-named temp file per call instead of a deterministic path, preventing concurrent callers from interleaving bytes into the same destination. putIfAbsent elects the winner; losers delete their temp files. - Add proper imports (FilterInputStream, AtomicBoolean, TimeUnit, Files, UUID, InputStream) and drop all fully-qualified references introduced by this patch from RemoteFileDataStorageManager.java. - Fix stale Javadoc on writeMultipartIndexFileDirect: "chunked acquire to allow smaller concurrent uploads to interleave" was inaccurate — the semaphore reserves the full clamped count for the duration of the upload. - probeCacheIsPopulatedAfterFirstProbe: replace weak byte-equality assertions with a direct existsObject counter (via CountingLocalObjectStorage inner class) that proves exactly one HEAD probe fires on first positive hit and zero additional probes on subsequent cache-hit calls. - Fix stale "caches negative result for JVM lifetime" docstring to reflect the positive-only caching policy introduced in the previous review. - S3DirectBulkUploadIT: drop tautological assertNotNull(long) and strengthen plain-storage assertion to assertTrue(existsObject(...)). - All hammer suites (DirectMultipleConcurrentUpdatesSuite*, BLinkConcurrentSearchInsertTest, LazyValueCacheConcurrentUpdatesSuite*) pass green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #638.
Changes
herddb-remote-file-service/src/main/java/herddb/remote/storage/S3TransferManagerFactory.java(NEW): factory helper that wraps a CRTS3AsyncClientin anS3TransferManager; keeps client ownership separate (TM close does not close the client)herddb-remote-file-service/src/main/java/herddb/remote/storage/ObjectStorage.java: three newdefaultmethods —uploadFile()(TM-based multipart upload),existsObject()(HEAD probe),downloadFileBulk()(TM-based bulk download);S3ObjectStorageoverrides all threeherddb-remote-file-service/src/main/java/herddb/remote/storage/S3ObjectStorage.java: accepts an optionalS3TransferManager; implementsuploadFile()viatm.uploadFile()(real S3 Multipart Upload with CRT-pipelined parts),existsObject()viaheadObject,downloadFileBulk()viatm.downloadFile(); innerProgressForwardingListenerconverts TM's cumulative byte counter to deltasherddb-remote-file-service/src/main/java/herddb/remote/RemoteFileDataStorageManager.java: direct-write path —enableDirectUpload(long)wires up an independent backpressureSemaphore;writeMultipartIndexFile()branches towriteMultipartIndexFileDirect()when enabled; bulk layout (.bulksuffix single S3 object) with probe-and-fallback for legacy per-block files;isBulkLayout()caches only positive results (false probes not cached, preventing write/probe race from permanently pinning wrong layout);ensureBulkLocalCacheFile()downloads to UUID-named temp files to prevent concurrent callers corrupting the same destination;close()cleans up local cache filesherddb-core/src/main/java/herddb/storage/DataStorageManager.java: addssupportsDirectMultipartUpload()symmetric to the existing download flagherddb-indexing-service/src/main/java/herddb/indexing/IndexingServerConfiguration.java: new config keysindexing.s3.direct.write.enabled(defaultfalse— opt-in for safe rollout) andindexing.remote.file.client.max.inflight.direct.write.bytes(default 512 MiB)herddb-indexing-service/src/main/java/herddb/indexing/IndexingServer.java: buildsS3TransferManageralongside theS3AsyncClient; callsenableDirectUpload()aftersetDirectObjectStorage()when the write flag is enabledherddb-indexing-service/src/main/java/herddb/indexing/optimizer/OptimizerConfiguration.java: new config keysindexoptimizer.s3.direct.write.enabled(defaultfalse) andindexoptimizer.remote.file.client.max.inflight.direct.write.bytesherddb-indexing-service/src/main/java/herddb/indexing/optimizer/IndexOptimizerMain.java: same wiring as IS — builds TM, callsenableDirectUpload()frommaybeEnableDirectS3()Tests
RemoteFileDataStorageManagerDirectUploadTest: 11 unit tests covering write/read/exists/delete for bulk layout, cache invalidation on delete, idempotency ofenableDirectUpload, and the write/probe cache race that would previously have pinned a path as "not bulk"RemoteFileDataStorageManagerDirectUploadBackpressureTest: 2 concurrency tests — verifies the independent semaphore blocks a second concurrent upload when the cap is exhausted (with active wait onheldCount()for reliable detection), and that gRPC inflight-write budget is untouched by direct uploadsRemoteFileDataStorageManagerLegacyFallbackTest: 5 backward-compat tests — legacy per-block files download correctly, the "no-gRPC-server" exists probe returns false, bulk wins when both layouts coexist, delete of legacy file succeeds (with real post-conditions), and positive-result probe cache is verified viaCountingLocalObjectStoragewith exactexistsObjectcall countsS3TransferManagerFactoryTest: 2 unit tests verifying TM construction and that close() does not close the wrapped clientS3DirectBulkUploadIT: 5 MinIO Testcontainers integration tests — multipart upload produces a single object (verified vialist()size assertion), byte-range reads work on bulk objects, small files upload correctly, existsObject returns false for missing keys, upload without TM falls back to single PutObjectIndexingServerConfigurationDirectWriteTest: 3 config key/default tests for IS-server config (defaultfalse)IndexingServerConfigDirectUploadFlowTest: 4 IS-server wiring tests — default-false skips upload, explicit-true enables both setters, explicit-false disables only write, propagates custom inflight capIndexOptimizerMainDirectUploadTest: 5 optimizer wiring tests mirroring the IS-server testsHammer suites
All three hammer suites passed green:
DirectMultipleConcurrentUpdatesSuiteNoIndexesTest,DirectMultipleConcurrentUpdatesSuiteWithNonUniqueIndexesTest,DirectMultipleConcurrentUpdatesSuiteWithUniqueIndexesTest(12/12)BLinkConcurrentSearchInsertTest(1/1)LazyValueCacheConcurrentUpdatesSuiteNoIndexesTest,LazyValueCacheConcurrentUpdatesSuiteWithNonUniqueIndexesTest,LazyValueCacheConcurrentUpdatesSuiteWithUniqueIndexesTest(12/12)Rollout guidance
Both
indexing.s3.direct.write.enabledandindexoptimizer.s3.direct.write.enableddefault tofalse. To opt in, set the flag(s) totrueinvalues.yamlafter verifying the bulk upload path in your environment. Existing direct-read deployments (from issue #381) are unaffected.🤖 Implemented by the
pr-workeragent.