Skip to content

issue #638: direct-write to S3/MinIO via S3TransferManager during compaction - #639

Merged
eolivelli merged 3 commits into
masterfrom
issue-638-is-direct-write-to-s3minio-during-compac
May 21, 2026
Merged

eolivelli merged 3 commits into
masterfrom
issue-638-is-direct-write-to-s3minio-during-compac

Conversation

@eolivelli

@eolivelli eolivelli commented May 21, 2026 •

Copy link
Copy Markdown
Owner

Fixes #638.

Changes

  • herddb-remote-file-service/src/main/java/herddb/remote/storage/S3TransferManagerFactory.java (NEW): factory helper that wraps a CRT S3AsyncClient in an S3TransferManager; keeps client ownership separate (TM close does not close the client)
  • herddb-remote-file-service/src/main/java/herddb/remote/storage/ObjectStorage.java: three new default methods — uploadFile() (TM-based multipart upload), existsObject() (HEAD probe), downloadFileBulk() (TM-based bulk download); S3ObjectStorage overrides all three
  • herddb-remote-file-service/src/main/java/herddb/remote/storage/S3ObjectStorage.java: accepts an optional S3TransferManager; implements uploadFile() via tm.uploadFile() (real S3 Multipart Upload with CRT-pipelined parts), existsObject() via headObject, downloadFileBulk() via tm.downloadFile(); inner ProgressForwardingListener converts TM's cumulative byte counter to deltas
  • herddb-remote-file-service/src/main/java/herddb/remote/RemoteFileDataStorageManager.java: direct-write path — enableDirectUpload(long) wires up an independent backpressure Semaphore; writeMultipartIndexFile() branches to writeMultipartIndexFileDirect() when enabled; bulk layout (.bulk suffix single S3 object) with probe-and-fallback for legacy per-block files; isBulkLayout() caches only positive results (false probes not cached, preventing write/probe race from permanently pinning wrong layout); ensureBulkLocalCacheFile() downloads to UUID-named temp files to prevent concurrent callers corrupting the same destination; close() cleans up local cache files
  • herddb-core/src/main/java/herddb/storage/DataStorageManager.java: adds supportsDirectMultipartUpload() symmetric to the existing download flag
  • herddb-indexing-service/src/main/java/herddb/indexing/IndexingServerConfiguration.java: new config keys indexing.s3.direct.write.enabled (default false — opt-in for safe rollout) and indexing.remote.file.client.max.inflight.direct.write.bytes (default 512 MiB)
  • herddb-indexing-service/src/main/java/herddb/indexing/IndexingServer.java: builds S3TransferManager alongside the S3AsyncClient; calls enableDirectUpload() after setDirectObjectStorage() when the write flag is enabled
  • herddb-indexing-service/src/main/java/herddb/indexing/optimizer/OptimizerConfiguration.java: new config keys indexoptimizer.s3.direct.write.enabled (default false) and indexoptimizer.remote.file.client.max.inflight.direct.write.bytes
  • herddb-indexing-service/src/main/java/herddb/indexing/optimizer/IndexOptimizerMain.java: same wiring as IS — builds TM, calls enableDirectUpload() from maybeEnableDirectS3()

Tests

  • RemoteFileDataStorageManagerDirectUploadTest: 11 unit tests covering write/read/exists/delete for bulk layout, cache invalidation on delete, idempotency of enableDirectUpload, and the write/probe cache race that would previously have pinned a path as "not bulk"
  • RemoteFileDataStorageManagerDirectUploadBackpressureTest: 2 concurrency tests — verifies the independent semaphore blocks a second concurrent upload when the cap is exhausted (with active wait on heldCount() for reliable detection), and that gRPC inflight-write budget is untouched by direct uploads
  • RemoteFileDataStorageManagerLegacyFallbackTest: 5 backward-compat tests — legacy per-block files download correctly, the "no-gRPC-server" exists probe returns false, bulk wins when both layouts coexist, delete of legacy file succeeds (with real post-conditions), and positive-result probe cache is verified via CountingLocalObjectStorage with exact existsObject call counts
  • S3TransferManagerFactoryTest: 2 unit tests verifying TM construction and that close() does not close the wrapped client
  • S3DirectBulkUploadIT: 5 MinIO Testcontainers integration tests — multipart upload produces a single object (verified via list() size assertion), byte-range reads work on bulk objects, small files upload correctly, existsObject returns false for missing keys, upload without TM falls back to single PutObject
  • IndexingServerConfigurationDirectWriteTest: 3 config key/default tests for IS-server config (default false)
  • IndexingServerConfigDirectUploadFlowTest: 4 IS-server wiring tests — default-false skips upload, explicit-true enables both setters, explicit-false disables only write, propagates custom inflight cap
  • IndexOptimizerMainDirectUploadTest: 5 optimizer wiring tests mirroring the IS-server tests

Hammer suites

All three hammer suites passed green:

  • DirectMultipleConcurrentUpdatesSuiteNoIndexesTest, DirectMultipleConcurrentUpdatesSuiteWithNonUniqueIndexesTest, DirectMultipleConcurrentUpdatesSuiteWithUniqueIndexesTest (12/12)
  • BLinkConcurrentSearchInsertTest (1/1)
  • LazyValueCacheConcurrentUpdatesSuiteNoIndexesTest, LazyValueCacheConcurrentUpdatesSuiteWithNonUniqueIndexesTest, LazyValueCacheConcurrentUpdatesSuiteWithUniqueIndexesTest (12/12)

Rollout guidance

Both indexing.s3.direct.write.enabled and indexoptimizer.s3.direct.write.enabled default to false. To opt in, set the flag(s) to true in values.yaml after verifying the bulk upload path in your environment. Existing direct-read deployments (from issue #381) are unaffected.

🤖 Implemented by the pr-worker agent.

eolivelli added 3 commits May 21, 2026 19:49
…paction, bypassing file server

Introduce a direct S3 upload path in RemoteFileDataStorageManager that
bypasses the gRPC file-server during checkpoint/compaction writes when a
direct ObjectStorage is available. Files are stored as a single S3 object
at the logical path suffixed with ".bulk", using S3TransferManager (real
S3 Multipart Upload with CRT-pipelined parts) for maximum throughput.

Legacy per-block files remain readable via an isBulkLayout() probe-and-
fallback mechanism with a ConcurrentHashMap probe cache to avoid per-read
HEAD round-trips.

An independent backpressure semaphore (separate from the gRPC client's
inflight-write budget) bounds concurrent direct uploads. Both IndexingServer
and IndexOptimizerMain wire the enableDirectUpload() call after
setDirectObjectStorage(), controlled by new config keys:
  indexing.s3.direct.write.enabled (default: true)
  indexing.remote.file.client.max.inflight.direct.write.bytes (default: 512 MiB)
  indexoptimizer.s3.direct.write.enabled (default: true)
  indexoptimizer.remote.file.client.max.inflight.direct.write.bytes (default: 512 MiB)

New tests: direct-upload unit tests, backpressure concurrency tests,
legacy-fallback tests, IS/optimizer wiring tests, and a MinIO
Testcontainers integration test for end-to-end S3 multipart upload.
- bulkLayoutCache: cache only positive results to prevent write/probe race
  from permanently pinning a logical path as "not bulk" for the JVM
  lifetime. A false probe result now returns without touching the cache so
  a concurrent writer's put(TRUE) is never overwritten by a stale probe.
- enableDirectUpload: add idempotency guard so repeated calls with the same
  cap are no-ops (prevents discarding an in-use semaphore).
- reserveDirectInflightUploadBytes: replace unbounded acquireUninterruptibly
  with tryAcquire(10 min, MINUTES) to surface stuck CRT futures as IOException
  instead of blocking forever.
- Reduce HEAD probe and bulk delete timeouts from 60 s to 15 s to avoid
  stalling vector-search reads for a full minute on slow S3 backends.
- Flip PROPERTY_S3_DIRECT_WRITE_ENABLED_DEFAULT from true to false in both
  IndexingServerConfiguration and OptimizerConfiguration. Operators opt in
  explicitly so no IS pod silently changes its on-disk segment format on
  upgrade.
- Backpressure test: replace Thread.sleep(200L) "confirm B is blocked" with
  active wait on heldCount()==1 (proves B has NOT called uploadFile yet, i.e.
  is genuinely blocked at the semaphore). Raise 1 s busy-wait ceilings to
  10 s for slow CI runners.
- Tests: rename perBlockOnlyFileIsDiscoverable to
  perBlockOnlyFileIsNotDiscoverableWithoutGrpc. Replace tautological
  assertTrue(true) in deleteLegacyOnlyFileSucceeds with real post-conditions.
  Tighten S3DirectBulkUploadIT list assertion to assertEquals(1, size()).
  Add enableDirectUploadIsIdempotentForSameCap and
  cacheRaceWriteThenProbeDoesNotPinNegativeResult tests.
- Update config tests for default=false in both IS and optimizer wiring tests.
- ensureBulkLocalCacheFile: download to a unique UUID-named temp file per
  call instead of a deterministic path, preventing concurrent callers from
  interleaving bytes into the same destination. putIfAbsent elects the
  winner; losers delete their temp files.
- Add proper imports (FilterInputStream, AtomicBoolean, TimeUnit, Files,
  UUID, InputStream) and drop all fully-qualified references introduced by
  this patch from RemoteFileDataStorageManager.java.
- Fix stale Javadoc on writeMultipartIndexFileDirect: "chunked acquire to
  allow smaller concurrent uploads to interleave" was inaccurate — the
  semaphore reserves the full clamped count for the duration of the upload.
- probeCacheIsPopulatedAfterFirstProbe: replace weak byte-equality
  assertions with a direct existsObject counter (via CountingLocalObjectStorage
  inner class) that proves exactly one HEAD probe fires on first positive hit
  and zero additional probes on subsequent cache-hit calls.
- Fix stale "caches negative result for JVM lifetime" docstring to reflect
  the positive-only caching policy introduced in the previous review.
- S3DirectBulkUploadIT: drop tautological assertNotNull(long) and strengthen
  plain-storage assertion to assertTrue(existsObject(...)).
- All hammer suites (DirectMultipleConcurrentUpdatesSuite*,
  BLinkConcurrentSearchInsertTest, LazyValueCacheConcurrentUpdatesSuite*)
  pass green.
@eolivelli
eolivelli merged commit bd18c0d into master May 21, 2026
8 checks passed
@eolivelli
eolivelli deleted the issue-638-is-direct-write-to-s3minio-during-compac branch May 21, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[IS] Direct-write to S3/MinIO during compaction — bypass file server when direct access is enabled

1 participant