Skip to content

issue #636: fix flaky S3ObjectStorageDownloadToFileTest publisher (idempotent request(n)) - #637

Merged
eolivelli merged 1 commit into
masterfrom
issue-636-flaky-test-s3objectstoragedownloadtofile
May 21, 2026
Merged

eolivelli merged 1 commit into
masterfrom
issue-636-flaky-test-s3objectstoragedownloadtofile

Conversation

@eolivelli

Copy link
Copy Markdown
Owner

Fixes #636.

Summary

The flake (expected:<10> but was:<15> in downloadToFileAppendsSubsequentBlocks) was caused by a Reactive Streams contract violation in the test helper, not in production code. The inline Subscription inside proxyClient re-delivered the payload on every request(n) invocation; the SDK's AsyncResponseTransformer.toFile() is free to issue more than one request(n) per subscription (more often in the append-mode defaultCreateOrAppend() path), and any second request silently doubled the bytes written.

The fix:

  • Extract the test publisher into a static singleBufferPublisher(byte[]) helper that delivers onNext + onComplete at most once via an AtomicBoolean.compareAndSet.
  • Treat cancel() as a terminal state — flip the same flag so any post-cancel request(n) is a no-op.
  • proxyClient now uses transformer.onStream(singleBufferPublisher(payload)); failingProxyClient is unchanged (it never creates a Subscription — it calls transformer.exceptionOccurred(error) directly).

Production code (S3ObjectStorage.downloadToFile) is unchanged.

Regression tests

Two new deterministic tests pin the Reactive Streams contract:

  • requestIsIdempotentUnderMultipleRequestCalls — hand-rolled Subscriber calls request(1) / request(1) / request(Long.MAX_VALUE) synchronously inside onSubscribe; asserts exactly one onNext, one onComplete, zero onError, and that the delivered bytes match the payload.
  • requestAfterCancelIsNoOp — onSubscribe calls cancel() then request(1) / request(Long.MAX_VALUE); asserts neither onNext nor onComplete ever fire.

The original four tests in this file continue to pass.

Verification

  • 50 successive full-class runs of S3ObjectStorageDownloadToFileTest on this branch: 50 / 50 green (pre-fix, the bare downloadToFileAppendsSubsequentBlocks test failed once over a handful of repeats).
  • Full module suite (mvn -pl herddb-remote-file-service test): 495 / 495 green (was 493 / 493; +2 new tests).
  • Pre-PR validation (mvn -B spotless:check apache-rat:check spotbugs:check install -DskipTests -Pci): green.

Test plan

  • 50× repeat of the affected test class — no failures.
  • Full herddb-remote-file-service module suite green.
  • Pre-PR validation green.
  • CI green.

🤖 Implemented by the pr-worker agent.

The proxyClient helper's inline Subscription delivered the payload on
every request(n) invocation. The Reactive Streams contract allows the
subscriber to call request(n) any number of times; AsyncResponseTransformer
.toFile() occasionally issued more than one request per subscription,
which silently doubled the bytes written to disk and caused the rare
"expected:<10> but was:<15>" flake in downloadToFileAppendsSubsequentBlocks.

Extract the publisher into a static singleBufferPublisher helper that
delivers onNext + onComplete at most once via an AtomicBoolean CAS, and
treats cancel() as a terminal state. Add two deterministic regression
tests that drive a hand-rolled Subscriber with multiple back-to-back
request(n) calls and assert exactly one onNext.

Verified by 50 successive full runs of the test class with no failures.
@eolivelli
eolivelli merged commit a992728 into master May 21, 2026
8 checks passed
@eolivelli
eolivelli deleted the issue-636-flaky-test-s3objectstoragedownloadtofile branch May 21, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: S3ObjectStorageDownloadToFileTest.downloadToFileAppendsSubsequentBlocks (length 10 vs 15)

1 participant