`ContentTooLargeException` when using ` RetryingClient` or `RedirectingClient`

[yzfeng2020]

Description

When a streaming request body larger than ~2 GiB is sent through RetryingClient or RedirectingClient, the request fails with ContentTooLargeException even though the application set no content-length limit.

Both clients need to replay the request body across retry attempts / redirect hops, so for streaming requests they wrap it in a duplicator:

// RetryingClient.doExecute(...) and RedirectingClient.execute(...)
if (ctx.exchangeType().isRequestStreaming()) {
    final HttpRequestDuplicator reqDuplicator =
            req.toDuplicator(ctx.eventLoop().withoutContext(), 0);
    ...
}

The 0 argument means "no limit", but DefaultStreamMessageDuplicator maps 0 (and any value> Integer.MAX_VALUE) to Integer.MAX_VALUE and then buffers the entire body in memory, accumulating the total into an int signalLength field:

// DefaultStreamMessageDuplicator.StreamMessageProcessor
private final int maxSignalLength;
private int signalLength;

// constructor
if (maxSignalLength == 0 || maxSignalLength > Integer.MAX_VALUE) {
    this.maxSignalLength = Integer.MAX_VALUE;
} else {
    this.maxSignalLength = (int) maxSignalLength;
}

// on each data chunk
final int dataLength = signalLengthGetter.length(obj);
if (dataLength > 0) {
    final int allowedMaxSignalLength = maxSignalLength - signalLength;
    if (dataLength > allowedMaxSignalLength) {
        final long transferred = LongMath.saturatedAdd(signalLength, dataLength);
        final ContentTooLargeException cause =
                ContentTooLargeException.builder()
                                        .maxContentLength(maxSignalLength)
                                        .transferred(transferred)
                                        ...
        // aborts the request
    }
    signalLength += dataLength;
}

Once the accumulated body exceeds Integer.MAX_VALUE (~2 GiB), the duplicator throws ContentTooLargeException, aborting the request.

There are really two coupled problems:

1. The int32 cap — the running total is an int, so replay-buffering is capped at ~2 GiB even when the caller explicitly asked for no limit (0).
2. Buffering at all — even below 2 GiB, the whole upload is held in heap solely to enable replay, which is undesirable for large uploads.

Expected behavior

A large streaming upload should be retriable / redirectable without being capped at 2 GiB and without buffering the whole body in memory.

Proposed direction

Rather than buffering the body for replay, allow the caller to declare that the request body is reproducible by supplying a factory that regenerates it on demand. When such a factory is present, RetryingClient / RedirectingClient would:

• stream the caller's original request straight through on the first attempt (no buffering), and
• obtain a fresh request from the factory for each subsequent attempt (retry or redirect hop).

[minwoox]

Hi! It sounds like a good idea. Will you prepare the PR?

[yzfeng2020]

yes will do!