StandardRetryPolicy: Full Jitter backoff can collapse to near-zero delays for throttling errors

[markhasper]

Describe the bug

StandardRetryPolicy.CalculateRetryDelay uses Full Jitter (random() * base_delay) for all error types, including throttling. When the random jitter value is low, retry delays can be near-zero, causing all retries to exhaust within seconds — too fast for rate limits that operate on per-minute windows.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Throttling retries should have a meaningful minimum delay. AWS's own "Exponential Backoff and Jitter" architecture blog recommends "Equal Jitter" for this scenario: base/2 + random(0, base/2), which guarantees at least 50% of the base delay.

Current Behavior

In StandardRetryPolicy.CalculateRetryDelay (StandardRetryPolicy.cs, line 243):

protected static int CalculateRetryDelay(int retries, int maxBackoffInMilliseconds)
{
    double jitter;
    lock (_randomJitter) {        
        jitter = _randomJitter.NextDouble(); // [0.0, 1.0)
    }
    return Convert.ToInt32(Math.Min(jitter * Math.Pow(2, retries - 1) * 1000.0, maxBackoffInMilliseconds));
}

When jitter = 0.05, 8 retries produce delays of approximately: 50ms, 100ms, 200ms, 400ms, 800ms, 1000ms, 1000ms, 1000ms — totalling ~4.5 seconds of backoff. The per-minute rate limit window needs 12+ seconds between requests to clear, so all retries fail.

This is fundamentally different from transient errors (network blips) where fast retries are fine. For throttling errors, the server has explicitly told the client to slow down.

Reproduction Steps

1. Use AmazonBedrockRuntimeClient with MaxErrorRetry = 8 and ThrottleRetries = true
2. Call ConverseStreamAsync against a model with a 5 requests/minute rate limit (e.g., eu.anthropic.claude-opus-4-6-v1 on Bedrock)
3. When the rate limit is hit, the SDK retries 8 times
4. With low jitter values (~0.05), all 8 retries complete in ~5 seconds total, never waiting long enough for the rate limit window to clear

Possible Solution

Use Equal Jitter for throttling errors, keep Full Jitter for transient errors:

protected static int CalculateRetryDelay(int retries, int maxBackoffInMilliseconds, 
    bool isThrottlingError = false)
{
    double jitter;
    lock (_randomJitter) {        
        jitter = _randomJitter.NextDouble();
    }
    
    double baseDelay = Math.Pow(2, retries - 1) * 1000.0;
    double delay = isThrottlingError
        ? baseDelay / 2 + jitter * baseDelay / 2  // Equal Jitter: guaranteed >= 50% of base
        : jitter * baseDelay;                       // Full Jitter: existing behavior
    
    return Convert.ToInt32(Math.Min(delay, maxBackoffInMilliseconds));
}

The isThrottlingError flag is already available in the retry pipeline — RetryForExceptionSync calls IsThrottlingError() and the result could be passed through to WaitBeforeRetry.

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

• Package: AWSSDK.BedrockRuntime 4.0.15.1 (also verified on latest main — CalculateRetryDelay is unchanged as of 4.0.16.1 / AWSSDK.Core 4.0.3.15)
• Runtime: .NET 10
• OS: Linux (Azure Container Apps)

Targeted .NET Platform

.NET 10

Operating System and version

Windows

[jnunn-aws]

Thanks for the detailed report. We're still working through the best approach and will follow up when we have more to share.

[normj]

You're bumping up against an interesting tuning philosophy question. The use-case example you gave is the perfect for adaptive retry mode: single-tenant caller that is throttle dominated. Try it out for this use-case and see if it behaves better for you. Unfortunately, we've recently identified a bug in adaptive retry mode in a few of our SDKs that results in it over-throttling callers, so it still may not behave perfectly. We're working to get a fix out as soon as possible.

But, you're totally right: equal jitter is better than full jitter for use-cases that have few calling threads (1-2) and are heavily limited by throttling responses. In these cases, the single caller is only ever competing against themselves for resources, and equal jitter limits their request rate in the very useful way you describe.

The SDK's default retry behavior are tuned for a different use-case: a large number of callers (hundreds-thousands) that are, by in large, fairly independent. This is more common in service environments, which is where the SDKs typically end up. In these cases, full jitter is preferred because a throttle response is just as likely to be received at the end of a "capacity refill" event as at the beginning. In this case, equal jitter's downsides become more prominent: increased caller latency and higher request clustering when traffic patterns aren't uniform.

We've experimented with a more intelligent default retry strategy that can use the adaptive retry strategy's dynamic backoff durations for throttle-dominant use-cases, but we've been unable to crack an unrelated noisy-neighbor risk that adaptive retry mode has in a multi-tenant environment (see the link above for information on how to use adaptive safely in a multi-tenant environment).

Let me know if you'd be interested in us creating a blog post on AWS SDK retry tuning.