CreateFleet batcher amplifies rate limit failures and lacks internal throttle protection

[nathangeology]

Problem

The CreateFleet batcher (pkg/batcher/createfleet.go) collects up to 1,000 individual instance requests into a single EC2 CreateFleet API call. If that call receives RequestLimitExceeded, all N batched requests receive the same error at once. There is no rate limiting before calling EC2 and no batcher-level retry. The error goes straight to all callers.

One rate-limited API call becomes N simultaneous NodeClaim failures. All N retry together on the same schedule, re-triggering the rate limit on the next attempt.

Evidence

During scale testing with 500+ pending pods, the batcher created a single CreateFleet call with TotalTargetCapacity=500. The call was rate-limited. All 500 NodeClaims failed and retried together. This repeated across retry cycles, sustaining the throttle until claims were manually deleted.

Root Causes

1. One API call failure fans out to the entire batch

File: pkg/batcher/createfleet.go

Batcher configuration:

• IdleTimeout = 35ms
• MaxTimeout = 1s
• MaxItems = 1000

Requests arriving within the timeout window go into one CreateFleet call. If that call errors (including transient throttle), every request in the batch gets the same error. Blast radius equals batch size.

2. No rate limiting before EC2 API calls

The batcher calls CreateFleet as fast as batches fill. No token bucket, no adaptive limiter, no concurrency cap. It reacts to RequestLimitExceeded after the fact rather than staying within limits proactively.

3. No batcher-level retry for transient errors

When CreateFleet returns RequestLimitExceeded, the error goes to all waiting callers immediately. The batcher does not retry. Retry is delegated to the lifecycle controller, which retries each of the N failed NodeClaims independently, recreating the thundering herd at the controller level.

4. Throttle errors are not classified differently from permanent failures

File: pkg/providers/instance/instance.go:376-384

Two error paths exist for CreateFleet:

1. API-level error (the call itself fails, e.g., RequestLimitExceeded): The batcher returns err to all callers. The instance provider wraps it at line 384:

   return ec2types.CreateFleetInstance{}, cloudprovider.NewCreateError(
       fmt.Errorf("creating fleet request, %w", err), reason, message)

2. Fleet-level error (the call succeeds but returns errors in the response): combineFleetErrors (line 766) classifies UnfulfillableCapacity as InsufficientCapacityError, but other errors (including throttle) become generic CreateError.

In both paths, RequestLimitExceeded has no special classification. The upstream lifecycle controller cannot tell that this is transient.

Proposed Improvements

A. Add a token bucket rate limiter before CreateFleet calls

Where: pkg/batcher/createfleet.go

Cap the rate of CreateFleet API calls with a rate.Limiter or similar. Parameters could be fixed or adaptive (slow down when throttle errors appear, speed up when calls succeed).

B. Batcher-level retry with backoff and jitter

Where: pkg/batcher/createfleet.go

When CreateFleet returns RequestLimitExceeded, retry the call within the batcher (2-3 attempts, jittered backoff at 500ms/1s/2s base) before surfacing the error to callers. This keeps the blast radius contained and avoids triggering N independent retry paths upstream.

C. Reduce maximum batch size (tradeoff)

Cap effective batch size below 1,000 (e.g., 100-200) or split large batches into staggered sub-batches. Fewer callers affected per failure. Tradeoff: more API calls total, but smaller blast radius when one fails.

D. Classify RequestLimitExceeded as retryable

Where: pkg/providers/instance/instance.go:384 and pkg/providers/instance/instance.go:766-780

Wrap RequestLimitExceeded errors with the RetryableError interface proposed in kubernetes-sigs/karpenter#3034. This lets the lifecycle controller apply different retry policies and avoid marking the NodePool unhealthy on transient throttle.

Update combineFleetErrors to check for throttle error codes alongside the existing UnfulfillableCapacity check.

Code References

File	Relevance
pkg/batcher/createfleet.go	Batcher config, error fan-out to all callers
pkg/providers/instance/instance.go:376-384	API-level error wrapping as generic CreateError
pkg/providers/instance/instance.go:766-780	combineFleetErrors, only classifies UnfulfillableCapacity specially

Related

Companion to kubernetes-sigs/karpenter#3034 which addresses lifecycle-level handling of transient errors (stuck Unknown NodeClaims, jittered backoff, health condition separation). The RetryableError interface proposed there is what fix D here would implement.

---

Suggested labels:

• kind/bug
• priority/important-longterm
• area/provisioning

[nathangeology]

Follow-up: Scale Test Evidence (May 2026)

We ran two scale tests looking at batcher behavior under rate limiting.

Batcher amplification confirmed

Self-managed Karpenter test (2000 replicas, 43.7-second throttle window):

• 381 CreateFleet/RequestLimitExceeded errors across 78 unique NodeClaims
• Peak burst: 43 errors in a single 100ms window, consistent with one API call fanning out to ~43 batched callers
• Repeated bursts of 15-43 per 100ms, matching expected batch sizes given IdleTimeout (35ms) and arrival rate

The fan-out mechanism described in this issue is confirmed.

Observed scale differs from description

• Effective batch sizes: 40-70, not 500-1000
• Max per-claim retry count: 2, not hundreds
• Throttle window: 43 seconds, not sustained across many retry cycles
• Did not observe multi-cycle thundering herd (distinct retry waves re-triggering the limit)

Likely because our test was smaller (2000 replicas vs. implied 500+ simultaneous NodeClaims), and the SDK rate limiter (Waited before sending request -- 7 events at ~10s intervals) partially mitigated re-triggering.

Client-side rate limiter exists but is insufficient

7 instances of "Waited before sending request" in the CreateFleet path, spaced ~10 seconds apart. A token bucket exists in the SDK path, but it only slowed things down. At ~40-70 per batch, it could not prevent the initial burst from triggering EC2 limits.

Self-recovery observed

All 78 affected NodeClaims eventually launched. No manual intervention. Total disruption: ~44 seconds of CreateFleet failures, after which the rate limit cleared and subsequent batches succeeded.

Additional observation: SDK retry pool exhaustion

AssignPrivateIpAddresses calls exhausted the shared AWS SDK retry pool (493 "retry quota exceeded" errors). This is the IPAM controller, not CreateFleet, but it shows that heavy throttling in one API path cascades to others through the shared retry budget.

Assessment

Batcher amplification is real. In our test, it made a ~44 second throttle window affect 78 claims simultaneously rather than distributing failures over time. Recovery was automatic, but the amplification is meaningful -- especially at larger scale where batch sizes would be proportionally larger.

Reproduction details

• Test: 2000 replica Deployment, self-managed Karpenter v1.3
• Trigger: EC2 rate limiting (IP pool exhaustion -> cascading throttle)
• Batch sizes: 15-43 per burst (not 500-1000)
• Throttle duration: 43.7 seconds
• Recovery: Automatic

[nathangeology]

Closing this as low-priority based on further testing. The batcher amplification mechanism is confirmed (see follow-up comment above), but in practice:

• Self-recovers in ~44 seconds at our test scale (78 claims affected)
• No manual intervention needed
• The root trigger is hitting EC2 rate limits, which is addressable at the VPC/config level (larger subnets, /32 mode, pod subnet isolation)

The amplification is a real design property but the operational impact is a brief scaling delay during an already-degraded state. If this becomes more pressing at larger scale or someone wants to pick it up, the proposed fixes (token bucket before CreateFleet, smaller batch cap) are still valid.

[nathangeology]

Reopening, due a reported issue that might be related to this or something like this. (Will add details as I am able to.)