What happened
A queued request evicted because its queue-wait TTL expired (flowControl.defaultRequestTTL) gets HTTP 503: translateFlowControlOutcome maps QueueOutcomeEvictedTTL to ServiceUnavailable (pkg/epp/requestcontrol/admission.go), with x-llm-d-request-dropped-reason: rejected-ttl-expired.
Why this looks wrong
#1751 defined the split:
- 429 (ResourceExhausted) — capacity exists but is contended (backpressure).
- 503 (ServiceUnavailable) — no serving capacity exists right now.
A TTL eviction in a non-empty pool is the first case: endpoints are serving, the request was shed after waiting out contention. The service is exhausted, not unavailable. It is the same overload class as QueueOutcomeRejectedCapacity (429) — the shed just happens after waiting instead of at enqueue. Today the same saturated pool returns 429 or 503 depending only on whether the request got a queue slot, and the 503 misleads clients/load balancers into treating a live-but-backlogged backend as down.
On retry semantics: #1022 noted a TTL-expired request shouldn't be blindly retried — but that is what the rejected-ttl-expired drop-reason header conveys. The status code should still truthfully signal overload of a live service; 429 + header preserves both.
Proposed change
|
Capacity rejection |
TTL eviction |
| Empty pool |
503 |
503 |
| Non-empty saturated |
429 |
429 (was 503) |
EvictedContextCancelled and shutdown-drained (#1612) stay 503, unchanged.
Happy to send a PR (mapping change + unit tests for the translation and the empty-pool discriminator).
What happened
A queued request evicted because its queue-wait TTL expired (
flowControl.defaultRequestTTL) gets HTTP 503:translateFlowControlOutcomemapsQueueOutcomeEvictedTTLtoServiceUnavailable(pkg/epp/requestcontrol/admission.go), withx-llm-d-request-dropped-reason: rejected-ttl-expired.Why this looks wrong
#1751 defined the split:
A TTL eviction in a non-empty pool is the first case: endpoints are serving, the request was shed after waiting out contention. The service is exhausted, not unavailable. It is the same overload class as
QueueOutcomeRejectedCapacity(429) — the shed just happens after waiting instead of at enqueue. Today the same saturated pool returns 429 or 503 depending only on whether the request got a queue slot, and the 503 misleads clients/load balancers into treating a live-but-backlogged backend as down.On retry semantics: #1022 noted a TTL-expired request shouldn't be blindly retried — but that is what the
rejected-ttl-expireddrop-reason header conveys. The status code should still truthfully signal overload of a live service; 429 + header preserves both.Proposed change
QueueOutcomeEvictedTTL→ResourceExhausted(429), keeping therejected-ttl-expiredheader.EvictedContextCancelledand shutdown-drained (#1612) stay 503, unchanged.Happy to send a PR (mapping change + unit tests for the translation and the empty-pool discriminator).