Summary
Both ResponsesSampler and ChatCompletionsSampler catch every exception other than BadRequestError in an unbounded exponential-backoff loop.
That includes permanent OpenAI API status failures such as authentication (401), permission (403), not found (404), and unprocessable request (422). Those conditions do not become valid by waiting, so an eval can sleep and retry forever instead of surfacing the configuration/API error.
Current behavior
Both samplers end with the equivalent of:
except Exception as e:
exception_backoff = 2**trial
time.sleep(exception_backoff)
trial += 1The OpenAI SDK exposes these permanent failures through APIStatusError subclasses with their HTTP status codes.
Impact
A bad API key, wrong endpoint/model, or denied permission can turn a deterministic startup failure into an indefinitely hanging eval with exponentially growing pauses.
Proposed resolution
Keep retry/backoff behavior for retryable API statuses (408, 409, 429 and 5xx) and connection errors, but immediately re-raise non-retryable APIStatusError values and unexpected exception types.
Preserve the existing explicit BadRequestError behavior.
Add regressions for both sampler implementations proving a permanent status failure is surfaced without invoking time.sleep.
Summary
Both
ResponsesSamplerandChatCompletionsSamplercatch every exception other thanBadRequestErrorin an unbounded exponential-backoff loop.That includes permanent OpenAI API status failures such as authentication (401), permission (403), not found (404), and unprocessable request (422). Those conditions do not become valid by waiting, so an eval can sleep and retry forever instead of surfacing the configuration/API error.
Current behavior
Both samplers end with the equivalent of:
The OpenAI SDK exposes these permanent failures through
APIStatusErrorsubclasses with their HTTP status codes.Impact
A bad API key, wrong endpoint/model, or denied permission can turn a deterministic startup failure into an indefinitely hanging eval with exponentially growing pauses.
Proposed resolution
Keep retry/backoff behavior for retryable API statuses (408, 409, 429 and 5xx) and connection errors, but immediately re-raise non-retryable
APIStatusErrorvalues and unexpected exception types.Preserve the existing explicit
BadRequestErrorbehavior.Add regressions for both sampler implementations proving a permanent status failure is surfaced without invoking
time.sleep.