Skip to content

Fail fast on permanent eval sampler API errors - #263

Open
sylvesterkaczmarek wants to merge 3 commits into
openai:mainfrom
sylvesterkaczmarek:fix/sampler-permanent-api-errors
Open

Fail fast on permanent eval sampler API errors#263
sylvesterkaczmarek wants to merge 3 commits into
openai:mainfrom
sylvesterkaczmarek:fix/sampler-permanent-api-errors

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Stop the eval samplers from retrying permanent API failures forever.

Both ResponsesSampler and ChatCompletionsSampler currently put every exception other than BadRequestError into an unbounded exponential-backoff loop. That includes non-retryable API status failures such as authentication, permission, not-found, and unprocessable-request errors.

Fixes #262.

Fix

Both samplers now:

  • preserve the existing explicit BadRequestError behavior;
  • retry API status codes 408, 409, 429, and 5xx with the existing exponential backoff;
  • retry APIConnectionError with the existing exponential backoff;
  • immediately re-raise other APIStatusError values;
  • allow unexpected non-API exceptions to propagate instead of treating them as rate limits.

Chat Completions still retries its intentional empty-success-response condition through a dedicated internal exception, preserving that existing behavior without making arbitrary ValueError instances retryable.

Regression coverage

Adds tests for both sampler implementations verifying that representative permanent status failures are surfaced immediately and do not invoke time.sleep.

The retry delay and retryable-status behavior are otherwise unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval samplers retry permanent API errors forever

1 participant