Skip to content

Report Promptfoo runtime failures as provider errors - #3010

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix/promptfoo-provider-errors
Open

Report Promptfoo runtime failures as provider errors#3010
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix/promptfoo-provider-errors

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • Return unexpected runtime/API failures through Promptfoo's provider error field instead of presenting them as assistant output.
  • Keep successful responses and explicit guardrail-tripwire blocks unchanged.

Motivation

The Promptfoo target currently catches any unexpected exception and returns [ERROR] ... in the normal output field. That makes infrastructure or provider failures look like model behavior to the red-team evaluator, so they can be scored as if the assistant actually produced that text.

Promptfoo's Python provider result contract has a dedicated error field for provider failures. Using it keeps evaluation failures distinct from governed assistant responses.

Validation

  • successful run -> existing output behavior unchanged
  • input/output guardrail tripwire -> existing [BLOCKED] output unchanged
  • unexpected runtime/API exception -> empty output plus populated error

Self-review

  • One-line correctness fix in one file.
  • No model, policy, dependency, notebook, registry, or configuration changes.
  • Searched open PRs for an existing Promptfoo provider-error fix and found none.

Maintainers may modify the branch if needed.

@sylvesterkaczmarek
sylvesterkaczmarek requested a review from a team as a code owner August 21, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant