Skip to content

Keep Chat Completions reasoning out of eval prompt history - #265

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/chat-sampler-reasoning-history
Open

Keep Chat Completions reasoning out of eval prompt history#265
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/chat-sampler-reasoning-history

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Keep model-generated Chat Completions reasoning separate from the message list reported as the prompt actually queried.

ChatCompletionsSampler currently appends choice.message.reasoning to message_list after the API returns, then exposes that mutated list as actual_queried_message_list. HealthBench uses that field as the conversation passed to its grader before adding the final response, so reasoning can become an extra assistant turn that affects grading even though it was not part of the original prompt.

Fixes #264.

Fix

The sampler now:

  • leaves actual_queried_message_list equal to the messages sent to the API;
  • keeps final answer content in response_text as before;
  • retains a non-empty reasoning field separately in response_metadata for diagnostics.

Regression coverage

Adds tests verifying that:

  • a response containing reasoning does not mutate the queried prompt history;
  • final content remains the sampled response;
  • reasoning is retained in metadata;
  • responses without reasoning preserve the existing metadata shape apart from usage.

The request payload, scoring logic, retry behavior, and final response extraction are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat Completions reasoning is injected into eval prompt history

1 participant