Chat Completions reasoning is injected into eval prompt history #264

Summary

ChatCompletionsSampler mutates the list reported as actual_queried_message_list after the API request by appending choice.message.reasoning as an assistant message.

That reasoning text was generated by the model and was not part of the messages sent to the API. More importantly, HealthBench passes actual_queried_message_list back into its grader as the conversation prompt before appending the final assistant response, so Chat Completions reasoning can become an extra assistant turn visible to the grader and affect scoring.

Current path

The sampler currently does:

choice = response.choices[0]
content = choice.message.content
if getattr(choice.message, "reasoning", None):
    message_list.append(self._pack_message("assistant", choice.message.reasoning))
...
return SamplerResponse(
    response_text=content,
    response_metadata={"usage": response.usage},
    actual_queried_message_list=message_list,
)

HealthBench later uses:

actual_queried_prompt_messages = sampler_response.actual_queried_message_list
...
self.grade_sample(
    prompt=actual_queried_prompt_messages,
    response_text=response_text,
    ...
)

and grade_sample() forms the grading conversation as that prompt plus the final assistant response.

Impact

For Chat Completions backends that expose a reasoning field, the grader can receive model reasoning that was not part of the original prompt and is distinct from the response being evaluated. This can change HealthBench scores and makes actual_queried_message_list inaccurate for reports and saved conversations.

Proposed resolution

Keep actual_queried_message_list equal to the messages actually sent to the model. If retaining reasoning is useful for diagnostics, store it in response metadata rather than inserting it into the prompt history.

Add a regression with a response containing both reasoning and final content that verifies:

• the queried message list is unchanged;
• the final response remains response_text;
• reasoning is retained separately in metadata.