Non-Reproducible Results Despite Fixed Parameters in Azure OpenAI API

[Aleksandra-Gr]

I am experiencing non-reproducible results when using the Azure OpenAI API, even when setting parameters explicitly to ensure deterministic behavior. My goal is to get identical outputs when running the same prompt under the same conditions. What I do is:

1. Set up the API connection by default, which is 2024-02-01. Fix the azure_openai_endpoint, and pf_deployment is gpt-4o-mini.

2. Run the prompt with the following fixed parameters:

{
"temperature": 0,
"top_p": str(top_p), # Changing from 0 to 1 in a loop to check if problem existis for all those values
"seed": 42,
"max_tokens": 100,
"frequency_penalty": 0,
"presence_penalty": 0,
"stream": false
}

3. Repeat the request 10 times per top_p value.

4. Observe the output variance.

Expected Behavior:

With temperature = 0 and a fixed seed, I expect deterministic outputs—meaning the same response should be returned each time for identical inputs.

Observed Behavior:

Despite fixing temperature = 0, seed = 42, and all other parameters, the responses vary.
Sometimes, all responses are different (in summaries outputs) for the same prompt.

This issue persists across multiple API versions, including this default 2024-02-01 and 2025-02-01-preview.

[github-actions]

Hello @Aleksandra-Gr. I'm an AI assistant for the azure-sdk-for-python repository. I have some suggestions that you can try out while the team gets back to you.

• Verify that the parameters you are using (especially seed) are actually supported by the underlying model and API version. The Azure OpenAI API may not honor the seed for determinism, as observed with some deployments.

• Double-check that the fixed parameters are being sent as intended by logging the complete request payload. Differences in preprocessing or model state (especially with models like gpt-4o-mini) can lead to variability.

• Experiment with different API versions to see if determinism improves, and review the latest Azure AI Inference documentation for any noted limitations regarding deterministic responses.

• Consider reaching out on GitHub with a minimal reproducible example to verify if this behavior is expected or if it's an unresolved issue with the model deployment.

The team will get back to you shortly, hopefully this helps in the meantime.

[kristapratico]

Hi @Aleksandra-Gr, this repo is for issues with the SDK, and given this question is related to behavior of the Azure OpenAI service, I suggest you check out the support options for where you will find experts better suited to answer your question.

With temperature = 0 and a fixed seed, I expect deterministic outputs—meaning the same response should be returned each time for identical inputs.

My understanding is that determinism is not guaranteed (source: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/reproducible-output?tabs=pyton). I recommend checking out the linked article as it does provide some suggestions on getting more consistent results.

[github-actions]

Hi @Aleksandra-Gr. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @Aleksandra-Gr, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!