azure-ai-evaluation: AI quality metrics starts to return error for missing api_key when passing entra ID based model configs since 1.5.0

[kennethsongms]

• Package Name:
• azure-ai-evaluation
• Package Version:
• 1.5.0
• Operating System:
• Linux
• Python Version:
  3.13.3

Describe the bug
After we upgraded our evaluation code from 1.3.0 to 1.5.0, we started to see this error below when using some of the ai quality metrics e.g. relevancy evaluator. This error only started to occur since the new version in 1.5.0 (could be in 1.4.0 too, but we have not tested), the same code works when azure-ai-evaluation 1.3.0 is installed.

Error example:
"
ERROR 1/1 flow run failed, indexes: [0], exception of index 0: (UserError) The following required fields are missing for connection azure_openai: api_key
"

Our model config:

model_config = { "azure_endpoint": os.getenv("AZURE_OPENAI_ENDPOINT"), "azure_deployment": os.getenv("AZURE_OPENAI_EVALUATION_DEPLOYMENT"), "api_version": os.getenv("AZURE_OPENAI_API_VERSION"), }

We have credfree requirement where we are not allowed to use API key-based connections, and our models authentication is based on entraID.

To Reproduce
Steps to reproduce the behavior:

1. Configure the authentication of the model to be based on entraID based instead of API key/access key based.
2. Install azure-ai-evaluation 1.5.0, use any of the out of box ai quality evaluator from the SDK, for example relevance evaluator.
3. Run evaluation
4. The error in description would occur

Expected behavior
EntraID based connection should be supported, and api_key would not be required when configuring a model.

Additional context
Add any other context about the problem here.

[kashifkhan]

Thank you for the feedback @kennethsongms . I'm routing this over to the team who can help you further

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.

[jjshao-ms]

Same here, 1.4.0 works, but 1.5.0 failed.
Repro snippet

from azure.ai.evaluation import SimilarityEvaluator
SimilarityEvaluator({ "azure_endpoint": "<endpoint>", "azure_deployment": "gpt-4o" })(query="1+1=?", response="10", ground_truth="2")

[YaelMagrafta]

I also encountered this issue. And I agree the evaluators should support access options like azure_ad_token_provider and default_headers.
Running with
azure-ai-evaluation==1.5.0
running on python 3.12
Is there any expectation for a fix in the matter?
Thanks!

[singankit]

@kennethsongms Thanks for your patience. This issue is fixed in latest release 1.6.0