[FeatureAsk] Instantiating a model config for Azure OAI using AAD instead of `api_key`

[brynn-code]

Discussed in #3552

^{Originally posted by labeebee July 17, 2024}
Hello,

I am trying to use Prompt Flow and my organization has enforced AAD for accessing Azure OAI resources. But I see no way to define a model config using AzureOpenAIModelConfiguration(). Can anyone help?

[brynn-code]

@labeebee could you please let us know your usage? We'd like to see if there is a way to use AzureOpenAIConnection instead of ModelConfig to workaround this problem.

[labeebee]

Well, I 'm just starting out with the examples and was trying out the evaluation section in this notebook. That's when I hit the blocker. However, I did have a hunch this could be solved using connections. But could not properly get it working.

[brynn-code]

Just take a look at the example, the evaluator in this notebook is using Prompty to achieve llm calls which is not support AAD for now, we have this example https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/eval-criteria-with-langchain/langchain-eval.ipynb running evaluation with connection, to use the AAD AOAI connection, you could write your own evaluator, something like:

from promptflow.tools.common import init_azure_openai_client
from openai import AzureOpenAI

class MyEvaluator:
    def __init__(self, connection: AzureOpenAIConnection):
        self.connection = connection
        self._client = init_azure_openai_client(self.connection)

    @trace
    def __call__(
        self,
        input: str,
        prediction: str,
    ) -> Result:
        # ... your own logic ...

[labeebee]

Will definitely try this out. But I think it is fair to have this as a feature. Thanks ☺️

[brynn-code]

Of course, AAD is an important part of model config, as connection has supported it, it could be a long-term work so you could use connection to unblock yourself at first.

[labeebee]

Hey @brynn-code I took some time to try this out. However, it fails during batch tests. I have working custom connection to AOI, configured via AAD, yet this is what I am seeing at the end, when I run it in batches.

[promptflow._sdk._orchestrator.run_submitter][WARNING] - Run promptflow_trials_20240723_013918_173917 failed when executing in executor with exception Failed to initialize flow entry with '{'connection': 'aoi_connection', 'flow_config': '{}'}', ex:'Connection 'aoi_connection' is not found...

This is my line of code that threw the error:

pf.run(flow="eval_demo:PreviewEvaluator", init={"connection": "aoi_connection", "flow_config": "{}"}, data="./data.jsonl")

[brynn-code]

Hi there, the error says Connection not found, does the 'aoi_connection' exists?

[labeebee]

Yea. It does. I just defined it in the cell above to the one I ran this piece in. Kind regards, Labeeb

…

On Tue, Jul 23, 2024 at 2:21 AM Brynn Yin ***@***.***> wrote: Hi there, the error says Connection not found, does the 'aoi_connection' exists? — Reply to this email directly, view it on GitHub <#3555 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDV2VMQOKRDXK3LLA6OW23ZNXY7TAVCNFSM6AAAAABK7X6KXCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBUGM2DGOBXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[brynn-code]

We will fetch the connection from local sqlite db in this step, and created connection can be used in different flow so that you don't need to define it every time, if you have defined the connection object below, could you please try create it at first?

connection = AzureOpenAIConnection(name="aoi_connection", ...)
pfclient.connections.create_or_update(connection)

More about manage connections: https://microsoft.github.io/promptflow/how-to-guides/manage-connections.html

[labeebee]

@brynn-code Oh yea, I was using the name for the connection wrong. But, is there any dependency over any of the LangChain modules? Despite having successfully integrated the connection that does not require OpenAI API key, Prompt Flow threw an Exception as this,

ValidationError: 2 validation errors for ChatOpenAI model none is not an allowed value (type=type_error.none.not_allowed) __root__ Did not find openai_api_key, please add an environment variable OPENAI_API_KEYwhich contains it, or passopenai_api_key as a named parameter. (type=value_error)

I am not even using the ChatOpenAI class, which I believe is from LangChain.