Construct non-Azure OpenAI clients for default and EU endpoints #43228

micah-press · 2024-12-04T22:01:45Z

Description

OpenAI is releasing an EU-based endpoint for their API. Users can set up a project that is regionally restricted so that API requests using a key for that project will fail if they don't hit the proper endpoint.

The current logic for choosing the client implementation to use with the OpenAIClient object checks for an unset endpoint value, or one that exactly matches "https://api.openai.com/v1". Using the EU endpoint, "https://eu.api.openai.com/v1", leads to runtime failures because an Azure-hosted OpenAI client implementation is used instead of a non-Azure implementation.

This PR adds the EU endpoint as a new public static field of the non-Azure OpenAI client class and checks for its presence when deciding which implementation to create in OpenAIClientBuilder.

Additional Testing

I wanted to make sure these changes did what I wanted them to do, so I created a POC project with both normal and EU-only API keys. I ran the following end to end tests:

ensure the currently available version of the Azure OpenAI package works with my normal API key and fails with my EU-only key (it fails with a 403 response from OpenAI if the endpoint is left as null and with a 404 if I set the endpoint to the EU-based URL)
ensure the modified version of the package works with my normal key
ensure the modified version of the package works with my EU-only key if I set the endpoint to the EU-based one

All SDK Contribution checklist:

The pull request does not introduce [breaking changes]
CHANGELOG is updated for new features, bug fixes or other significant changes.
I have read the contribution guidelines.

General Guidelines and Best Practices

Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

Pull request includes test coverage for the included changes.

With the release of OpenAI's EU endpoint, the client builder needs to create non-Azure OpenAI clients if the endpoint is null or if it matches the default API URL or the EU URL.

…ilder

github-actions · 2024-12-04T22:02:03Z

Thank you for your contribution @micah-press! We will review the pull request and get back to you soon.

micah-press · 2024-12-04T22:12:21Z

sdk/openai/azure-ai-openai/src/main/java/com/azure/ai/openai/OpenAIClientBuilder.java

-        return endpoint == null || endpoint.startsWith(OPEN_AI_ENDPOINT);
+        return endpoint == null || endpoint.startsWith(OPEN_AI_ENDPOINT) || endpoint.startsWith(EU_OPEN_AI_ENDPOINT);


This isn't the most extensible design. Might make more sense to check for whether the endpoint contains the OpenAI domain, but my security best practices are a bit shaky and I'm not sure how easily that could be exploited to direct requests somewhere they're not supposed to go.

micah-press · 2024-12-04T22:13:10Z

...ure-ai-openai/src/main/java/com/azure/ai/openai/implementation/NonAzureOpenAIClientImpl.java

+    public String getEndpoint() {
+        return this.endpoint;
+    }


Added this mostly to conform to the way the other implementation is written.

micah-press · 2024-12-04T22:14:33Z

sdk/openai/azure-ai-openai/src/test/java/com/azure/ai/openai/OpenAIClientBuilderTest.java

+        Field azureClient = OpenAIClient.class.getDeclaredField("serviceClient");
+        azureClient.setAccessible(true);
+        Field nonAzureClient = OpenAIClient.class.getDeclaredField("openAIServiceClient");
+        nonAzureClient.setAccessible(true);


This is pretty brittle, but I'm not sure of a better way to test that the correct implementation is used. Maybe it's better to leave it un-unit-tested since it's all under the hood. Open to suggestions!

azure-sdk · 2024-12-04T22:16:39Z

API change check

API changes are not detected in this pull request.

micah-press · 2024-12-05T22:08:18Z

@microsoft-github-policy-service agree company="Everlaw"

mssfang · 2024-12-05T23:05:53Z

@micah-press Thank you for the contribution!
Where did you get this EU-based URL? I haven't seen any news about this support in the public OpenAI specification.

micah-press · 2024-12-06T01:36:44Z

@mssfang I'm using the EU-based endpoint for work. It hasn't been publicly announced by OpenAI yet, but if you use any normal OpenAI API key and send a request to https://eu.api.openai.com/v1, you'll get a non-404 response that changes depending on the endpoint. I saw a 403 for the chat completion endpoint and 503 for embeddings.

If my approach is too strict considering there hasn't been a public announcement yet, maybe testing whether the openai.com domain is present in the configured endpoint string would be a better implementation?

… to use This commit generalizes the logic a bit more for deciding whether an Azure OpenAI client should be used or not. The regex allows for a single subdomain in the endpoint as long as the rest of it matches.

This PR contains the same exact changes as my PR targeting the upstream repo[1], but consolidated into a single commit for convenience. 1: Azure#43228

micah-press · 2025-02-06T18:47:04Z

EU data residency has been announced! Link here with API documentation linked towards the end of the page.

micah-press · 2025-02-26T21:45:46Z

Light bump here -- happy to incorporate review comments!

github-actions · 2025-05-02T05:05:12Z

Hi @micah-press. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

micah-press · 2025-05-02T16:45:18Z

I think this is still a change worth making, so I'd appreciate getting eyes back on it.

micah-press added 4 commits November 27, 2024 13:44

Change conditional logic for constructing non-Azure OpenAI clients

30ece0b

With the release of OpenAI's EU endpoint, the client builder needs to create non-Azure OpenAI clients if the endpoint is null or if it matches the default API URL or the EU URL.

Use getEndpoint() in API call implementations

4666e4b

Add unit tests for the endpoint configuration option of the client bu…

c85acd4

…ilder

Fix capitalization in javadoc

ea6f97e

micah-press requested review from brandom-msft, jpalvarezl and mssfang as code owners December 4, 2024 22:01

github-actions bot added Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. OpenAI labels Dec 4, 2024

micah-press commented Dec 4, 2024

View reviewed changes

mssfang self-assigned this Dec 5, 2024

micah-press added 2 commits December 10, 2024 15:21

Use a regex instead of hard-coded endpoints to determine which client…

e20a0c3

… to use This commit generalizes the logic a bit more for deciding whether an Azure OpenAI client should be used or not. The regex allows for a single subdomain in the endpoint as long as the rest of it matches.

Fix import order for build check

446114b

seannie2 mentioned this pull request Apr 22, 2025

Update OpenAIClientBuilder.java to be able to handle all non-azure endpoints #45074

Open

6 tasks

github-actions bot added the no-recent-activity There has been no recent activity on this issue. label May 2, 2025

github-actions bot removed the no-recent-activity There has been no recent activity on this issue. label May 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Construct non-Azure OpenAI clients for default and EU endpoints #43228

Construct non-Azure OpenAI clients for default and EU endpoints #43228

micah-press commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024

micah-press Dec 4, 2024

micah-press Dec 4, 2024

micah-press Dec 4, 2024

azure-sdk commented Dec 4, 2024

micah-press commented Dec 5, 2024

mssfang commented Dec 5, 2024

micah-press commented Dec 6, 2024

micah-press commented Feb 6, 2025

micah-press commented Feb 26, 2025

github-actions bot commented May 2, 2025

micah-press commented May 2, 2025

		return endpoint == null \|\| endpoint.startsWith(OPEN_AI_ENDPOINT);
		return endpoint == null \|\| endpoint.startsWith(OPEN_AI_ENDPOINT) \|\| endpoint.startsWith(EU_OPEN_AI_ENDPOINT);

Construct non-Azure OpenAI clients for default and EU endpoints #43228

Are you sure you want to change the base?

Construct non-Azure OpenAI clients for default and EU endpoints #43228

Conversation

micah-press commented Dec 4, 2024 • edited Loading

Description

Additional Testing

All SDK Contribution checklist:

General Guidelines and Best Practices

Testing Guidelines

github-actions bot commented Dec 4, 2024

micah-press Dec 4, 2024

Choose a reason for hiding this comment

micah-press Dec 4, 2024

Choose a reason for hiding this comment

micah-press Dec 4, 2024

Choose a reason for hiding this comment

azure-sdk commented Dec 4, 2024

micah-press commented Dec 5, 2024

mssfang commented Dec 5, 2024

micah-press commented Dec 6, 2024

micah-press commented Feb 6, 2025

micah-press commented Feb 26, 2025

github-actions bot commented May 2, 2025

micah-press commented May 2, 2025

micah-press commented Dec 4, 2024 •

edited

Loading