[FEATURE REQ] Support 'stored' & 'metadata' in Azure OpenAI client for ChatCompetionsOptions

[thai-op]

Is your feature request related to a problem? Please describe.

Support these 2 new parameters in the latest OpenAI chat completions API:

• store = [true|false]: https://platform.openai.com/docs/api-reference/chat/create#chat-create-store
• metadata = Map<String, String> : https://platform.openai.com/docs/api-reference/chat/create#chat-create-metadata

Additional context
These parameters are used for distillation. I don't think Azure supports it yet, but I'd like to at least use the client API against OpenAI endpoints. https://platform.openai.com/docs/guides/distillation

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Description Added
• Expected solution specified

[vivekkothari]

We would also like to use these params, any chance this can be picked up?

[alzimmermsft]

Thanks for reporting this @thai-op, @mssfang and @jpalvarezl could you provide a reply for this.

[mssfang]

@thai-op
azure-ai-openai supports both Azure OpenAI and OpenAI endpoints. We are waiting for Azure OpenAI to support above new parameters before we can release the SDK. metadata will be supported in 2024-10-01-preview Azure OpenAI service API version, but I haven't heard any news about store yet. I will keep you posted when the release it out.

[thai-op]

For those of you that want to continue to use Azure SDK but still also want to enjoy the latest OpenAI features, here's how I intercepted the HTTP requests and add store & metadata parameter into the chat completion request before sending it to OpenAI.

    static class OpenAIInterceptorHttpClient implements HttpClient {
        private final HttpClient delegate;
        private final ObjectMapper objectMapper = new ObjectMapper();

        OpenAIInterceptorHttpClient(HttpClient delegate) {
            this.delegate = delegate;
        }

        @Override
        public Mono<HttpResponse> send(HttpRequest request) {
            return delegate.send(request);
        }

        @Override
        public Mono<HttpResponse> send(HttpRequest request, Context context) {
            return delegate.send(request, context);
        }

        @SneakyThrows
        @Override
        public HttpResponse sendSync(HttpRequest request, Context context) {
            if (request.getUrl().getPath().endsWith("chat/completions")) {
                var modifiedRequest =
                        objectMapper.readValue(request.getBodyAsBinaryData().toString(),
                                InterceptedChatCompletionsOptions.class);
                var firstMessage = modifiedRequest.getMessages().getFirst();
                var metadata = new HashMap<String, String>();

                // if name is set on the first system or user message, then we'll automatically
                // store and tag those for distillation later
                switch (firstMessage) {
                    case ChatRequestSystemMessage system -> {
                        if (system.getName() != null) {
                            metadata.put("1stMessageRole", "system");
                            metadata.put("1stMessageName", system.getName());
                            modifiedRequest.setMetadata(metadata);
                            modifiedRequest.setStore(true);
                        }
                    }
                    case ChatRequestUserMessage user -> {
                        if (user.getName() != null) {
                            metadata.put("1stMessageRole", "user");
                            metadata.put("1stMessageName", user.getName());
                            modifiedRequest.setMetadata(metadata);
                            modifiedRequest.setStore(true);
                        }
                    }
                    default -> {
                    }
                };

                request.setBody(BinaryData.fromObject(modifiedRequest));
            }
            return delegate.sendSync(request, context);
        }
    }

Where the new intercepted requested payload is:

   @Data
    static class InterceptedChatCompletionsOptions {
        // new fields
        @JsonProperty(value = "store")
        private Boolean store;

        @JsonProperty(value = "metadata")
        private Map<String, String> metadata;

        // existing fields supported by Azure SDK belows
        @JsonProperty(value = "messages")
        private List<ChatRequestMessage> messages;

        @JsonProperty(value = "max_tokens")
        private Integer maxTokens;

        @JsonProperty(value = "temperature")
        private Double temperature;

        @JsonProperty(value = "top_p")
        private Double topP;

        @JsonProperty(value = "logit_bias")
        private Map<String, Integer> logitBias;

        @JsonProperty(value = "user")
        private String user;

        @JsonProperty(value = "n")
        private Integer n;

        @JsonProperty(value = "stop")
        private List<String> stop;

        @JsonProperty(value = "presence_penalty")
        private Double presencePenalty;

        @JsonProperty(value = "frequency_penalty")
        private Double frequencyPenalty;

        @JsonProperty(value = "stream")
        private Boolean stream;

        @JsonProperty(value = "model")
        private String model;

        @JsonProperty(value = "seed")
        private Long seed;

        @JsonProperty(value = "response_format")
        private ChatCompletionsResponseFormat responseFormat;

        @JsonProperty(value = "tools")
        private List<ChatCompletionsToolDefinition> tools;

        @JsonProperty(value = "tool_choice")
        private BinaryData toolChoice;

        @JsonProperty(value = "logprobs")
        private Boolean logprobs;

        @JsonProperty(value = "top_logprobs")
        private Integer topLogprobs;
    }

Then you can configure your OpenAI client to use the interceptor http client:

OpenAIClient openaiClient = new OpenAIClientBuilder()
                                .httpClient(new OpenAIInterceptorHttpClient(
                                        HttpClient.createDefault(new HttpClientOptions()
                                                .responseTimeout(Duration.ofSeconds(300))
                                                .setApplicationId("openai"))))
                                .buildClient();

It's a hack, but it's what we have to deal with to be able to both use Azure OpenAI & the regular OpenAI via the same SDK. Hopefully the Azure SDK team can use this example improve their API so that we can easily add new parameters without much customization while waiting for Azure to catch up with OpenAI new feature.