[QUERY] [openai] Difference in response for tools call using java and python library to same model with same parameters

[sravanthi-tl]

Query/Question
I have a openai fn defined as tool which makes a function call cancel_booking with all required user details. We are using 2024-02-15-preview model which is deployed on azure. Keeping the tool definition same and model version same along with temperature ... we made calls using

1. py script using openai sdk and
2. azure-ai-openai.1.0.0-beta.8

On user_message: Hello this is Barry Spencer. Im personal assistant to the Smith's. Can you please cancel appointments of Will Smith and Jaden Smith.

1. py script is returning 2 function calls .. cancel_booking(Will Smith), cancel_booking(Jaden Smith) consistently
2. java sdk is returning only one call cancel_booking(Will Smith) even though all params seem to be the same

Am unable to figure out the difference between these 2 calls and why there is a difference in response.

Tool Definition

{
        "system": "You are a cancel bot helping people cancel their appointment. When a user asks for cancelling their booking, you should call cancel_booking function with appropriate parameters. If there are requests for multiple cancels make multiple function calls. Don't make assumptions about what values to use with functions. Ask for clarification if a user request is ambiguous. \n For example\n USER:I would like to cancel two appointments, one that is scheduled on 5th May at 10am on phone +18978191080 on my name Gary Shepard and another appointment for my wife (Noel Shepard) scheduled on 10th May on phone +1234567890.\nOutput: cancel_booking({{Gary Shepard, +18978191080, 5th May at 10am}}), cancel_booking({{Noel Shepard, +1234567890, 10th May}}) ",
        "tool_choice": "auto",
        "tools": [
            {
                "function": {
                    "description": "When a user asks for cancelling their booking, you should call cancel_booking function with appropriate parameters. If there are requests for multiple cancels make multiple function calls. Don't make assumptions about what values to use with functions. Ask for clarification if a user request is ambiguous.",
                    "name": "cancel_booking",
                    "parameters": {
                        "properties": {
                            "date_time_of_appointment": {
                                "description": "time and date  of the appointment that is being cancelled",
                                "type": "string"
                            },
                            "name": {
                                "description": "Name of user whose appointment is to be cancelled, maybe different from caller",
                                "type": "string"
                            },
                            "phone": {
                                "description": "comma separated phone number(s) of user whose appointment is to be cancelled ",
                                "type": "string"
                            }
                        },
                        "required": [
                            "name",
                            "phone"
                        ],
                        "type": "object"
                    }
                },
                "type": "function"
            }

java call

 ChatCompletions chatCompletions = client.getChatCompletions(this.deploymentOrModelId,
                            new ChatCompletionsOptions(chatMessages)
                                    .setTemperature(0.0)
                                    .setTools(toolDefinitionsList)
                                    .setToolChoice(BinaryData.fromObject("auto")));
                                    

log stmt of request sent to openai

May 05 21:02:22.514 [ForkJoinPool.commonPool-worker-9] INFO com.azure.ai.openai.implementation.OpenAIClientImpl$OpenAIClientService.getChatCompletionsSync - {"az.sdk.message":"HTTP response","statusCode":200,"url":"https://HOST.openai.azure.com/openai/deployments/gpt-35-turbo-0125/chat/completions?api-version=2024-02-15-preview","durationMs":1663}

java call response received

py side log of request made and response received

[joshfree]

@mssfang please take a look

[mssfang]

Hi, @sravanthi-tl Thank you for letting us know about this.

I have tested by using OpenAI cookbook parallel tool call sample: https://platform.openai.com/docs/guides/function-calling/parallel-function-calling

Find if I use "gpt-35-turbo-1106", a simple one response returned which is

Final Result: The weather in San Francisco, Tokyo, and Paris is  -7 degrees Celsius.

However, if I use gpt-4-turbo, all three cities' weather are returned.

- **San Francisco:** The temperature is around 12°C with clear skies.
- **Tokyo:** It is approximately 5°C with cloudy conditions.
- **Paris:** The temperature is about -7°C with snow falling.

Please note, the weather can change rapidly, so it’s good to check a reliable source if you need real-time updates.

The sample code pasted in below:

public class ParallelToolCalls {
    public static void main(String[] args) {
        String azureOpenaiKey = Configuration.getGlobalConfiguration().get("AZURE_OPENAI_KEY");
        String endpoint = Configuration.getGlobalConfiguration().get("AZURE_OPENAI_ENDPOINT");
        String deploymentOrModelId = "gpt-35-turbo-1106";
//        String deploymentOrModelId = "gpt-4-turbo";
        OpenAIClient client = new OpenAIClientBuilder()
                .serviceVersion(OpenAIServiceVersion.V2024_02_15_PREVIEW)
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();

        List<ChatRequestMessage> chatMessages = Arrays.asList(
                new ChatRequestSystemMessage("You are a helpful assistant."),
                new ChatRequestUserMessage("What's the weather like in San Francisco, Tokyo, and Paris?")
        );
        ChatCompletionsToolDefinition toolDefinition = new ChatCompletionsFunctionToolDefinition(
                getCurrentWeather());

        ChatCompletionsOptions chatCompletionsOptions = new ChatCompletionsOptions(chatMessages)
                .setTemperature(0.0)
                .setTools(Arrays.asList(toolDefinition))
                .setToolChoice(BinaryData.fromObject("auto"));

        IterableStream<ChatCompletions> chatCompletionsStream = client.getChatCompletionsStream(deploymentOrModelId,
                chatCompletionsOptions);

        String toolCallId = null;
        String functionName = null;
        StringBuilder functionArguments = new StringBuilder();
        CompletionsFinishReason finishReason = null;
        for (ChatCompletions chatCompletions : chatCompletionsStream) {
            // In the case of Azure, the 1st message will contain filter information but no choices sometimes
            if (chatCompletions.getChoices().isEmpty()) {
                continue;
            }
            ChatChoice choice = chatCompletions.getChoices().get(0);
            if (choice.getFinishReason() != null) {
                finishReason = choice.getFinishReason();
            }
            List<ChatCompletionsToolCall> toolCalls = choice.getDelta().getToolCalls();
            // We take the functionName when it's available, and we aggregate the arguments.
            // We also monitor FinishReason for TOOL_CALL. That's the LLM signaling we should
            // call our function
            if (toolCalls != null) {
                ChatCompletionsFunctionToolCall toolCall = (ChatCompletionsFunctionToolCall) toolCalls.get(0);
                if (toolCall != null) {
                    functionArguments.append(toolCall.getFunction().getArguments());
                    if (toolCall.getId() != null) {
                        toolCallId = toolCall.getId();
                    }

                    if (toolCall.getFunction().getName() != null) {
                        functionName = toolCall.getFunction().getName();
                    }
                }
            }
        }

        System.out.println("Tool Call Id: " + toolCallId);
        System.out.println("Function Name: " + functionName);
        System.out.println("Function Arguments: " + functionArguments);
        System.out.println("Finish Reason: " + finishReason);

        // We verify that the LLM wants us to call the function we advertised in the original request
        // Preparation for follow-up with the service we add:
        // - All the messages we sent
        // - The ChatCompletionsFunctionToolCall from the service as part of a ChatRequestAssistantMessage
        // - The result of function tool as part of a ChatRequestToolMessage
        if (finishReason == CompletionsFinishReason.TOOL_CALLS) {
            // Here the "content" can be null if used in non-Azure OpenAI
            // We prepare the assistant message reminding the LLM of the context of this request. We provide:
            // - The tool call id
            // - The function description
            FunctionCall functionCall = new FunctionCall(functionName, functionArguments.toString());
            ChatCompletionsFunctionToolCall functionToolCall = new ChatCompletionsFunctionToolCall(toolCallId, functionCall);
            ChatRequestAssistantMessage assistantRequestMessage = new ChatRequestAssistantMessage("");
            assistantRequestMessage.setToolCalls(Arrays.asList(functionToolCall));

            // As an additional step, you may want to deserialize the parameters, so you can call your function
            FunctionArguments parameters = BinaryData.fromString(functionArguments.toString()).toObject(FunctionArguments.class);
            System.out.println("Location Name: " + parameters.locationName);
            System.out.println("Date: " + parameters.date);
            String functionCallResult = futureTemperature(parameters.locationName, parameters.date);

            // This message contains the information that will allow the LLM to resume the text generation
            ChatRequestToolMessage toolRequestMessage = new ChatRequestToolMessage(functionCallResult, toolCallId);
            List<ChatRequestMessage> followUpMessages = Arrays.asList(
                    // We add the original messages from the request
                    chatMessages.get(0),
                    chatMessages.get(1),
                    assistantRequestMessage,
                    toolRequestMessage
            );

            IterableStream<ChatCompletions> followUpChatCompletionsStream = client.getChatCompletionsStream(
                    deploymentOrModelId, new ChatCompletionsOptions(followUpMessages));

            StringBuilder finalResult = new StringBuilder();
            CompletionsFinishReason finalFinishReason = null;
            for (ChatCompletions chatCompletions : followUpChatCompletionsStream) {
                if (chatCompletions.getChoices().isEmpty()) {
                    continue;
                }
                ChatChoice choice = chatCompletions.getChoices().get(0);
                if (choice.getFinishReason() != null) {
                    finalFinishReason = choice.getFinishReason();
                }
                if (choice.getDelta().getContent() != null) {
                    finalResult.append(choice.getDelta().getContent());
                }
            }

            // We verify that the LLM has STOPPED as a finishing reason
            if (finalFinishReason == CompletionsFinishReason.STOPPED) {
                System.out.println("Final Result: " + finalResult);
            }
        }
    }


    private static Map<String, String> getCurrentWeather(String location) {
        String unit = "fahrenheit";

        // Get the current weather in a given location
        if ("tokyo".equalsIgnoreCase(location)) {
            Map<String, String> res = new HashMap<>();
            res.put("location", "Tokyo");
            res.put("temperature", "10");
            res.put("unit", unit);

            return res;
        }
        else if ("san francisco".equalsIgnoreCase(location)) {
            Map<String, String> res = new HashMap<>();
            res.put("location", "San Francisco");
            res.put("temperature", "72");
            res.put("unit", unit);

            return res;
        }
        else if ("paris".equalsIgnoreCase(location)) {
            Map<String, String> res = new HashMap<>();
            res.put("location", "Paris");
            res.put("temperature", "22");
            res.put("unit", unit);

            return res;
        }
        else {
            Map<String, String> res = new HashMap<>();
            res.put("location", location);
            res.put("temperature", "unknown");

            return res;
        }
    }


    // In this example we ignore the parameters for our tool function
    private static String futureTemperature(String locationName, String data) {
        return "-7 C";
    }

    private static FunctionDefinition getCurrentWeather() {
        FunctionDefinition functionDefinition = new FunctionDefinition("GetCurrentWeather");
        functionDefinition.setDescription("Get the current weather for a given location.");
        FunctionArguments parameters = new FunctionArguments();
        functionDefinition.setParameters(BinaryData.fromObject(parameters));
        return functionDefinition;
    }


    private static FunctionDefinition getFutureTemperatureFunctionDefinition() {
        FunctionDefinition functionDefinition = new FunctionDefinition("FutureTemperature");
        functionDefinition.setDescription("Get the future temperature for a given location and date.");
        FutureTemperatureParameters parameters = new FutureTemperatureParameters();
        functionDefinition.setParameters(BinaryData.fromObject(parameters));
        return functionDefinition;
    }

    private static class FunctionArguments {
        @JsonProperty(value = "location_name")
        private String locationName;

        @JsonProperty(value = "date")
        private String date;
    }

    private static class FutureTemperatureParameters {
        @JsonProperty(value = "type")
        private String type = "object";

        @JsonProperty(value = "properties")
        private FutureTemperatureProperties properties = new FutureTemperatureProperties();
    }

    private static class FutureTemperatureProperties {
        @JsonProperty(value = "unit") StringField unit = new StringField("Temperature unit. Can be either Celsius or Fahrenheit. Defaults to Celsius.");
        @JsonProperty(value = "location_name") StringField locationName = new StringField("The name of the location to get the future temperature for.");
        @JsonProperty(value = "date") StringField date = new StringField("The date to get the future temperature for. The format is YYYY-MM-DD.");
    }

    private static class StringField {
        @JsonProperty(value = "type")
        private final String type = "string";

        @JsonProperty(value = "description")
        private String description;

        @JsonCreator
        StringField(@JsonProperty(value = "description") String description) {
            this.description = description;
        }
    }
}

Which made me wondering what is your sample looks like?

[jayshetti]

Hi, @sravanthi-tl Thank you for letting us know about this.

I have tested by using OpenAI cookbook parallel tool call sample: https://platform.openai.com/docs/guides/function-calling/parallel-function-calling

Find if I use "gpt-35-turbo-1106", a simple one response returned which is

Final Result: The weather in San Francisco, Tokyo, and Paris is  -7 degrees Celsius.

However, if I use gpt-4-turbo, all three cities' weather are returned.

- **San Francisco:** The temperature is around 12°C with clear skies.
- **Tokyo:** It is approximately 5°C with cloudy conditions.
- **Paris:** The temperature is about -7°C with snow falling.

Please note, the weather can change rapidly, so it’s good to check a reliable source if you need real-time updates.

The sample code pasted in below:

public class ParallelToolCalls {
    public static void main(String[] args) {
        String azureOpenaiKey = Configuration.getGlobalConfiguration().get("AZURE_OPENAI_KEY");
        String endpoint = Configuration.getGlobalConfiguration().get("AZURE_OPENAI_ENDPOINT");
        String deploymentOrModelId = "gpt-35-turbo-1106";
//        String deploymentOrModelId = "gpt-4-turbo";
        OpenAIClient client = new OpenAIClientBuilder()
                .serviceVersion(OpenAIServiceVersion.V2024_02_15_PREVIEW)
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();

        List<ChatRequestMessage> chatMessages = Arrays.asList(
                new ChatRequestSystemMessage("You are a helpful assistant."),
                new ChatRequestUserMessage("What's the weather like in San Francisco, Tokyo, and Paris?")
        );
        ChatCompletionsToolDefinition toolDefinition = new ChatCompletionsFunctionToolDefinition(
                getCurrentWeather());

        ChatCompletionsOptions chatCompletionsOptions = new ChatCompletionsOptions(chatMessages)
                .setTemperature(0.0)
                .setTools(Arrays.asList(toolDefinition))
                .setToolChoice(BinaryData.fromObject("auto"));

        IterableStream<ChatCompletions> chatCompletionsStream = client.getChatCompletionsStream(deploymentOrModelId,
                chatCompletionsOptions);

        String toolCallId = null;
        String functionName = null;
        StringBuilder functionArguments = new StringBuilder();
        CompletionsFinishReason finishReason = null;
        for (ChatCompletions chatCompletions : chatCompletionsStream) {
            // In the case of Azure, the 1st message will contain filter information but no choices sometimes
            if (chatCompletions.getChoices().isEmpty()) {
                continue;
            }
            ChatChoice choice = chatCompletions.getChoices().get(0);
            if (choice.getFinishReason() != null) {
                finishReason = choice.getFinishReason();
            }
            List<ChatCompletionsToolCall> toolCalls = choice.getDelta().getToolCalls();
            // We take the functionName when it's available, and we aggregate the arguments.
            // We also monitor FinishReason for TOOL_CALL. That's the LLM signaling we should
            // call our function
            if (toolCalls != null) {
                ChatCompletionsFunctionToolCall toolCall = (ChatCompletionsFunctionToolCall) toolCalls.get(0);
                if (toolCall != null) {
                    functionArguments.append(toolCall.getFunction().getArguments());
                    if (toolCall.getId() != null) {
                        toolCallId = toolCall.getId();
                    }

                    if (toolCall.getFunction().getName() != null) {
                        functionName = toolCall.getFunction().getName();
                    }
                }
            }
        }

        System.out.println("Tool Call Id: " + toolCallId);
        System.out.println("Function Name: " + functionName);
        System.out.println("Function Arguments: " + functionArguments);
        System.out.println("Finish Reason: " + finishReason);

        // We verify that the LLM wants us to call the function we advertised in the original request
        // Preparation for follow-up with the service we add:
        // - All the messages we sent
        // - The ChatCompletionsFunctionToolCall from the service as part of a ChatRequestAssistantMessage
        // - The result of function tool as part of a ChatRequestToolMessage
        if (finishReason == CompletionsFinishReason.TOOL_CALLS) {
            // Here the "content" can be null if used in non-Azure OpenAI
            // We prepare the assistant message reminding the LLM of the context of this request. We provide:
            // - The tool call id
            // - The function description
            FunctionCall functionCall = new FunctionCall(functionName, functionArguments.toString());
            ChatCompletionsFunctionToolCall functionToolCall = new ChatCompletionsFunctionToolCall(toolCallId, functionCall);
            ChatRequestAssistantMessage assistantRequestMessage = new ChatRequestAssistantMessage("");
            assistantRequestMessage.setToolCalls(Arrays.asList(functionToolCall));

            // As an additional step, you may want to deserialize the parameters, so you can call your function
            FunctionArguments parameters = BinaryData.fromString(functionArguments.toString()).toObject(FunctionArguments.class);
            System.out.println("Location Name: " + parameters.locationName);
            System.out.println("Date: " + parameters.date);
            String functionCallResult = futureTemperature(parameters.locationName, parameters.date);

            // This message contains the information that will allow the LLM to resume the text generation
            ChatRequestToolMessage toolRequestMessage = new ChatRequestToolMessage(functionCallResult, toolCallId);
            List<ChatRequestMessage> followUpMessages = Arrays.asList(
                    // We add the original messages from the request
                    chatMessages.get(0),
                    chatMessages.get(1),
                    assistantRequestMessage,
                    toolRequestMessage
            );

            IterableStream<ChatCompletions> followUpChatCompletionsStream = client.getChatCompletionsStream(
                    deploymentOrModelId, new ChatCompletionsOptions(followUpMessages));

            StringBuilder finalResult = new StringBuilder();
            CompletionsFinishReason finalFinishReason = null;
            for (ChatCompletions chatCompletions : followUpChatCompletionsStream) {
                if (chatCompletions.getChoices().isEmpty()) {
                    continue;
                }
                ChatChoice choice = chatCompletions.getChoices().get(0);
                if (choice.getFinishReason() != null) {
                    finalFinishReason = choice.getFinishReason();
                }
                if (choice.getDelta().getContent() != null) {
                    finalResult.append(choice.getDelta().getContent());
                }
            }

            // We verify that the LLM has STOPPED as a finishing reason
            if (finalFinishReason == CompletionsFinishReason.STOPPED) {
                System.out.println("Final Result: " + finalResult);
            }
        }
    }


    private static Map<String, String> getCurrentWeather(String location) {
        String unit = "fahrenheit";

        // Get the current weather in a given location
        if ("tokyo".equalsIgnoreCase(location)) {
            Map<String, String> res = new HashMap<>();
            res.put("location", "Tokyo");
            res.put("temperature", "10");
            res.put("unit", unit);

            return res;
        }
        else if ("san francisco".equalsIgnoreCase(location)) {
            Map<String, String> res = new HashMap<>();
            res.put("location", "San Francisco");
            res.put("temperature", "72");
            res.put("unit", unit);

            return res;
        }
        else if ("paris".equalsIgnoreCase(location)) {
            Map<String, String> res = new HashMap<>();
            res.put("location", "Paris");
            res.put("temperature", "22");
            res.put("unit", unit);

            return res;
        }
        else {
            Map<String, String> res = new HashMap<>();
            res.put("location", location);
            res.put("temperature", "unknown");

            return res;
        }
    }


    // In this example we ignore the parameters for our tool function
    private static String futureTemperature(String locationName, String data) {
        return "-7 C";
    }

    private static FunctionDefinition getCurrentWeather() {
        FunctionDefinition functionDefinition = new FunctionDefinition("GetCurrentWeather");
        functionDefinition.setDescription("Get the current weather for a given location.");
        FunctionArguments parameters = new FunctionArguments();
        functionDefinition.setParameters(BinaryData.fromObject(parameters));
        return functionDefinition;
    }


    private static FunctionDefinition getFutureTemperatureFunctionDefinition() {
        FunctionDefinition functionDefinition = new FunctionDefinition("FutureTemperature");
        functionDefinition.setDescription("Get the future temperature for a given location and date.");
        FutureTemperatureParameters parameters = new FutureTemperatureParameters();
        functionDefinition.setParameters(BinaryData.fromObject(parameters));
        return functionDefinition;
    }

    private static class FunctionArguments {
        @JsonProperty(value = "location_name")
        private String locationName;

        @JsonProperty(value = "date")
        private String date;
    }

    private static class FutureTemperatureParameters {
        @JsonProperty(value = "type")
        private String type = "object";

        @JsonProperty(value = "properties")
        private FutureTemperatureProperties properties = new FutureTemperatureProperties();
    }

    private static class FutureTemperatureProperties {
        @JsonProperty(value = "unit") StringField unit = new StringField("Temperature unit. Can be either Celsius or Fahrenheit. Defaults to Celsius.");
        @JsonProperty(value = "location_name") StringField locationName = new StringField("The name of the location to get the future temperature for.");
        @JsonProperty(value = "date") StringField date = new StringField("The date to get the future temperature for. The format is YYYY-MM-DD.");
    }

    private static class StringField {
        @JsonProperty(value = "type")
        private final String type = "string";

        @JsonProperty(value = "description")
        private String description;

        @JsonCreator
        StringField(@JsonProperty(value = "description") String description) {
            this.description = description;
        }
    }
}

Which made me wondering what is your sample looks like?

@mssfang I tried your sample, if i pass "What's the weather like in Australia?" as prompt i get empty functionArguments, if i pass your prompt "What's the weather like in San Francisco, Tokyo, and Paris?" i see the functionArguments as below

{"location": "San Francisco"}{"location": "Tokyo"}{"location": "Paris"}

Am i missing something here? why this weird behavior?

[brunoborges]

@sravanthi-tl is this still something causing trouble?