Skip to content

Handle OpenAI tool_choice requests#1070

Draft
eloe wants to merge 1 commit intoBlaizzy:mainfrom
eloe:codex/issue-1045-tool-choice
Draft

Handle OpenAI tool_choice requests#1070
eloe wants to merge 1 commit intoBlaizzy:mainfrom
eloe:codex/issue-1045-tool-choice

Conversation

@eloe
Copy link
Copy Markdown

@eloe eloe commented Apr 25, 2026

Summary

Addresses #1045 by adding explicit OpenAI-compatible tool_choice handling to the /chat/completions and /responses endpoints.

This follows the approach discussed in the issue thread:

  • tool_choice omitted: preserve existing behavior and pass tools through when provided.
  • tool_choice: "none": omit tools from chat-template rendering and suppress tool-call parsing.
  • tool_choice: "auto": render tools normally and parse model-emitted tool calls.
  • tool_choice: "required" and named function choices: return a clear 400 because current VLM chat templates do not provide a reliable way to force required or pinned tool use.

Why

The larger tool-capable VLMs we checked, including Qwen3.5/Qwen3.6 and Gemma-4/Hermes-Gemma-4 variants, support tool schemas in their chat templates, but the templates do not consume tool_choice to change prompt rendering. Accepting none and auto improves compatibility with OpenAI-style clients while avoiding a false guarantee for unsupported enforcement modes.

The Responses API path also now emits parsed function-call output items when the loaded template has a supported parser, and streaming response.completed events preserve the final output_text aggregate.

Validation

  • uv run --with pytest python -m pytest mlx_vlm/tests/test_server.py
    • 51 passed, 3 warnings
  • uv run --with pytest python -m pytest mlx_vlm/tests/test_server.py mlx_vlm/tests/test_gemma4_tool_parser.py
    • 58 passed, 3 warnings
  • python3 -m compileall -q mlx_vlm/server.py mlx_vlm/tests/test_server.py
  • git diff --check

Warnings were the existing MLX/Pydantic serializer warnings observed in the local test environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant