Support reasoning_effort in the OpenAI client#37428
Merged
Merged
Conversation
Reasoning models count thinking tokens against max_completion_tokens and some default to unbounded dynamic thinking, so without this option there is no way to bound latency or keep thinking from consuming the completion budget through OpenAI-compatible endpoints (observed with Gemini via Vertex AI: 5-9s per call, or truncated JSON when capped by maxTokens). Adds InferenceParameters.OPTION_REASONING_EFFORT and a reasoningEffort config on llm-client, mapped to the chat completions reasoning_effort field. Values pass through as-is since providers accept different sets (OpenAI: minimal/low/medium/high; Google's compat API also takes none). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
onurkaracali
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Reasoning models (Gemini 2.5+/3.x, OpenAI o-series/gpt-5) count thinking tokens against
max_completion_tokens, and some default to unbounded dynamic thinking. Throughai.vespa.llm.clients.OpenAIthere is currently no way to control this: the client maps a fixed set of options and drops everything else, so callers can neither bound latency nor stop thinking from consuming the completion budget.Observed concretely with Gemini via Vertex AI's OpenAI-compatible endpoint (from an LLM query-planning Searcher, where the call is on the query path):
gemini-3.5-flashspends 5–9 s thinking per call, and with a smallmaxTokensthe visible output is truncated mid-JSON because thinking consumed the budget.What
InferenceParameters.OPTION_REASONING_EFFORT(reasoningEffort) + vespajlib abi-spec updatereasoningEffort string default=""inllm-client.defso it can be set per component; per-request option overrides it, matching the other optionsReasoningEffort.of(...)— values pass through as-is, since accepted sets differ per provider (OpenAI:minimal|low|medium|high; Google's compat API also acceptsnone). openai-java 1.6.1 (already pinned) supports this; no dependency change.getChatCompletionCreateParams/prepareParametersmade package-private for testing, consistent with the file's existing test seams.mvn testforai.vespa.llm.clients.*: 89 run, 0 failures.🤖 Generated with Claude Code