Provide unified thinking config among different providers

[hustxiayang]

Feature Proposal: Unified Thinking/Reasoning Configuration Interface

1. Goal

Provide a single, unified interface for configuring advanced thinking and reasoning settings across different Large Language Model (LLM) providers. This standardization is a key requirement and major benefit of using our gateway proxy, significantly improving the developer experience.

2. Current Status: Vendor-Specific Configuration

Currently, users must manage different configuration fields and structures for similar functionality when calling different vendor APIs. This complexity defeats the purpose of a unified gateway.

for anthropic models, we have

type AnthropicVendorFields struct {
	Thinking *anthropic.ThinkingConfigParamUnion `json:"thinking,omitzero"`
}

type ThinkingConfigParamUnion struct {
	OfEnabled  *ThinkingConfigEnabledParam  `json:",omitzero,inline"`
	OfDisabled *ThinkingConfigDisabledParam `json:",omitzero,inline"`
	paramUnion
}

type ThinkingConfigEnabledParam struct {
	BudgetTokens int64 `json:"budget_tokens,required"`
	Type constant.Enabled `json:"type,required"`
}

type ThinkingConfigDisabledParam struct {
	Type constant.Disabled `json:"type,required"`
}

for gemini models, we have

type GCPVertexAIGenerationConfig struct {
	ThinkingConfig *genai.ThinkingConfig `json:"thinkingConfig,omitzero"`
}
type ThinkingConfig struct {
	IncludeThoughts bool `json:"includeThoughts,omitempty"`
	ThinkingBudget *int32 `json:"thinkingBudget,omitempty"`
}

As a result, users need to use different inference configs for different models. For example:

if (model is a anthropic model){
     client.completions.create(
          ...
          thinking = {"type": "enabled", "budget_tokens": 1024}
          ...
   )
}
elif (model is a gemini model){
      client.completions.create(
          ...
          thinkingConfig = {"thinkingBudget": 1024, "includeThoughts": false}
          ...
      )

}

3. Proposal

3.1 Reason

However, the interface among these two are actually very similar, most of them are the same thing with different names. For example ThinkingConfig and Thinking, ThinkingBudget and BudgetTokens.

3.2 Unified interface

Thus, we propose to use a single unified interface directly in ChatCompletionRequest for both providers. The unified interface is similar to anthropic's thinking config, but extended with includeThoughts.

Here is the proposed definition:

Thinking *ThinkingUnion `json:"thinking,omitzero"`

type ThinkingUnion struct {
	OfEnabled  *ThinkingEnabled  `json:",omitzero,inline"`
	OfDisabled *ThinkingDisabled `json:",omitzero,inline"`
}

type ThinkingEnabled struct {
	BudgetTokens int64 `json:"budget_tokens"`
        Type string `json:"type"`
	IncludeThoughts bool `json:"includeThoughts,omitempty"`
}

type ThinkingDisabled struct {
	Type string `json:"type,"`
}

3.3 Translation

The translation from ThinkingUnion into anthropic's thinking is trivial as it just need to ignore IncludeThoughts and extract other fields.

The translation from ThinkingUnion into gemini models' thinkingConfig is also straightforward: It just need to extract the filed BudgetTokens as thinkingBudget and directly use IncludeThoughts

With this change, users can use the same codes for models from these two providers, which is a key advantage to use a proxy.

4. Further scope of unified interface

Reasoning models from other providers/backends also have different config definitions. For example in vLLM, to disable thinking for qwen3 models, users need to use

client.completions.create(
    ...
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
    ...
)

We can also provide a unified interface for this case, then users can write:

client.completions.create(
    ...
    thinking={"type": disabled"}
    ...
)

and we translate it into extra_body={"chat_template_kwargs": {"enable_thinking": False}} internally.

In summary, we propose to achieve the goal:

• For openai's gpt models, we use reasoning_effort to config thinking parameters.
• For all other backends, including models using anthropic api, converse api, gcp vertex api, vLLM etc, we provide a single unified ThinkingUnion interface.

[mathetake]

Is there any prior art similar to this in the industry? I remember some folks trying to have a "standard" LLM API and i guess we can borrow some ideas from there if the reasoning/thinking related field was discussed there.

[hustxiayang]

@mathetake Yeah, I think liteLLM for example supported a unified interface. https://github.com/BerriAI/litellm/blob/main/litellm/llms/base_llm/chat/transformation.py. They further provide translations between reasoning effort and thinking: for example, reasoning_effort=low can be hard-coded as budget_tokens = 1024.

[nacx]

For openai's gpt models, we use reasoning_effort to config thinking parameters.
For all other backends, including models using anthropic api, converse api, gcp vertex api, vLLM etc, we provide a single unified ThinkingUnion interface.

Doesn't this defeat the goal of the Unified API? Having a unified API for some providers is a bad UX IMHO and makes it harder to learn and adopt the API.

In my opinion, we need to clearly define, at least, the following, in order to implement the right thing for users:

• Do we expect users to only use our Unified API? Or will we still keep the vendor-specific fields, should users want to use "native" requests?
• How will the unified API translate into each of the supported providers.
• What happens when users configure thinking parameters using the unified API and the backend does not support them (I guess simply ignore that, but we need to explicitly define the behavior of the abstractions we make).

[hustxiayang]

@nacx Hi, thanks a lot for your comment!

I think the supported parameters are out of control of us. When using a proxy for models from different providers. Really have a unified API is not achievable because backends are not at our hands. Previously, we have a Openai-first API. Not only we have vendor-specific parameters, but also we have parameters supported in OpenAI, but not supported by some other models. For example echo, n, logprobs , top_logprobs, presence_penalty and so on.

If we set different vendors specific parameters to describe the same thing, for example thinking and thinkingConig to config the thinking parameters, I feel it's quite messy internally, and not convenient for users. Also, it puts a limit to add features too. for example, we are not extend the response_format to support structural_tag.

Generally, it's not avoidable to have a "unified" api, but with some parameters not available for some models. Thus I suggest we can provide an endpoint/api for users to check what API can be used for different providers.

Thanks a lot for your comments!