[Bug] Assistant reasoning_content stripped on forwarding -> 400 (DeepSeek V4 thinking mode)

[RemyHaijie]

Proposed Workgroup

Unsure / Maintainer triage

Describe the bug

Summary

When a request goes through the semantic router (vllm-sr) to an OpenAI-compatible upstream that serves a DeepSeek V4 model (deepseek-v4-flash) over the Chat Completions API, long multi-turn agent / tool-calling conversations start failing with:

HTTP 400
{"message":"The `reasoning_content` in the thinking mode must be passed back to the API.","type":"invalid_request_error","param":"","code":"invalid_request_error"}

The same client request sent directly to the same upstream (same messages, same tools, same reasoning_content fields) succeeds. Capturing the request after the router showed why: the router's ext_proc layer removes reasoning_content from every assistant message before forwarding upstream, so the upstream sees a tool-using conversation history with no reasoning content at all, which DeepSeek-V4-style gateways reject in thinking mode.

Environment

• vllm-sr v0.3 (CLI vllm-sr validate reports Version: v0.3)
• Images: ghcr.io/vllm-project/semantic-router/vllm-sr:latest, .../dashboard:latest, envoyproxy/envoy:v1.34-latest (2026-09)
• Topology: Envoy :8891 -> ext_proc (router) -> upstream OpenAI-compatible gateway (in our case a New API-based self-hosted gateway) -> deepseek-v4-flash

To Reproduce

Repro

1. Configure a cloud model card whose backend is an OpenAI-compatible DeepSeek V4 endpoint, e.g.:

providers:
  models:
    - name: cloud/frontier
      provider_model_id: deepseek-v4-flash
      api_format: openai
      backend_refs:
        - name: cloud-gateway
          base_url: <your-openai-compatible-gateway>/v1
          provider: openai
          type: openai
          api_key_env: OPENAI_API_KEY
          weight: 100

2. Send a multi-turn agent-style request through vllm-sr/auto: several assistant turns with reasoning_content (real text and empty string) plus tool_calls, followed by tool result messages, e.g. 10+ assistant turns / 10 tool calls, max_completion_tokens ~8k.
3. The router selects the cloud backend; the upstream answers 400 with the message above.

Direct vs routed comparison

	messages' assistant reasoning_content
Client request sent to the router	present on every assistant message (15/15, incl. real text and empty strings)
Request as actually forwarded by the router to upstream (captured)	removed on every assistant message (0/15)

model was rewritten (vllm-sr/auto -> deepseek-v4-flash), everything else was preserved — only reasoning_content is stripped.

Notes observed from testing

• The same upstream tolerates the request when it is sent directly (with or without reasoning_content in many shapes), but in thinking mode (the model returns reasoning_content and the conversation uses tools) its strict validation requires the field to be passed back on the assistant messages; once one request fails, later turns in the same conversation keep failing because the (rc-less) assistant messages are replayed.
• Official DeepSeek (api.deepseek.com) currently tolerates missing reasoning_content, which is why the issue is only visible with stricter OpenAI-compatible gateways — but the protocol explicitly documents that reasoning_content must be passed back in thinking mode, so any conforming DeepSeek V4 endpoint will exhibit this.
• Setting use_reasoning: true/false on the model decisions did not change the stripping.

Expected behavior

Add a way to preserve / pass through reasoning_content on assistant messages to the upstream — either per model/backend (e.g. a flag on the model card, since OpenAI-compatible DeepSeek-V4 style endpoints need the round-trip) or a global toggle — and/or document that the router always strips this field so operators can pick a compatible upstream.

Impact

Multi-turn agent / tool-calling traffic (the main use case for routing autonomous agents) routed to DeepSeek V4 thinking-model endpoints over OpenAI-compatible gateways is currently broken after the first tool round.

Additional context

Upstream error response (body returned on the failing request, 173 bytes):

{"message":"The `reasoning_content` in the thinking mode must be passed back to the API.","type":"invalid_request_error","param":"","code":"invalid_request_error"}

Envoy access log line for the failing request (hostnames sanitized):

[2026-09-07T10:34:02.136Z] "POST /v1/chat/completions HTTP/1.1" 400 - 186842 173 496 241 "-" "<client-user-agent>" "<request-id>" "<gateway-host>" "<gateway-ip>:443"

Router decision for the same request (trimmed):

{"msg":"routing_decision","request_id":"<request-id>","original_model":"vllm-sr/auto","decision":"long-context-to-cloud","selected_model":"cloud/frontier","reasoning_enabled":false}

[seven7763]

Verified against current main with an executed roundtrip — I don't think the OpenAI Chat codec is what strips reasoning_content on your shape. Worth ruling that out before anyone patches a codec that already round-trips this field.

What I ran (a test against pkg/protocolcodec with llmprotocol.DefaultPolicy()): decode a Chat Completions request whose assistant turns carry reasoning_content — real text, empty string, and one assistant turn with tool_calls — then rewrite the model (vllm-sr/auto → deepseek-v4-flash) and bump Generation (what any routing mutation does; it retires byte-level replay per Envelope.CanReplay, policy.go:139), then re-encode. All three shapes come back with reasoning_content intact on the assistant messages; the tool-call turn forwards as {"role":"assistant","reasoning_content":"…","tool_calls":[…]}. On current main the strip is not in codec_openai_chat.go's request path: assembleChatMessage maps reasoning_content/reasoning into ContentReasoning, and the encode side re-emits it as reasoning_content (codec_openai_chat_encode.go, ContentReasoning → wire.Reasoning).

Given that, the three places I'd look next, in order:

1. Where the capture was taken. If the "forwarded request" came from the New API gateway's logs, check whether that log line shows what the gateway received or what it forwarded to the DeepSeek backend — New API rebuilds request bodies between those two points. The clean discriminator: point backend_refs at a dump endpoint (netcat / tiny echo server) and capture the bytes the router itself puts on the wire, or tcpdump the router→gateway leg. If the router's own bytes contain reasoning_content and DeepSeek still 400s, the strip is downstream of vllm-sr, not in it.

2. Looper / recipe path. If the decision that actually fires executes through looper workflows, the provider request is rebuilt from extracted content instead of replayed, and assistant-history reasoning_content won't survive that rebuild. Your model card is a plain backend ref, but it's worth confirming the winning decision doesn't attach a looper/algorithm.

3. The extproc wiring, not the codec. My roundtrip called the codec directly; the extproc layer constructs its own envelope/policy (encodeDispatchRequest in processor_protocol_contract.go, and prepareProviderDispatch bumps Generation in processor_req_body_routing.go). Logging ctx.SemanticRequest right before EncodeRequest, or a test through the extproc path with your exact yaml, would close that gap.

A regression-test gap worth closing regardless: TestOpenAIChatDecodesSupportedReasoningAliases covers response decode only. There is no test asserting assistant-history reasoning_content survives a request re-encode with a model rewrite + tools — and that shape (empty-string and non-empty reasoning, one assistant turn with tool_calls) is exactly what agent traffic looks like. Cheap to pin.

FWIW: I run an OpenAI-compatible gateway (daoxe), so I've hit this exact class of problem. Nothing here depends on it — the steps above work with any endpoint.

[Xunzhuo]

/accept

[Xunzhuo]

This behavior is already resolved on current main by #3076: the neutral OpenAI Chat request path preserves assistant reasoning_content through decode, model rewrite, and re-encode. The v0.3 report is valid release evidence, but there is no remaining main-bound fix to accept here.