[Bug]: deepseek_v4 parser: reply without </think> routes the whole answer to reasoning_content (content empty), trailing EOS not stripped

[fank]

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-4.18.0-553.104.1.el8_10.x86_64-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H100 NVL
GPU 1: NVIDIA H100 NVL

Nvidia driver version        : 580.95.05
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               128
On-line CPU(s) list:                  0-127
Vendor ID:                            GenuineIntel
Model name:                           INTEL(R) XEON(R) GOLD 6530
CPU family:                           6
Model:                                207
Thread(s) per core:                   2
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             2
CPU max MHz:                          4000.0000
CPU min MHz:                          800.0000
BogoMIPS:                             4200.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            3 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             128 MiB (64 instances)
L3 cache:                             320 MiB (2 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-31,64-95
NUMA node1 CPU(s):                    32-63,96-127
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Vulnerability Vmscape:                Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.13
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cccl-cu12==12.9.27
[pip3] nvidia-cuda-crt==13.3.73
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvcc==13.2.78
[pip3] nvidia-cuda-nvcc-cu12==12.9.86
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime==13.3.29
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cuda-tileiras==13.2.78
[pip3] nvidia-cudnn-cu12==9.17.1.4
[pip3] nvidia-cudnn-frontend==1.26.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-ml-py==13.610.43
[pip3] nvidia-nccl-cu12==2.28.9
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] nvidia-nvvm==13.2.78
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.8.10.post20260709
[pip3] torch==2.11.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu129
[pip3] torchcodec==0.14.0
[pip3] torchvision==0.26.0+cu129
[pip3] transformers==5.13.1
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.23.1rc1.dev1060+g9e57de719 (git sha: 9e57de719)
vLLM Build Flags:
  CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	�[4mGPU0	GPU1	NIC0	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	NV12	NODE	0-31,64-95	0		N/A
GPU1	NV12	 X 	NODE	0-31,64-95	0		N/A
NIC0	NODE	NODE	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_bond_0

==============================
     Environment Variables
==============================
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
CUDA_VERSION=12.9.1
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_ENABLE_CUDA_COMPATIBILITY=0
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0
VLLM_USAGE_SOURCE=production-docker-image
VLLM_BUILD_COMMIT=9e57de7197f234f9d9187715d96e07e007048c0f
VLLM_BUILD_PIPELINE=019d130e-464e-4ff7-b84b-492992c0c06b
VLLM_BUILD_URL=https://buildkite.com/vllm/release-v2/builds/3812
VLLM_IMAGE_TAG=vllm/vllm-openai:cu129-nightly-9e57de7197f234f9d9187715d96e07e007048c0f
VLLM_USE_V2_MODEL_RUNNER=1
VLLM_LOGGING_LEVEL=DEBUG
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_USE_DEEP_GEMM_E8M0=1
VLLM_USE_BREAKABLE_CUDAGRAPH=1

Serve args (not covered by collect_env):

--reasoning-parser deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice
--default-chat-template-kwargs={"thinking":true,"reasoning_effort":"high"}
--tokenizer-mode deepseek_v4 --trust-remote-code --tensor-parallel-size 2
--max-model-len 262144 --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.95
--max-num-seqs 4 --max-num-batched-tokens 2048 --max-cudagraph-capture-size 32
--attention_config.use_fp4_indexer_cache False
--speculative-config={"method":"dspark","num_speculative_tokens":5}
--compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE"}

Model: deepseek-ai/DeepSeek-V4-Flash-DSpark. vllm/parser/deepseek_v4.py is byte-identical on current main, so this is not a stale-nightly artifact.

The bug

When the deepseek_v4 reasoning parser is constructed with thinking=true, a response in which the model never emits </think> has its entire answer routed to reasoning_content, leaving content empty. The trailing <｜end▁of▁sentence｜> (EOS) is also left un-stripped at the end of reasoning_content.

The stray EOS is the visible symptom, but the misrouting is the actual bug: the user-facing answer never reaches content. Chat UIs render it as collapsed "thinking" text, and agents that read message.content receive nothing at all. Stripping the EOS alone would not fix this — it would leave the answer buried in the reasoning channel.

Observed with a real client (pi), where a greeting rendered entirely as grey "thinking" text with no answer:

{"role": "assistant",
 "reasoning_content": "Good morning! How can I help you today?<｜end▁of▁sentence｜>",
 "content": ""}

Expected: content = "Good morning! How can I help you today?", reasoning_content empty (the model did not deliberate), and no EOS in either field.

Root cause

In vllm/parser/deepseek_v4.py#L128:

initial_state=ParserState.REASONING if thinking else ParserState.CONTENT,

With thinking=true the engine starts in ParserState.REASONING, and the only exit is the THINK_END (</think>) transition at L160. DeepSeek-V4-Flash does not always emit <think>…</think>: on short replies it sometimes answers directly and hits EOS with no </think> at all. The state machine then never leaves REASONING, so every chunk is emitted as REASONING_CHUNK and content stays empty. The terminal EOS is not dropped on that path either.

The initial_state assumption holds only if the model is guaranteed to emit </think> whenever thinking is enabled. It isn't — thinking=true gates the template/parser, not whether the model actually deliberates.

I have not traced why the EOS specifically survives here; streaming carries real token ids, so the id-based drop would be expected to fire. Reporting that as observed rather than explained.

Why clients hit this

The parser reads thinking from chat_template_kwargs (L220-L222). Any deployment setting --default-chat-template-kwargs={"thinking":true,...} (as the DeepSeek-V4-Flash recipe suggests) applies it to every request — including from OpenAI-compatible clients that never send chat_template_kwargs and have no notion of the kwarg. That is the common case for agent clients, so the server default silently puts all of their requests on the REASONING-start path.

Reproduction

Plain OpenAI-compatible streaming request, no chat_template_kwargs in the request (so the server default thinking=true applies):

body = {
    "model": "DeepSeek-V4-Flash",
    "stream": True,
    "messages": [
        {"role": "system", "content": "<agent-style system prompt, ~23k chars>"},
        {"role": "user", "content": "good morning"},
    ],
}
# Accumulate delta["content"] and delta["reasoning_content"] across the stream.
# Misroute signature: content empty AND reasoning_content holds the answer + EOS.

Rates over 20-24 samples each, temperature 1.0 (the V4 default):

request	misrouted
agent system prompt + 50 tool defs (real client request)	4/20 and 6/24 (~20-25%)
same system prompt, tools removed	2/24
generic short system prompt + the same 50 tool defs	0/24
no system prompt ("good morning" alone)	0/24
explicit chat_template_kwargs={"thinking":false} (CONTENT-start path)	0/20

It is sampling-dependent and provoked by a realistic agent system prompt that biases toward answering immediately; tool definitions raise the rate but do not trigger it alone. A long synthetic prompt (51k chars of repeated filler) did not reproduce it — prompt length is not the trigger, the instruction content is. That is why I can't reduce it to a self-contained prompt to paste here.

Given that sampling dependence, the deterministic repro is at the parser level: construct the parser with thinking=True and feed it a model output containing no </think>, e.g. "Good morning! How can I help you today?<｜end▁of▁sentence｜>". Current behaviour routes all of it to reasoning; expected is content.

Suggested fix

Roughly in order of preference:

1. Treat a response with no </think> as content. If generation ends while still in REASONING and no THINK_START/THINK_END was ever seen, flush the buffered reasoning to content. Fixes the "model didn't think" case without touching the deliberating case.
2. Only enter REASONING on evidence of thinking — keep initial_state=CONTENT and rely on the existing (CONTENT, THINK_START) -> REASONING transition. Depends on whether the chat template pre-fills an opening <think> the parser never sees.
3. Independently, drop the terminal EOS on the reasoning path as is done on the content path.

Happy to test a patch against this deployment and report rates back — it reproduces reliably enough at ~20-25% to validate a fix.

Related

• #45877 — ported DeepSeek V4/V3.2 to the streaming parser engine (merged 2026-07-04). This report is against the merged code, not a regression from it.
• #46875 — general "special tokens don't leak in tool parsing" rework; does not cover this path.

[fank]

Adding a runnable repro: https://gist.github.com/fank/5f7d85df63956c52cf9fa08f8f140576

export VLLM_BASE_URL=http://localhost:8000/v1
export VLLM_API_KEY=dummy
python3 repro_48645.py 24

It replays the real coding-agent request (system prompt + 50 tool definitions from pi, scrubbed of local paths) and reports the misroute signature — content empty while reasoning_content holds the answer plus the trailing EOS. On the deployment described above it reports 11/24, and the two files are self-contained apart from the endpoint env vars.

The repro_payload.json half matters as much as the script: as noted in the issue, a generic or synthetic system prompt does not trigger this (0/24, including 51k chars of filler). What provokes it is an agent prompt that tells the model to answer directly and without preamble, so on a trivial greeting it sometimes replies without ever emitting </think>. The request sends no chat_template_kwargs, so the server default thinking=true applies — the ordinary case for OpenAI-compatible agent clients.

Sample output:

misrouted (content empty + answer&EOS in reasoning): 11/24
  reasoning='Good morning! How can I help you today?<｜end▁of▁sentence｜>'
  content=''

[fank]

@bbrowning I think I found another leak by accident, hard to reproduce, repro should be attached. I just updated my vllm version to ensure I am up2date with latest changes.