[Bug]: vllm server crashed when use "--kv-offloading-backend native --kv-offloading-size 300" in long context

[zhushidi666]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.1.52-9-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20

Nvidia driver version        : 535.161.08
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             384
On-line CPU(s) list:                0-383
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9K84 96-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 2
Core(s) per socket:                 96
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        3701.9529
CPU min MHz:                        1500.0000
BogoMIPS:                           5200.25
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                     AMD-V
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           192 MiB (192 instances)
L3 cache:                           768 MiB (24 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-95,192-287
NUMA node1 CPU(s):                  96-191,288-383
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Vulnerable
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:           Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.11.post2
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cccl==13.3.3.3.1
[pip3] nvidia-cuda-crt==13.3.33
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.3.33
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.3.33
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.7.10.post20260505
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.9.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.22.0
vLLM Build Flags:
  CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	GPU1	GPU2	GPU3	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU1	NV18	 X 	NV18	NV18	NODE	PIX	PHB	NODE	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU2	NV18	NV18	 X 	NV18	NODE	PHB	PIX	NODE	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
GPU3	NV18	NV18	NV18	 X 	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	0-95,192-287	0		N/A
NIC0	PIX	NODE	NODE	NODE	 X 	NODE	NODE	NODE	SYS	SYS	SYS	SYS				
NIC1	NODE	PIX	PHB	NODE	NODE	 X 	PHB	NODE	SYS	SYS	SYS	SYS				
NIC2	NODE	PHB	PIX	NODE	NODE	PHB	 X 	NODE	SYS	SYS	SYS	SYS				
NIC3	NODE	NODE	NODE	PIX	NODE	NODE	NODE	 X 	SYS	SYS	SYS	SYS				
NIC4	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	NODE	NODE	NODE				
NIC5	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	 X 	NODE	NODE				
NIC6	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	 X 	PHB				
NIC7	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	NODE	NODE	PHB	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_bond_0
  NIC1: mlx5_bond_1
  NIC2: mlx5_bond_2
  NIC3: mlx5_bond_3
  NIC4: mlx5_bond_4
  NIC5: mlx5_bond_5
  NIC6: mlx5_bond_6
  NIC7: mlx5_bond_7

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=1,2,3,0
NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=13.0.2
VLLM_ENABLE_CUDA_COMPATIBILITY=0
NVIDIA_DISABLE_REQUIRE=1
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/lib
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_***

🐛 Describe the bug

comand：VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 vllm serve /mnt/autovision-cbs/DeepSeek-V4-Flash --port 30000 --served-model-name default --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --tensor-parallel-size 4 --max-num-seqs 512 --max-num-batched-tokens 4096 --no-enable-flashinfer-autotune --compilation-config '{"mode": 0, "cudagraph_mode": "FULL_DECODE_ONLY"}' --gpu-memory-utilization 0.95 --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --speculative_config '{"method":"mtp","num_speculative_tokens":2}' --enable-log-requests --enable-logging-iteration-details --enable-expert-parallel --enable-prefix-caching --kv-offloading-backend native --kv-offloading-size 300

evaluation command：
evalscope eval --eval-type openai_api --model default --api-url http://10.74.45.28:30000/v1 --eval-batch-size 1 --datasets longbench_v2 --dataset-args '{"longbench_v2":{"subset_list":["long"]}}' --generation-config '{"temperature":1.0,"top_p":1.0,"extra_body": {"chat_template_kwargs":{"thinking": true}}}'

APIServer pid=11323) INFO 06-07 12:04:02 [async_llm.py:721] Aborted request(s) chatcmpl-8578d05b1c52d296-a6413c84.
(APIServer pid=11323) INFO 06-07 12:04:02 [async_llm.py:595] Request chatcmpl-8578d05b1c52d296 aborted.
(EngineCore pid=11599) INFO 06-07 12:04:02 [core.py:407] Iteration(57795): 0 context requests, 0 context tokens, 1 generation requests, 3 generation tokens, iteration elapsed time: 6.56 ms
(EngineCore pid=11599) INFO 06-07 12:04:02 [core.py:407] Iteration(57796): 0 context requests, 0 context tokens, 1 generation requests, 3 generation tokens, iteration elapsed time: 0.24 ms
(EngineCore pid=11599) INFO 06-07 12:04:02 [core.py:407] Iteration(57797): 0 context requests, 0 context tokens, 0 generation requests, 0 generation tokens, iteration elapsed time: 0.12 ms
(APIServer pid=11323) INFO 06-07 12:04:03 [logger.py:63] Received request chatcmpl-94a14e5f589d91c5: params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=False, max_tokens=628319, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None), lora_request: None.
(APIServer pid=11323) INFO 06-07 12:04:03 [async_llm.py:415] Added request chatcmpl-94a14e5f589d91c5-b7763ff8.
!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in cuMemcpyBatchAsync
  File "<unknown>", line 0, in swap_blocks_batch(at::Tensor const&, at::Tensor const&, at::Tensor const&, bool)
  File "<unknown>", line 0, in c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<void (*)(at::Tensor const&, at::Tensor const&, at::Tensor const&, bool), void, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, at::Tensor const&, bool> >, false>::call(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*)
  File "<unknown>", line 0, in c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const [clone .isra.0]
  File "<unknown>", line 0, in torch::jit::invokeOperatorFromPython(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, pybind11::args const&, pybind11::kwargs const&, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in torch::jit::_get_operation_for_overload_or_packet(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptr<torch::jit::Operator>, std::allocator<std::shared_ptr<torch::jit::Operator> > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args const&, pybind11::kwargs const&)#1}, pybind11::object, pybind11::args const&, pybind11::kwargs const&, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args const&, pybind11::kwargs const&)#1}&&, pybind11::object (*)(pybind11::args const&, pybind11::kwargs const&), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#1}::_FUN(pybind11::detail::function_call&)
  File "<unknown>", line 0, in pybind11::cpp_function::dispatcher(_object*, _object*, _object*)
  File "<unknown>", line 0, in _PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_FastCallDictTstate
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in PyEval_EvalCode
  File "<unknown>", line 0, in PyRun_StringFlags
  File "<unknown>", line 0, in PyRun_SimpleStringFlags
  File "<unknown>", line 0, in Py_RunMain
  File "<unknown>", line 0, in Py_BytesMain
  File "<unknown>", line 0, in _start
  File "<unknown>", line 0, in 0xffffffffffffffff

!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in cuMemcpyBatchAsync
  File "<unknown>", line 0, in swap_blocks_batch(at::Tensor const&, at::Tensor const&, at::Tensor const&, bool)
  File "<unknown>", line 0, in c10::impl::make_boxed_from_unboxed_functor<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<void (*)(at::Tensor const&, at::Tensor const&, at::Tensor const&, bool), void, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, at::Tensor const&, bool> >, false>::call(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*)
  File "<unknown>", line 0, in c10::Dispatcher::callBoxed(c10::OperatorHandle const&, std::vector<c10::IValue, std::allocator<c10::IValue> >*) const [clone .isra.0]
  File "<unknown>", line 0, in torch::jit::invokeOperatorFromPython(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, pybind11::args const&, pybind11::kwargs const&, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in torch::jit::_get_operation_for_overload_or_packet(c10::ArrayRef<std::shared_ptr<torch::jit::Operator> >, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptr<torch::jit::Operator>, std::allocator<std::shared_ptr<torch::jit::Operator> > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optional<c10::DispatchKey>)
  File "<unknown>", line 0, in pybind11::cpp_function::initialize<torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args const&, pybind11::kwargs const&)#1}, pybind11::object, pybind11::args const&, pybind11::kwargs const&, pybind11::name, pybind11::doc>(torch::jit::initJITBindings(_object*)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#2}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const::{lambda(pybind11::args const&, pybind11::kwargs const&)#1}&&, pybind11::object (*)(pybind11::args const&, pybind11::kwargs const&), pybind11::name const&, pybind11::doc const&)::{lambda(pybind11::detail::function_call&)#1}::_FUN(pybind11::detail::function_call&)
  File "<unknown>", line 0, in pybind11::cpp_function::dispatcher(_object*, _object*, _object*)
  File "<unknown>", line 0, in _PyObject_Call
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyObject_FastCallDictTstate
  File "<unknown>", line 0, in _PyObject_Call_Prepend
  File "<unknown>", line 0, in _PyObject_MakeTpCall
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in _PyEval_EvalFrameDefault
  File "<unknown>", line 0, in PyEval_EvalCode
  File "<unknown>", line 0, in PyRun_StringFlags
  File "<unknown>", line 0, in PyRun_SimpleStringFlags
  File "<unknown>", line 0, in Py_RunMain
  File "<unknown>", line 0, in Py_BytesMain
  File "<unknown>", line 0, in _start
  File "<unknown>", line 0, in 0xffffffffffffffff

(EngineCore pid=11599) ERROR 06-07 12:04:06 [multiproc_executor.py:283] Worker proc VllmWorker-2 died unexpectedly, shutting down executor.
(Worker_TP0_EP0 pid=11739) INFO 06-07 12:04:06 [multiproc_executor.py:775] Parent process exited, terminating worker queues
(Worker_TP3_EP3 pid=11742) INFO 06-07 12:04:06 [multiproc_executor.py:872] WorkerProc shutting down.
(Worker_TP0_EP0 pid=11739) INFO 06-07 12:04:06 [multiproc_executor.py:872] WorkerProc shutting down.
(APIServer pid=11323) INFO 06-07 12:04:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 94.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
(APIServer pid=11323) INFO 06-07 12:04:07 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 2.83, Accepted throughput: 61.00 tokens/s, Drafted throughput: 66.80 tokens/s, Accepted: 610 tokens, Drafted: 668 tokens, Per-position acceptance rate: 0.994, 0.832, Avg Draft acceptance rate: 91.3%
(APIServer pid=11323) INFO 06-07 12:04:07 [metrics.py:103] KV Transfer metrics: GPU_to_CPU_total_bytes=233522176, GPU_to_CPU_total_time=0.01344751998782158
(EngineCore pid=11599) ERROR 06-07 12:04:14 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.22.0) with config: model='/mnt/autovision-cbs/DeepSeek-V4-Flash', speculative_config=SpeculativeConfig(method='mtp', model='/mnt/autovision-cbs/DeepSeek-V4-Flash', num_spec_tokens=2), tokenizer='/mnt/autovision-cbs/DeepSeek-V4-Flash', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=1048576, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=True), seed=0, served_model_name=default, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'all', '+quant_fp8', '+quant_fp8', '+quant_fp8'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [256, 256, 4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_DECODE_ONLY: (2, 0)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=False, moe_backend='auto', linear_backend='auto'), 
(EngineCore pid=11599) ERROR 06-07 12:04:14 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={}, total_num_scheduled_tokens=0, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 0, 0], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=OffloadingConnectorMetadata(load_jobs={12684: TransferJob(req_id='chatcmpl-94a14e5f589d91c5-b7763ff8', transfer_spec=[CPULoadStoreSpec(block_ids=array([54722, 54723, 37112, ..., 46293, 46294, 46295], shape=(1663,))), GPULoadStoreSpec(block_ids=array([43696, 40476, 42880, ..., 47458,  1151,  1149], shape=(1663,)), group_sizes=[1641, 2, 2, 2, 16], block_indices=[0, 6562, 6562, 105022, 52496])])}, store_jobs={}, jobs_to_flush=[]), ec_connector_metadata=null, new_block_ids_to_zero=null)
(EngineCore pid=11599) ERROR 06-07 12:04:14 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=0, num_waiting_reqs=0, num_skipped_waiting_reqs=1, step_counter=0, current_wave=0, kv_cache_usage=0.03480098773699414, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=420257, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=420257, hits=420096, preempted_requests=0, preempted_queries=0, preempted_hits=0), kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167] EngineCore encountered a fatal error.
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167] Traceback (most recent call last):
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1158, in run_engine_core
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     engine_core.run_busy_loop()
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1199, in run_busy_loop
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     self._process_engine_step()
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1238, in _process_engine_step
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     outputs, model_executed = self.step_fn()
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]                               ^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 547, in step_with_batch_queue
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     model_output = future.result()
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]                    ^^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 90, in result
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     return super().result()
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]            ^^^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     return self.__get_result()
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     raise self._exception
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 94, in _wait_for_response
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     response = self.aggregate(self.get_response())
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]                               ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 386, in get_response
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     status, result = mq.dequeue(timeout=dequeue_timeout)
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 772, in dequeue
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     with self.acquire_read(timeout, indefinite) as buf:
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     return next(self.gen)
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]            ^^^^^^^^^^^^^^
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 694, in acquire_read
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167]     raise RuntimeError("cancelled")
(EngineCore pid=11599) ERROR 06-07 12:04:14 [core.py:1167] RuntimeError: cancelled
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704]     outputs = await engine_core.get_output_async()
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 1030, in get_output_async
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704]     raise self._format_exception(outputs) from None
(APIServer pid=11323) ERROR 06-07 12:04:14 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(APIServer pid=11323) INFO 06-07 12:04:14 [async_llm.py:601] Request chatcmpl-94a14e5f589d91c5 failed (engine dead).
(APIServer pid=11323) INFO:     10.16.212.240:42086 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=11323) INFO:     Shutting down
(APIServer pid=11323) INFO:     Waiting for application shutdown.
(APIServer pid=11323) INFO:     Application shutdown complete.
(APIServer pid=11323) INFO:     Finished server process [11323]
(APIServer pid=11323) Exception ignored in: <function AsyncLLM.__del__ at 0x7f32495c7560>
(APIServer pid=11323) Traceback (most recent call last):
(APIServer pid=11323)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 257, in __del__
(APIServer pid=11323)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 261, in shutdown
(APIServer pid=11323) TypeError: 'NoneType' object is not callable
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[JOSH1024]

hi i have uploaded the solution for this issue please review it and suggest me if any changes.