[Bug]: Encounter an out of memory error with the tutorial of V0.12.0RC1 for Qwen3-Next

[Aurorahsy]

Your current environment

Hardware environment: Atlas 800I A2 with 64 GB of NPU card memory
Using image: quay.io/ascend/vllm-ascend:v0.12.0rc1-openeuler

🐛 Describe the bug

Command :

vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
--tensor-parallel-size 4 --max-model-len 4096 \
--gpu-memory-utilization 0.85 --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

Error:RuntimeError: NPU out of memory. Tried to allocate 2.00 GiB (NPU 0; 60.96 GiB total capacity; 55.91 GiB already allocated; 55.91 GiB current active; 1.80 GiB free; 58.73 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

[Yikun]

@zhangxinyuehfad Pls try to reproduce this

Looks like cmd already follow the doc: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/Qwen3-Next.html

[Meihan-chen]

The GPU utilization can be set to 0.8 for 4-card inference " --gpu-memory-utilization 0.8 "

[Aurorahsy]

请尽量复现这一点

看起来cmd已经跟着文档做了：https://docs.vllm.ai/projects/ascend/en/latest/tutorials/Qwen3-Next.html

The GPU utilization can be set to 0.8 for 4-card inference " --gpu-memory-utilization 0.8 "

Yes, except for the local weights used, all other parameters were copied from the documentation. Additionally, in the aforementioned image, even when the memory utilization was set to 0.7, the same error still occurred. However, after switching to an Ubuntu-based image, the model could be successfully launched with a memory utilization of 0.7, while a setting of 0.85 still resulted in an "out of memory" error.

[zhangxinyuehfad]

@zhangxinyuehfad Pls try to reproduce this

Looks like cmd already follow the doc: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/Qwen3-Next.html

I reproduce this with v0.12.0rc1-a3-openeuler and v0.12.0rc1-a3

v0.12.0rc1-a3-openeuler

set --gpu-memory-utilization 0.85 failed, need to set --gpu-memory-utilization 0.75

vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --tensor-parallel-size 4 --max-model-len 4096 --gpu-memory-utilization 0.85 --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

--gpu-memory-utilization 0.85 error log

(EngineCore_DP0 pid=804) INFO 12-16 02:10:29 [kv_cache_utils.py:1286] GPU KV cache size: 282,240 tokens
(EngineCore_DP0 pid=804) INFO 12-16 02:10:29 [kv_cache_utils.py:1291] Maximum concurrency for 4,096 tokens per request: 210.07x
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822] WorkerProc hit an exception.
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822] Traceback (most recent call last):
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 817, in worker_busy_loop
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     output = func(*args, **kwargs)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 320, in initialize_from_config
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     self.worker.initialize_from_config(kv_cache_config)  # type: ignore
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 391, in initialize_from_config
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     self.model_runner.initialize_kv_cache(kv_cache_config)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2399, in initialize_kv_cache
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     kv_caches = self.initialize_kv_cache_tensors(kv_cache_config)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2423, in initialize_kv_cache_tensors
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2520, in _allocate_kv_cache_tensors
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     v_tensor = torch.zeros(v_tensor_size,
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822] RuntimeError: NPU out of memory. Tried to allocate 552.00 MiB (NPU 2; 60.97 GiB total capacity; 59.68 GiB already allocated; 59.68 GiB current active; 192.90 MiB free; 59.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822] Traceback (most recent call last):
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 817, in worker_busy_loop
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     output = func(*args, **kwargs)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 320, in initialize_from_config
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     self.worker.initialize_from_config(kv_cache_config)  # type: ignore
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 391, in initialize_from_config
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     self.model_runner.initialize_kv_cache(kv_cache_config)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2399, in initialize_kv_cache
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     kv_caches = self.initialize_kv_cache_tensors(kv_cache_config)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2423, in initialize_kv_cache_tensors
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2520, in _allocate_kv_cache_tensors
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]     v_tensor = torch.zeros(v_tensor_size,
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822]                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=967) ERROR 12-16 02:10:29 [multiproc_executor.py:822] RuntimeError: NPU out of memory. Tried to allocate 552.00 MiB (NPU 2; 60.97 GiB total capacity; 59.68 GiB already allocated; 59.68 GiB current active; 192.90 MiB free; 59.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

gpu-memory-utilization 0.8 error log

Capturing ACL graphs (decode, FULL):  83%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                    | 29/35 [03:53<00:48,  8.04s/it]
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822] WorkerProc hit an exception.
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822] Traceback (most recent call last):
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 817, in worker_busy_loop
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     output = func(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 358, in compile_or_warm_up_model
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     self.model_runner.capture_model()
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 3183, in capture_model
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     self._capture_model()
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 3164, in _capture_model
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     self._capture_aclgraphs(
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 3099, in _capture_aclgraphs
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     self._dummy_run(num_tokens,
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return func(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2252, in _dummy_run
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     hidden_states = self._generate_dummy_run_hidden_states(
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2033, in _generate_dummy_run_hidden_states
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     hidden_states = self.model(input_ids=input_ids,
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/acl_graph.py", line 111, in __call__
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return self.runnable(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_next.py", line 1226, in forward
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     hidden_states = self.model(
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]                     ^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 433, in __call__
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return TorchCompileWithNoGuardsWrapper.__call__(self, *args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 174, in __call__
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return self.forward(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_next.py", line 999, in forward
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     def forward(
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return fn(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return self.optimized_call(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 848, in call_wrapped
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 424, in __call__
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     raise e
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 411, in __call__
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]   File "<eval_with_key>.2", line 346, in forward
(Worker_TP0 pid=3300) ERROR 12-16 02:19:13 [multiproc_executor.py:822]     submod_0 = self.submod_0(l_input_ids_, s72, l_self_modules_embed_tokens_parameters_weight_, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, 

gpu-memory-utilization 0.75 log

(APIServer pid=55412) INFO 12-16 02:27:09 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=55412) INFO:     Started server process [55412]
(APIServer pid=55412) INFO:     Waiting for application startup.
(APIServer pid=55412) INFO:     Application startup complete.
(APIServer pid=55412) INFO 12-16 02:27:26 [chat_utils.py:574] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(EngineCore_DP0 pid=55684) INFO 12-16 02:28:26 [shm_broadcast.py:501] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=55828) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=55868) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=55847) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=55821) INFO 12-16 02:28:59 [acl_graph.py:185] Replaying aclgraph
(Worker_TP3 pid=55868) INFO 12-16 02:28:59 [acl_graph.py:185] Replaying aclgraph
(Worker_TP1 pid=55828) INFO 12-16 02:28:59 [acl_graph.py:185] Replaying aclgraph
(Worker_TP2 pid=55847) INFO 12-16 02:28:59 [acl_graph.py:185] Replaying aclgraph
(APIServer pid=55412) INFO 12-16 02:29:00 [loggers.py:236] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 2.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0%
(APIServer pid=55412) INFO:     127.0.0.1:44130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=55412) INFO 12-16 02:29:10 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=55412) INFO 12-16 02:29:20 [loggers.py:236] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

[root@liteserver-for-vllm-ascend-00002 workspace]# curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
    {"role": "user", "content": "Who are you?"}
  ],
>   "model": "Qwen/Qwen3-Next-80B-A3B-Instruct",
>   "messages": [
>     {"role": "user", "content": "Who are you?"}
>   ],
>   "temperature": 0.6,
>   "top_p": 0.95,
>   "top_k": 20,
>   "max_tokens": 32
> }'
{"id":"chatcmpl-86488254abac4607","object":"chat.completion","created":1765852046,"model":"Qwen/Qwen3-Next-80B-A3B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! I am Qwen, a large-scale language model independently developed by the Tongyi Lab under Alibaba Group. I am designed to assist you in generating various","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":12,"total_tokens":44,"completion_tokens":32,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}[root@liteserver-for-vllm-ascend-00002 workspace]#

v0.12.0rc1-a3

set --gpu-memory-utilization 0.85 or 0.8 failed, need to set --gpu-memory-utilization 0.75

vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --tensor-parallel-size 4 --max-model-len 4096 --gpu-memory-utilization 0.8 --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

gpu-memory-utilization 0.85 error log

(EngineCore_DP0 pid=712) INFO 12-16 02:36:34 [kv_cache_utils.py:1286] GPU KV cache size: 282,240 tokens
(EngineCore_DP0 pid=712) INFO 12-16 02:36:34 [kv_cache_utils.py:1291] Maximum concurrency for 4,096 tokens per request: 210.07x
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822] WorkerProc hit an exception.
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822] Traceback (most recent call last):
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 817, in worker_busy_loop
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     output = func(*args, **kwargs)
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 320, in initialize_from_config
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     self.worker.initialize_from_config(kv_cache_config)  # type: ignore
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 391, in initialize_from_config
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     self.model_runner.initialize_kv_cache(kv_cache_config)
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2399, in initialize_kv_cache
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     kv_caches = self.initialize_kv_cache_tensors(kv_cache_config)
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2423, in initialize_kv_cache_tensors
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     kv_cache_raw_tensors = self._allocate_kv_cache_tensors(kv_cache_config)
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2520, in _allocate_kv_cache_tensors
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]     v_tensor = torch.zeros(v_tensor_size,
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822]                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=849) ERROR 12-16 02:36:34 [multiproc_executor.py:822] RuntimeError: NPU out of memory. Tried to allocate 552.00 MiB (NPU 0; 60.97 GiB total capacity; 59.68 GiB already allocated; 59.68 GiB current active; 180.32 MiB free; 59.76 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

gpu-memory-utilization 0.8 error log

(APIServer pid=2807) INFO 12-16 02:44:59 [chat_utils.py:574] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822] WorkerProc hit an exception.
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822] Traceback (most recent call last):
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 817, in worker_busy_loop
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     output = func(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/v1/worker/worker_base.py", line 369, in execute_model
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self.worker.execute_model(scheduler_output, *args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 295, in execute_model
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     output = self.model_runner.execute_model(scheduler_output,
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return func(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1529, in execute_model
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     hidden_states = self._generate_process_reqs_hidden_states(
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1178, in _generate_process_reqs_hidden_states
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     hidden_states = self.model(input_ids=input_ids,
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm-ascend/vllm_ascend/compilation/acl_graph.py", line 111, in __call__
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self.runnable(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_next.py", line 1226, in forward
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     hidden_states = self.model(
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]                     ^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/compilation/decorators.py", line 433, in __call__
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return TorchCompileWithNoGuardsWrapper.__call__(self, *args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/compilation/wrapper.py", line 174, in __call__
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self.forward(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/model_executor/models/qwen3_next.py", line 999, in forward
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     def forward(
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 929, in _fn
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return fn(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/vllm-workspace/vllm/vllm/compilation/caching.py", line 54, in __call__
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self.optimized_call(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 848, in call_wrapped
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self._wrapped_call(self, *args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 424, in __call__
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     raise e
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/fx/graph_module.py", line 411, in __call__
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]   File "<eval_with_key>.2", line 346, in forward
(Worker_TP0 pid=3216) ERROR 12-16 02:45:12 [multiproc_executor.py:822]     submod_0 = self.submod_0(l_input_ids_, s72, l_self_modules_embed_tokens_parameters_wei

gpu-memory-utilization 0.75 log

(APIServer pid=66629) INFO 12-16 02:52:31 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=66629) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-Next-80B-A3B-Instruct
(APIServer pid=66629) INFO 12-16 02:52:32 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=66629) Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3-Next-80B-A3B-Instruct
(APIServer pid=66629) INFO 12-16 02:52:34 [serving_chat.py:133] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=66629) INFO 12-16 02:52:34 [api_server.py:1847] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:38] Available routes are:
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=66629) INFO 12-16 02:52:34 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=66629) INFO:     Started server process [66629]
(APIServer pid=66629) INFO:     Waiting for application startup.
(APIServer pid=66629) INFO:     Application startup complete.
(APIServer pid=66629) INFO 12-16 02:54:35 [chat_utils.py:574] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP3 pid=67085) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP1 pid=67045) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(EngineCore_DP0 pid=66901) INFO 12-16 02:55:35 [shm_broadcast.py:501] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP2 pid=67064) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP0 pid=67038) [WARNING] Please DO NOT tune args ['num_warps', 'num_stages']!
(Worker_TP2 pid=67064) INFO 12-16 02:55:44 [acl_graph.py:185] Replaying aclgraph
(Worker_TP1 pid=67045) INFO 12-16 02:55:44 [acl_graph.py:185] Replaying aclgraph
(Worker_TP3 pid=67085) INFO 12-16 02:55:44 [acl_graph.py:185] Replaying aclgraph
(Worker_TP0 pid=67038) INFO 12-16 02:55:44 [acl_graph.py:185] Replaying aclgraph
(APIServer pid=66629) INFO:     127.0.0.1:55124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=66629) INFO 12-16 02:55:54 [loggers.py:236] Engine 000: Avg prompt throughput: 1.2 tokens/s, Avg generation throughput: 3.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

root@liteserver-for-vllm-ascend-00002:/workspace# curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-Next-80B-A3B-Instruct",
  "messages": [
    {"role": "user", "content": "Who are you?"}
  ],
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "max_tokens": 32
}'
{"id":"chatcmpl-9f2a724e30bd6db8","object":"chat.completion","created":1765853675,"model":"Qwen/Qwen3-Next-80B-A3B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! I am Qwen, a large-scale language model independently developed by the Tongyi Lab under Alibaba Group. I am designed to assist you in generating various","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null,"reasoning_content":null},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":12,"total_tokens":44,"completion_tokens":32,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00002:/workspace#