[Bug]: TPU Inference issue - Qwen3-VL-8B

[Dineshkumar-Anandan-ZS0367]

Your current environment

TPU Info:
TPU info: node_name=v6e-test | tpu_type=v6e-4 | worker_id=0 | num_chips=4 | num_cores_per_chip=1

Command to start vLLM server:
docker run -it --rm
--net=host
--privileged
-v /dev:/dev
-v $HOME/.cache/huggingface:/root/.cache/huggingface
vllm/vllm-tpu:nightly-20260303-1f005dc-4034c3d
vllm serve Qwen/Qwen3-VL-8B-Instruct
--tensor-parallel-size 4
--dtype bfloat16
--max-model-len 22528
--max-num-seqs 16
--host 0.0.0.0
--port 7000
--trust-remote-code
--enable-log-requests

🐛 Describe the bug

Issue:

(APIServer pid=1) INFO 03-04 01:49:51 [async_llm.py:421] Added request chatcmpl-b61864175ca7091d-a87942ae.
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] Traceback (most recent call last):
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 1093, in run_engine_core
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] engine_core.run_busy_loop()
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 1128, in run_busy_loop
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] self._process_engine_step()
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 1165, in _process_engine_step
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 507, in step_with_batch_queue
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] engine_core_outputs = self.scheduler.update_from_output(
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] File "/workspace/vllm/vllm/v1/core/sched/scheduler.py", line 1316, in update_from_output
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] req_index = model_runner_output.req_id_to_index[req_id]
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-04 01:49:51 [core.py:1102] KeyError: 'chatcmpl-b61864175ca7091d-a87942ae'
(EngineCore_DP0 pid=564) Process EngineCore_DP0:
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] AsyncLLM output_handler failed.
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] Traceback (most recent call last):
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] File "/workspace/vllm/vllm/v1/engine/async_llm.py", line 664, in output_handler
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] outputs = await engine_core.get_output_async()
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] File "/workspace/vllm/vllm/v1/engine/core_client.py", line 1004, in get_output_async
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] raise self._format_exception(outputs) from None
(APIServer pid=1) ERROR 03-04 01:49:51 [async_llm.py:708] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(EngineCore_DP0 pid=564) Traceback (most recent call last):
(APIServer pid=1) INFO 03-04 01:49:51 [async_llm.py:605] Request chatcmpl-b61864175ca7091d failed (engine dead).
(EngineCore_DP0 pid=564) File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=564) self.run()
(EngineCore_DP0 pid=564) File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=564) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=564) File "/workspace/vllm/vllm/v1/engine/core.py", line 1104, in run_engine_core
(EngineCore_DP0 pid=564) raise e
(EngineCore_DP0 pid=564) File "/workspace/vllm/vllm/v1/engine/core.py", line 1093, in run_engine_core
(EngineCore_DP0 pid=564) engine_core.run_busy_loop()
(EngineCore_DP0 pid=564) File "/workspace/vllm/vllm/v1/engine/core.py", line 1128, in run_busy_loop
(EngineCore_DP0 pid=564) self._process_engine_step()
(EngineCore_DP0 pid=564) File "/workspace/vllm/vllm/v1/engine/core.py", line 1165, in _process_engine_step
(EngineCore_DP0 pid=564) outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=564) ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) File "/workspace/vllm/vllm/v1/engine/core.py", line 507, in step_with_batch_queue
(EngineCore_DP0 pid=564) engine_core_outputs = self.scheduler.update_from_output(
(EngineCore_DP0 pid=564) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) File "/workspace/vllm/vllm/v1/core/sched/scheduler.py", line 1316, in update_from_output
(EngineCore_DP0 pid=564) req_index = model_runner_output.req_id_to_index[req_id]
(EngineCore_DP0 pid=564) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
(EngineCore_DP0 pid=564) KeyError: 'chatcmpl-b61864175ca7091d-a87942ae'
(APIServer pid=1) INFO: 2409:40f4:40d6:c21e:acdc:1d47:3335:e212:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO: 2409:40f4:40d6:c21e:acdc:1d47:3335:e212:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO: 2409:40f4:40d6:c21e:acdc:1d47:3335:e212:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
(APIServer pid=1) INFO: Shutting down
(APIServer pid=1) INFO: Waiting for application shutdown.
(APIServer pid=1) INFO: Application shutdown complete.
(APIServer pid=1) INFO: Finished server process [1]

Before submitting a new issue...

• Make sure you already searched for relevant issues and checked the documentation page, which can answer lots of frequently asked questions.

[lk-chen]

"KeyError" usually means there's an un-caught exception. Could you try again with "--no-async-scheduler"

[Dineshkumar-Anandan-ZS0367]

sudo docker run -it --rm --net=host --privileged -v /dev:/dev -v $HOME/.cache/huggingface:/root/.cache/huggingface vllm/vllm-tpu:nightly-20260303-1f005dc-4034c3d vllm serve Qwen/Qwen3-VL-8B-Instruct --tensor-parallel-size 4 --dtype bfloat16 --max-model-len 22528 --max-num-seqs 16 --host 0.0.0.0 --port 7000 --trust-remote-code --enable-log-requests --max-num-batched-tokens 22528 --gpu-memory-utilization 0.5 --no-async-scheduler
INFO 03-04 09:52:23 [init.py:59] TPU info: node_name=v6e-test | tpu_type=v6e-4 | worker_id=0 | num_chips=4 | num_cores_per_chip=1
WARNING:absl:Tensorflow library not found, tensorflow.io.gfile operations will use native shim calls. GCS paths (i.e. 'gs://...') cannot be accessed.
INFO 03-04 09:52:26 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
INFO 03-04 09:52:26 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 03-04 09:52:27 [interface.py:226] Failed to import from vllm._C: ModuleNotFoundError("No module named 'vllm._C'")
INFO 03-04 09:52:27 [tpu_platform.py:101] Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e.
INFO 03-04 09:52:27 [tpu_platform.py:101] Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e.
INFO 03-04 09:52:27 [tpu_platform.py:101] Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e.
INFO 03-04 09:52:27 [tpu_platform.py:101] Automatically using fp8_e5m2 for FP8 KV cache on TPU v6e.
WARNING 03-04 09:52:29 [init.py:80] The quantization method 'awq' already exists and will be overwritten by the quantization config <class 'tpu_inference.layers.vllm.quantization.awq.VllmAWQConfig'>.
WARNING 03-04 09:52:29 [init.py:80] The quantization method 'compressed-tensors' already exists and will be overwritten by the quantization config <class 'tpu_inference.layers.vllm.quantization.compressed_tensors.compressed_tensors.VllmCompressedTensorsConfig'>.
WARNING 03-04 09:52:29 [init.py:80] The quantization method 'fp8' already exists and will be overwritten by the quantization config <class 'tpu_inference.layers.vllm.quantization.fp8.VllmFp8Config'>.
WARNING 03-04 09:52:29 [init.py:80] The quantization method 'mxfp4' already exists and will be overwritten by the quantization config <class 'tpu_inference.layers.vllm.quantization.mxfp4.VllmMxfp4Config'>.
usage: vllm [-h] [-v] {chat,complete,serve,bench,collect-env,run-batch} ...
vllm: error: unrecognized arguments: --no-async-scheduler

I don't think this could be the solution (vllm: error: unrecognized arguments: --no-async-scheduler)

[lk-chen]

Sorry, should be "--no-async-scheduling" https://docs.vllm.ai/en/stable/configuration/engine_args/#-async-scheduling-no-async-scheduling

[Dineshkumar-Anandan-ZS0367]

@lk-chen I have noticed that and thanks a lot

[Dineshkumar-Anandan-ZS0367]

@lk-chen Facing this Nonetype and scheduler dumped issue in Qwen3-VL

(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.16.1rc1.dev140+g4034c3d32) with config: model='Qwen/Qwen3-VL-8B-Instruct', speculative_config=None, token
izer='Qwen/Qwen3-VL-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=22528, download_dir=None, load_
format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=aut
o, device_config=None, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_
parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_ca
che_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_nam
e=Qwen/Qwen3-VL-8B-Instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.DYNAMO_TRACE_ONCE: 2>, 'debug_dump_path': Non
e, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'openxla', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': None, 'compile_ranges_split_points': [22
528], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_
of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant
': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'ba
cked'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []},
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-816a099b2aab3cb4-963ced5d,prompt_
token_ids_len=8189,prefill_token_ids_len=None,mm_features=[MultiModalFeatureSpec(data={'image_grid_thw': MultiModalFieldElem(data=tensor([ 1, 214, 152]), field=MultiModalBatchedField(keep_on_cpu=True)), 'pixel_
values': MultiModalFieldElem(data=tensor([[0.9922, 0.9922, 0.9922, ..., 0.9922, 0.9922, 0.9922],
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] [0.9922, 0.9922, 0.9922, ..., 0.9922, 0.9922, 0.9922],
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] [0.9922, 0.9922, 0.9922, ..., 0.9922, 0.9922, 0.9922],
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] ...,
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] [0.9922, 0.9922, 0.9922, ..., 0.9922, 0.9922, 0.9922],
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] [0.9922, 0.9922, 0.9922, ..., 0.9922, 0.9922, 0.9922],
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] [0.9922, 0.9922, 0.9922, ..., 0.9922, 0.9922, 0.9922]],
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:79] dtype=torch.bfloat16), field=MultiModalFlatField(keep_on_cpu=False, slices=[[slice(0, 32528, None)]], dim=0))}, modality='image', identifie
r='d7909c869ed476b7895707147c628979f4885fba336b569180443328a1667380', mm_position=PlaceholderRange(offset=51, length=8132, is_embed=None), mm_hash='d7909c869ed476b7895707147c628979f4885fba336b569180443328a166738
0')],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.0, top_p=1.0, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_wor
ds=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=2304, min_tokens=0, logprobs=0, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None,
extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48,
49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 1
01, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 14
3, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185
, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227,
228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269,
270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 3
12, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 35
4, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396
, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438,
439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480,
481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512],),num_computed_tokens=0,lora_request=None,prompt_em
beds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_schedule
d_tokens={chatcmpl-816a099b2aab3cb4-963ced5d: 8189}, total_num_scheduled_tokens=8189, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={chatcmpl-816a099b2aab3cb4-963ced5d: [0]}, num_common_prefix_blocks
=[512], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=nul
l, ec_connector_metadata=null)
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.01182775827019033
6, encoder_cache_usage=0.36097301136363635, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=8189, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache
_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] Traceback (most recent call last):
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 1093, in run_engine_core
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] engine_core.run_busy_loop()
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 1128, in run_busy_loop
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] self._process_engine_step()
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 1165, in _process_engine_step
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/engine/core.py", line 397, in step
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] model_output = future.result()
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] return self.__get_result()
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] raise self._exception
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/executor/uniproc_executor.py", line 80, in collective_rpc
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] return func(*args, **kwargs)
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/vllm/vllm/v1/worker/worker_base.py", line 365, in execute_model
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] return self.worker.execute_model(scheduler_output)
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/tpu_inference/tpu_inference/worker/tpu_worker.py", line 350, in execute_model
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/tpu_inference/tpu_inference/utils.py", line 360, in wrapper
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] result = func(*args, **kwargs)
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/tpu_inference/tpu_inference/runner/tpu_runner.py", line 609, in execute_model
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] output = self._execute_model(scheduler_output,
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/tpu_inference/tpu_inference/runner/tpu_runner.py", line 738, in _execute_model
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] self.persistent_batch_manager.update_states(
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] File "/workspace/tpu_inference/tpu_inference/runner/persistent_batch_manager.py", line 180, in update_states
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] req_id].mrope_position_delta = get_mrope_input_positions_fn(
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=564) ERROR 03-05 15:45:44 [core.py:1102] TypeError: 'NoneType' object is not callable
(EngineCore_DP0 pid=564) Process EngineCore_DP0:
(EngineCore_DP0 pid=564) Traceback (most recent call last):
(APIServer pid=1) ERROR 03-05 15:45:44 [async_llm.py:708] AsyncLLM output_handler failed.

[lk-chen]

I think @muskansh-google is working on muskansh-google@17f4d6c#diff-1aa20442c7dfbac299386b46a7ce3009199c85f891708f66e365df5d6b7ffcb1

[Dineshkumar-Anandan-ZS0367]

@lk-chen Thanks a lot, when can i expect the PR. Did you know?