ValueError: Checkpoint model type 'deepseek_v32' not recognized by Transformers.

[chensuyue]

Model quantization with AutoRound failed.

Impact Model list

deepseek-ai/DeepSeek-Math-V2
deepseek-ai/DeepSeek-V3.2
deepseek-ai/DeepSeek-V3.2-Speciale

Issue details

Traceback (most recent call last):
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1325, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1028, in __getitem__
    raise KeyError(key)
KeyError: 'deepseek_v32'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/uttest/miniforge3/envs/autoround_test/bin/auto-round", line 10, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/auto_round/__main__.py", line 869, in run
    tune(args)
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/auto_round/__main__.py", line 600, in tune
    autoround: BaseCompressor = AutoRound(
                                ^^^^^^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/auto_round/autoround.py", line 177, in __new__
    ar = dynamic_compressor(
         ^^^^^^^^^^^^^^^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/auto_round/compressors/base.py", line 230, in __init__
    model, tokenizer = llm_load_model(
                       ^^^^^^^^^^^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/auto_round/utils/model.py", line 291, in llm_load_model
    model = model_cls.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 318, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/uttest/miniforge3/envs/autoround_test/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1327, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `deepseek_v32` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`

[yiliu30]

• Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound failed on vllm 0.13 -> fixed in vllm 0.14rc2

INFO 01-15 02:16:46 [vllm.py:722] Cudagraph is disabled under eager mode
INFO 01-15 02:16:46 [core.py:93] Initializing a V1 LLM engine (v0.13.1.dev0+g72506c983.d20260114) with config: model='/storage/yiliu7/Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound/', speculative_config=None, tokenizer='/storage/yiliu7/Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=auto-round, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=/storage/yiliu7/Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound/, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
WARNING 01-15 02:16:46 [multiproc_executor.py:882] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 01-15 02:16:56 [parallel_state.py:1203] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:60123 backend=nccl
INFO 01-15 02:16:56 [parallel_state.py:1203] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:60123 backend=nccl
INFO 01-15 02:16:57 [pynccl.py:111] vLLM is using nccl==2.27.5
INFO 01-15 02:16:58 [parallel_state.py:1411] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
INFO 01-15 02:16:58 [parallel_state.py:1411] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
(Worker_TP0 pid=207489) INFO 01-15 02:16:59 [gpu_model_runner.py:3562] Starting to load model /storage/yiliu7/Intel/Qwen3-30B-A3B-Instruct-2507-int4-AutoRound/...
(Worker_TP1 pid=207490) INFO 01-15 02:16:59 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(Worker_TP0 pid=207489) INFO 01-15 02:16:59 [gptq_marlin.py:376] Using MarlinLinearKernel for GPTQMarlinLinearMethod
(Worker_TP0 pid=207489) INFO 01-15 02:16:59 [cuda.py:351] Using FLASHINFER attention backend out of potential backends: ('FLASHINFER', 'FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION')
(Worker_TP1 pid=207490) INFO 01-15 02:16:59 [selector.py:112] Using HND KV cache layout for FLASHINFER backend.
(Worker_TP0 pid=207489) INFO 01-15 02:16:59 [selector.py:112] Using HND KV cache layout for FLASHINFER backend.
(Worker_TP0 pid=207489) INFO 01-15 02:16:59 [layer.py:372] Enabled separate cuda stream for MoE shared_experts
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:02<00:06,  2.02s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:04<00:04,  2.05s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:06<00:01,  2.00s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:06<00:00,  1.45s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:06<00:00,  1.66s/it]
(Worker_TP0 pid=207489) 
(Worker_TP0 pid=207489) INFO 01-15 02:17:06 [default_loader.py:308] Loading weights took 6.67 seconds
(Worker_TP0 pid=207489) INFO 01-15 02:17:07 [gpu_model_runner.py:3659] Model loading took 7.8882 GiB memory and 7.860251 seconds
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824] WorkerProc hit an exception.
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824] Traceback (most recent call last):
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 819, in worker_busy_loop
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     output = func(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return func(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/worker/gpu_worker.py", line 340, in determine_available_memory
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     self.model_runner.profile_run()
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 4474, in profile_run
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                                         ^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return func(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     outputs = self.model(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]               ^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 734, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states = self.model(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                     ^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/compilation/decorators.py", line 372, in __call__
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self.forward(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 456, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states, residual = layer(positions, hidden_states, residual)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 387, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states = self.mlp(hidden_states)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                     ^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 199, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     final_hidden_states = self.experts(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                           ^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/custom_op.py", line 47, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._forward_method(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1765, in forward_cuda
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self.forward_native(hidden_states, router_logits)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1727, in forward_native
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     fused_output = torch.ops.vllm.moe_forward(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 2082, in moe_forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self.forward_impl(hidden_states, router_logits)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1960, in forward_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     final_hidden_states = self.quant_method.apply(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                           ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/quantization/gptq_marlin.py", line 908, in apply
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return fused_marlin_moe(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/fused_marlin_moe.py", line 327, in fused_marlin_moe
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     moe_output = _fused_marlin_moe(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                  ^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/fused_marlin_moe.py", line 121, in _fused_marlin_moe
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     intermediate_cache1 = ops.moe_wna16_marlin_gemm(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/_custom_ops.py", line 2192, in moe_wna16_marlin_gemm
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return torch.ops._moe_C.moe_wna16_marlin_gemm(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824] RuntimeError: _moe_C::moe_wna16_marlin_gemm() expected at most 29 argument(s) but received 30 argument(s). Declaration: _moe_C::moe_wna16_marlin_gemm(Tensor($0! -> ) a, Tensor? c_or_none, Tensor($1! -> ) b_q_weight, Tensor? b_bias_or_none, Tensor($2! -> ) b_scales, Tensor? a_scales, Tensor? global_scale, Tensor? b_zeros_or_none, Tensor? g_idx_or_none, Tensor? perm_or_none, Tensor($3! -> ) workspace, Tensor sorted_token_ids, Tensor($4! -> ) expert_ids, Tensor($5! -> ) num_tokens_past_padded, Tensor($6! -> ) topk_weights, int moe_block_size, int top_k, bool mul_topk_weights, int b_type_id, int size_m, int size_n, int size_k, bool is_full_k, bool use_atomic_add, bool use_fp32_reduce, bool is_zp_float, int thread_k, int thread_n, int blocks_per_sm) -> Tensor
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824] Traceback (most recent call last):
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 819, in worker_busy_loop
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     output = func(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]              ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return func(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/worker/gpu_worker.py", line 340, in determine_available_memory
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     self.model_runner.profile_run()
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 4474, in profile_run
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                                         ^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return func(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/v1/worker/gpu_model_runner.py", line 4198, in _dummy_run
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     outputs = self.model(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]               ^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 734, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states = self.model(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                     ^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/compilation/decorators.py", line 372, in __call__
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self.forward(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 456, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states, residual = layer(positions, hidden_states, residual)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 387, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     hidden_states = self.mlp(hidden_states)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                     ^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/models/qwen3_moe.py", line 199, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     final_hidden_states = self.experts(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                           ^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._call_impl(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return forward_call(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/custom_op.py", line 47, in forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._forward_method(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1765, in forward_cuda
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self.forward_native(hidden_states, router_logits)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1727, in forward_native
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     fused_output = torch.ops.vllm.moe_forward(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 2082, in moe_forward
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self.forward_impl(hidden_states, router_logits)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/layer.py", line 1960, in forward_impl
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     final_hidden_states = self.quant_method.apply(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                           ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/quantization/gptq_marlin.py", line 908, in apply
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return fused_marlin_moe(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/fused_marlin_moe.py", line 327, in fused_marlin_moe
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     moe_output = _fused_marlin_moe(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                  ^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/model_executor/layers/fused_moe/fused_marlin_moe.py", line 121, in _fused_marlin_moe
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     intermediate_cache1 = ops.moe_wna16_marlin_gemm(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/vllm/vllm/_custom_ops.py", line 2192, in moe_wna16_marlin_gemm
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return torch.ops._moe_C.moe_wna16_marlin_gemm(
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]   File "/home/yiliu7/workspace/ds32/lib/python3.12/site-packages/torch/_ops.py", line 1255, in __call__
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]     return self._op(*args, **kwargs)
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824] RuntimeError: _moe_C::moe_wna16_marlin_gemm() expected at most 29 argument(s) but received 30 argument(s). Declaration: _moe_C::moe_wna16_marlin_gemm(Tensor($0! -> ) a, Tensor? c_or_none, Tensor($1! -> ) b_q_weight, Tensor? b_bias_or_none, Tensor($2! -> ) b_scales, Tensor? a_scales, Tensor? global_scale, Tensor? b_zeros_or_none, Tensor? g_idx_or_none, Tensor? perm_or_none, Tensor($3! -> ) workspace, Tensor sorted_token_ids, Tensor($4! -> ) expert_ids, Tensor($5! -> ) num_tokens_past_padded, Tensor($6! -> ) topk_weights, int moe_block_size, int top_k, bool mul_topk_weights, int b_type_id, int size_m, int size_n, int size_k, bool is_full_k, bool use_atomic_add, bool use_fp32_reduce, bool is_zp_float, int thread_k, int thread_n, int blocks_per_sm) -> Tensor
(Worker_TP0 pid=207489) ERROR 01-15 02:17:16 [multiproc_executor.py:824] 
Traceback (most recent call last):
  File "/home/yiliu7/workspace/inc/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/deepseek/generate.py", line 67, in <module>
    main(args)
  File "/home/yiliu7/workspace/inc/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/auto_round/deepseek/generate.py", line 33, in main
    llm = LLM(**args)
          ^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/entrypoints/llm.py", line 351, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/engine/llm_engine.py", line 183, in from_engine_args
    return cls(
           ^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/engine/llm_engine.py", line 109, in __init__
    self.engine_core = EngineCoreClient.make_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/engine/core_client.py", line 95, in make_client
    return InprocClient(vllm_config, executor_class, log_stats)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/engine/core_client.py", line 268, in __init__
    self.engine_core = EngineCore(*args, **kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/engine/core.py", line 109, in __init__
    num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/engine/core.py", line 240, in _initialize_kv_caches
    available_gpu_memory = self.model_executor.determine_available_memory()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
    return self.collective_rpc("determine_available_memory")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 359, in collective_rpc
    return aggregate(get_response())
                     ^^^^^^^^^^^^^^
  File "/home/yiliu7/workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 342, in get_response
    raise RuntimeError(
RuntimeError: Worker failed with error '_moe_C::moe_wna16_marlin_gemm() expected at most 29 argument(s) but received 30 argument(s). Declaration: _moe_C::moe_wna16_marlin_gemm(Tensor($0! -> ) a, Tensor? c_or_none, Tensor($1! -> ) b_q_weight, Tensor? b_bias_or_none, Tensor($2! -> ) b_scales, Tensor? a_scales, Tensor? global_scale, Tensor? b_zeros_or_none, Tensor? g_idx_or_none, Tensor? perm_or_none, Tensor($3! -> ) workspace, Tensor sorted_token_ids, Tensor($4! -> ) expert_ids, Tensor($5! -> ) num_tokens_past_padded, Tensor($6! -> ) topk_weights, int moe_block_size, int top_k, bool mul_topk_weights, int b_type_id, int size_m, int size_n, int size_k, bool is_full_k, bool use_atomic_add, bool use_fp32_reduce, bool is_zp_float, int thread_k, int thread_n, int blocks_per_sm) -> Tensor', please check the stack trace above for the root cause
(Worker_TP0 pid=207489) INFO 01-15 02:17:16 [multiproc_executor.py:709] Parent process exited, terminating worker
(Worker_TP1 pid=207490) INFO 01-15 02:17:16 [multiproc_executor.py:709] Parent process exited, terminating worker
(ds32) root@42648c2c9ded:/home/yiliu7/work

diff --git a/vllm/_custom_ops.py b/vllm/_custom_ops.py
index 2319655008..bce486e429 100644
--- a/vllm/_custom_ops.py
+++ b/vllm/_custom_ops.py
@@ -2176,7 +2176,6 @@ def moe_wna16_marlin_gemm(
     moe_block_size: int,
     top_k: int,
     mul_topk_weights: bool,
-    is_ep: bool,
     b_q_type: ScalarType,
     size_m: int,
     size_n: int,
@@ -2208,7 +2207,6 @@ def moe_wna16_marlin_gemm(
         moe_block_size,
         top_k,
         mul_topk_weights,
-        is_ep,
         b_q_type.id,
         size_m,
         size_n,
@@ -2270,7 +2268,6 @@ if hasattr(torch.ops, "
_moe_C") and hasattr(torch.ops._moe_C, "marli
n_gemm_moe")
         moe_block_size: int,
         top_k: int,
         mul_topk_weights: bool,
-        is_ep: bool,
         b_q_type: ScalarType,
         size_m: int,
         size_n: int,

[yiliu30]

@chensuyue @XuehaoSun Please follow this BKC.
#1291

[yiliu30]

FYI: Here is the result of iters=0.

local-completions (model=/storage/yiliu7/DeepSeek-V3.2-fp8-w4a16/,base_url=http://0.0.0.0:8000/v1/completions,max_length=8192,tokenized_requests=False,tokenizer_backend=None,num_concurrent=32), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9507|±  |0.0060|
|     |       |strict-match    |     5|exact_match|↑  |0.9515|±  |0.0059|