Skip to content

[Bug]: Wan2.2 TI2V USP=2 failed #1219

New issue
New issue
Open
#1233
Open
[Bug]: Wan2.2 TI2V USP=2 failed#1219
#1233
Assignees
mxuax
Labels
bugSomething isn't working
@gcanlin

Description

@gcanlin
gcanlin
opened on Feb 5, 2026

Your current environment

The output of python collect_env.py 
Your output of `python collect_env.py` here

Your code version

vllm 
0.15.0
vllm-omni 
latest commit in the main

🐛 Describe the bug

python image_to_video.py   --model Wan-AI/Wan2.2-TI2V-5B-Diffusers   --image ~/vllm-omni-workspace/vllm-omni/test1.png   --prompt "A cat playing with yarn, smooth motion"   --negative_prompt "<optional quality filter>"   --height 480   --width 832   --num_frames 48   --guidance_scale 4.0   --num_inference_steps 40   --flow_shift 12.0   --fps 16   --output i2v_output.mp4  --enforce_eager

And change config in image_to_video.py:

    parallel_config = DiffusionParallelConfig(
        cfg_parallel_size=args.cfg_parallel_size,
        ulysses_degree=2
    )
Details 
============================================================
Generation Configuration:
  Model: Wan-AI/Wan2.2-TI2V-5B-Diffusers
  Inference steps: 40
  Frames: 48
  Parallel configuration: cfg_parallel_size=1
  Video size: 832x480
============================================================

Adding requests:   0%|                                                                                                  | 0/1 [00:00<?, ?it/s[Stage-0] INFO 02-05 08:24:01 [diffusion_engine.py:75] Pre-processing completed in 0.0113 secondsspeed input: 0.00 unit/s, output: 0.00 unit/s]
[Stage-0] INFO 02-05 08:24:01 [manager.py:538] Deactivating all adapters: 0 layers
[Stage-0] WARNING 02-05 08:24:01 [kv_transfer_manager.py:356] No connector available for receiving KV cache
[Stage-0] INFO 02-05 08:24:01 [manager.py:538] Deactivating all adapters: 0 layers
[Stage-0] WARNING 02-05 08:24:01 [kv_transfer_manager.py:356] No connector available for receiving KV cache
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315] Error executing RPC: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315] Traceback (most recent call last):
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 310, in execute_rpc
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     result = func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 130, in generate
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self.execute_model(request, self.od_config)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 152, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self.model_runner.execute_model(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 175, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     output = self.pipeline.forward(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 613, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     noise_pred = self.predict_noise_maybe_with_cfg(
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/distributed/cfg_parallel.py", line 80, in predict_noise_maybe_with_cfg
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     positive_noise_pred = self.predict_noise(**positive_kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 665, in predict_noise
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return current_model(**kwargs)[0]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 688, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     hidden_states = block(hidden_states, encoder_hidden_states, timestep_proj, rotary_emb)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 135, in __call__
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return registry.dispatch(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 220, in dispatch
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return hook.new_forward(self.module, *args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 110, in new_forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     output = module._original_forward(*args, **kwargs)  # type: ignore[attr-defined]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 479, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     norm_hidden_states = (self.norm1(hidden_states.float()) * (1 + scale_msa) + shift_msa).type_as(hidden_states)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315] RuntimeError: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344] Error processing RPC: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344] Traceback (most recent call last):
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 340, in worker_busy_loop
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     result, should_reply = self.execute_rpc(msg)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 316, in execute_rpc
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     raise e
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 310, in execute_rpc
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     result = func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 130, in generate
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self.execute_model(request, self.od_config)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 152, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self.model_runner.execute_model(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 175, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     output = self.pipeline.forward(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 613, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     noise_pred = self.predict_noise_maybe_with_cfg(
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/distributed/cfg_parallel.py", line 80, in predict_noise_maybe_with_cfg
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     positive_noise_pred = self.predict_noise(**positive_kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 665, in predict_noise
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return current_model(**kwargs)[0]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 688, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     hidden_states = block(hidden_states, encoder_hidden_states, timestep_proj, rotary_emb)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 135, in __call__
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return registry.dispatch(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 220, in dispatch
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return hook.new_forward(self.module, *args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 110, in new_forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     output = module._original_forward(*args, **kwargs)  # type: ignore[attr-defined]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 479, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     norm_hidden_states = (self.norm1(hidden_states.float()) * (1 + scale_msa) + shift_msa).type_as(hidden_states)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344] RuntimeError: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315] Error executing RPC: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315] Traceback (most recent call last):
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 310, in execute_rpc
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     result = func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 130, in generate
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self.execute_model(request, self.od_config)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 152, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self.model_runner.execute_model(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 175, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     output = self.pipeline.forward(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 613, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     noise_pred = self.predict_noise_maybe_with_cfg(
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/distributed/cfg_parallel.py", line 80, in predict_noise_maybe_with_cfg
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     positive_noise_pred = self.predict_noise(**positive_kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 665, in predict_noise
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return current_model(**kwargs)[0]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 688, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     hidden_states = block(hidden_states, encoder_hidden_states, timestep_proj, rotary_emb)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 135, in __call__
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return registry.dispatch(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 220, in dispatch
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     return hook.new_forward(self.module, *args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 110, in new_forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     output = module._original_forward(*args, **kwargs)  # type: ignore[attr-defined]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 479, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]     norm_hidden_states = (self.norm1(hidden_states.float()) * (1 + scale_msa) + shift_msa).type_as(hidden_states)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315]                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:315] RuntimeError: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344] Error processing RPC: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344] Traceback (most recent call last):
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 340, in worker_busy_loop
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     result, should_reply = self.execute_rpc(msg)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 316, in execute_rpc
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     raise e
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 310, in execute_rpc
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     result = func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]              ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 130, in generate
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self.execute_model(request, self.od_config)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 152, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self.model_runner.execute_model(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return func(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 175, in execute_model
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     output = self.pipeline.forward(req)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 613, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     noise_pred = self.predict_noise_maybe_with_cfg(
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/distributed/cfg_parallel.py", line 80, in predict_noise_maybe_with_cfg
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     positive_noise_pred = self.predict_noise(**positive_kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2.py", line 665, in predict_noise
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return current_model(**kwargs)[0]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 688, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     hidden_states = block(hidden_states, encoder_hidden_states, timestep_proj, rotary_emb)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return self._call_impl(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return forward_call(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 135, in __call__
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return registry.dispatch(*args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 220, in dispatch
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     return hook.new_forward(self.module, *args, **kwargs)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/hooks/base.py", line 110, in new_forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     output = module._original_forward(*args, **kwargs)  # type: ignore[attr-defined]
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py", line 479, in forward
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]     norm_hidden_states = (self.norm1(hidden_states.float()) * (1 + scale_msa) + shift_msa).type_as(hidden_states)
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344]                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
[Stage-0] ERROR 02-05 08:24:01 [diffusion_worker.py:344] RuntimeError: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999] Failed on batch ['0_f0ae7d95-5ea8-456d-a13c-79ca1b15afad']: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999] Traceback (most recent call last):
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/entrypoints/omni_stage.py", line 907, in _stage_worker
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]     diffusion_results = stage_engine.generate(
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]                         ^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/entrypoints/omni_diffusion.py", line 104, in generate
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]     return self._run_engine(request)
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]            ^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/entrypoints/omni_diffusion.py", line 107, in _run_engine
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]     return self.engine.step(request)
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]            ^^^^^^^^^^^^^^^^^^^^^^^^^
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]   File "/home/guocanlin/vllm-omni-workspace/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 79, in step
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999]     raise Exception(f"{output.error}")
[Stage-0] ERROR 02-05 08:24:01 [omni_stage.py:999] Exception: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1
ERROR 02-05 08:24:01 [omni.py:728] [Orchestrator] Stage 0 error on request 0_f0ae7d95-5ea8-456d-a13c-79ca1b15afad: The size of tensor a (2535) must match the size of tensor b (5070) at non-singleton dimension 1

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions