请问有Ascend npu的推理文档吗

[OrangePure]

readme好像都是关于gpu的

[SHYuanBest]

具体可以参考（这两环境都能跑通）：
https://docs.sglang.io/platforms/ascend/ascend_npu.html
https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/npu/

[OrangePure]

具体可以参考（这两环境都能跑通）： https://docs.sglang.io/platforms/ascend/ascend_npu.html https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/npu/

我使用了vllm-omni v0.18.0，跑Helios-Distilled的T2V任务，会报oom，请问是版本不对吗
vllm 0.18.0
vllm_ascend 0.18.0rc1
vllm-omni 0.18.0+npu

启动命令

python end2end.py \
    --model BestWishYsh/Helios-Distilled \
    --sample-type t2v \
    --prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
    --num-frames 240 \
    --guidance-scale 1.0 \
    --is-enable-stage2 \
    --pyramid-num-inference-steps-list 2 2 2 \
    --is-amplify-first-chunk \
    --output helios_t2v_distilled.mp4

报错信息

ERROR 04-09 21:56:46 [diffusion_worker.py:744] Traceback (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 740, in execute_method
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return func(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 184, in generate
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self.execute_model(request, self.od_config)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 223, in execute_model
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self.model_runner.execute_model(req)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 275, in execute_model
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     output = self.pipeline.forward(req)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/vllm-omni/vllm_omni/diffusion/models/helios/pipeline_helios.py", line 638, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     current_video = self.vae.decode(current_latents, return_dict=False)[0]
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return method(self, *args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 1230, in decode
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     decoded = self._decode(z).sample
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 1197, in _decode
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     out_ = self.decoder(x[:, :, i : i + 1, :, :], feat_cache=self._feat_map, feat_idx=self._conv_idx)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 891, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     x = up_block(x, feat_cache=feat_cache, feat_idx=feat_idx, first_chunk=first_chunk)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 769, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     x = resnet(x, feat_cache=feat_cache, feat_idx=feat_idx)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 353, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     x = self.conv1(x, feat_cache[idx])
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 173, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return super().forward(x)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 717, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return self._conv_forward(input, self.weight, self.bias)
ERROR 04-09 21:56:46 [diffusion_worker.py:744]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 712, in _conv_forward
ERROR 04-09 21:56:46 [diffusion_worker.py:744]     return F.conv3d(
ERROR 04-09 21:56:46 [diffusion_worker.py:744] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:46 [diffusion_worker.py:744] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:46 [diffusion_worker.py:744] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:46 [diffusion_worker.py:744] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:46 [diffusion_worker.py:744] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:46 [diffusion_worker.py:744]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:46 [diffusion_worker.py:744]         Solution: Close applications not in use.
ERROR 04-09 21:56:46 [diffusion_worker.py:744]         TraceBack (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:744]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:744]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:744]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:46 [diffusion_worker.py:744] 
ERROR 04-09 21:56:46 [diffusion_worker.py:452] Error executing RPC: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Solution: Close applications not in use.
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         TraceBack (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:46 [diffusion_worker.py:452] Traceback (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 449, in execute_rpc
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     result = self.worker.execute_method(method, *args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 745, in execute_method
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     raise e
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 740, in execute_method
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return func(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 184, in generate
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self.execute_model(request, self.od_config)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 223, in execute_model
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self.model_runner.execute_model(req)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 275, in execute_model
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     output = self.pipeline.forward(req)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/vllm-omni/vllm_omni/diffusion/models/helios/pipeline_helios.py", line 638, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     current_video = self.vae.decode(current_latents, return_dict=False)[0]
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return method(self, *args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 1230, in decode
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     decoded = self._decode(z).sample
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 1197, in _decode
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     out_ = self.decoder(x[:, :, i : i + 1, :, :], feat_cache=self._feat_map, feat_idx=self._conv_idx)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 891, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     x = up_block(x, feat_cache=feat_cache, feat_idx=feat_idx, first_chunk=first_chunk)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 769, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     x = resnet(x, feat_cache=feat_cache, feat_idx=feat_idx)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 353, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     x = self.conv1(x, feat_cache[idx])
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 173, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return super().forward(x)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 717, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return self._conv_forward(input, self.weight, self.bias)
ERROR 04-09 21:56:46 [diffusion_worker.py:452]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 712, in _conv_forward
ERROR 04-09 21:56:46 [diffusion_worker.py:452]     return F.conv3d(
ERROR 04-09 21:56:46 [diffusion_worker.py:452] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:46 [diffusion_worker.py:452] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Solution: Close applications not in use.
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         TraceBack (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:452]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:46 [diffusion_worker.py:452] 
ERROR 04-09 21:56:46 [diffusion_worker.py:481] Error processing RPC: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Solution: Close applications not in use.
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         TraceBack (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:46 [diffusion_worker.py:481] Traceback (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 477, in worker_busy_loop
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     result, should_reply = self.execute_rpc(msg)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 453, in execute_rpc
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     raise e
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 449, in execute_rpc
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     result = self.worker.execute_method(method, *args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 745, in execute_method
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     raise e
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 740, in execute_method
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return func(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 184, in generate
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self.execute_model(request, self.od_config)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py", line 223, in execute_model
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self.model_runner.execute_model(req)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 275, in execute_model
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     output = self.pipeline.forward(req)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/vllm-omni/vllm_omni/diffusion/models/helios/pipeline_helios.py", line 638, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     current_video = self.vae.decode(current_latents, return_dict=False)[0]
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return method(self, *args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 1230, in decode
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     decoded = self._decode(z).sample
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 1197, in _decode
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     out_ = self.decoder(x[:, :, i : i + 1, :, :], feat_cache=self._feat_map, feat_idx=self._conv_idx)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 891, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     x = up_block(x, feat_cache=feat_cache, feat_idx=feat_idx, first_chunk=first_chunk)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 769, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     x = resnet(x, feat_cache=feat_cache, feat_idx=feat_idx)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 353, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     x = self.conv1(x, feat_cache[idx])
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self._call_impl(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return forward_call(*args, **kwargs)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/diffusers/models/autoencoders/autoencoder_kl_wan.py", line 173, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return super().forward(x)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 717, in forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return self._conv_forward(input, self.weight, self.bias)
ERROR 04-09 21:56:46 [diffusion_worker.py:481]   File "/root/miniconda3/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 712, in _conv_forward
ERROR 04-09 21:56:46 [diffusion_worker.py:481]     return F.conv3d(
ERROR 04-09 21:56:46 [diffusion_worker.py:481] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:46 [diffusion_worker.py:481] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Solution: Close applications not in use.
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         TraceBack (most recent call last):
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_worker.py:481]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:46 [diffusion_worker.py:481] 
ERROR 04-09 21:56:46 [diffusion_engine.py:82] Dummy run failed: Dummy run failed: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:46 [diffusion_engine.py:82] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:46 [diffusion_engine.py:82] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:46 [diffusion_engine.py:82] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:46 [diffusion_engine.py:82] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:46 [diffusion_engine.py:82]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:46 [diffusion_engine.py:82]         Solution: Close applications not in use.
ERROR 04-09 21:56:46 [diffusion_engine.py:82]         TraceBack (most recent call last):
ERROR 04-09 21:56:46 [diffusion_engine.py:82]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_engine.py:82]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:46 [diffusion_engine.py:82]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:46 [diffusion_engine.py:82] 
INFO 04-09 21:56:46 [diffusion_worker.py:486] Worker 0: Received shutdown message
INFO 04-09 21:56:46 [diffusion_worker.py:507] event loop terminated.
INFO 04-09 21:56:47 [diffusion_worker.py:542] Worker 0: Shutdown complete.
ERROR 04-09 21:56:56 [async_omni_engine.py:542] [AsyncOmniEngine] Stage initialization failed; shutting down 0 initialized stage(s)
ERROR 04-09 21:56:56 [async_omni_engine.py:542] Traceback (most recent call last):
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 490, in _initialize_stages
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     stage_clients[stage_id] = initialize_diffusion_stage(
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/engine/stage_init_utils.py", line 460, in initialize_diffusion_stage
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     return StageDiffusionClient(model, od_config, metadata, batch_size=batch_size)
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/diffusion/stage_diffusion_client.py", line 50, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     self._engine = AsyncOmniDiffusion(model=model, od_config=od_config, batch_size=batch_size)
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/entrypoints/async_omni_diffusion.py", line 142, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     self.engine: DiffusionEngine = DiffusionEngine.make_engine(od_config)
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 276, in make_engine
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     return DiffusionEngine(config, scheduler=scheduler)
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 84, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     raise e
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 80, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     self._dummy_run()
ERROR 04-09 21:56:56 [async_omni_engine.py:542]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 382, in _dummy_run
ERROR 04-09 21:56:56 [async_omni_engine.py:542]     raise RuntimeError(f"Dummy run failed: {output.error}")
ERROR 04-09 21:56:56 [async_omni_engine.py:542] RuntimeError: Dummy run failed: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:56 [async_omni_engine.py:542] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:56 [async_omni_engine.py:542] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:56 [async_omni_engine.py:542] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:56 [async_omni_engine.py:542] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:56 [async_omni_engine.py:542]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:56 [async_omni_engine.py:542]         Solution: Close applications not in use.
ERROR 04-09 21:56:56 [async_omni_engine.py:542]         TraceBack (most recent call last):
ERROR 04-09 21:56:56 [async_omni_engine.py:542]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:56 [async_omni_engine.py:542]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:56 [async_omni_engine.py:542]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:56 [async_omni_engine.py:542] 
ERROR 04-09 21:56:56 [async_omni_engine.py:607] [AsyncOmniEngine] Orchestrator thread crashed
ERROR 04-09 21:56:56 [async_omni_engine.py:607] Traceback (most recent call last):
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 603, in _bootstrap_orchestrator
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     loop.run_until_complete(_run_orchestrator())
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     return future.result()
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 588, in _run_orchestrator
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     self._initialize_stages(stage_init_timeout)
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 490, in _initialize_stages
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     stage_clients[stage_id] = initialize_diffusion_stage(
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/engine/stage_init_utils.py", line 460, in initialize_diffusion_stage
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     return StageDiffusionClient(model, od_config, metadata, batch_size=batch_size)
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/diffusion/stage_diffusion_client.py", line 50, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     self._engine = AsyncOmniDiffusion(model=model, od_config=od_config, batch_size=batch_size)
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/entrypoints/async_omni_diffusion.py", line 142, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     self.engine: DiffusionEngine = DiffusionEngine.make_engine(od_config)
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 276, in make_engine
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     return DiffusionEngine(config, scheduler=scheduler)
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 84, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     raise e
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 80, in __init__
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     self._dummy_run()
ERROR 04-09 21:56:56 [async_omni_engine.py:607]   File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 382, in _dummy_run
ERROR 04-09 21:56:56 [async_omni_engine.py:607]     raise RuntimeError(f"Dummy run failed: {output.error}")
ERROR 04-09 21:56:56 [async_omni_engine.py:607] RuntimeError: Dummy run failed: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
ERROR 04-09 21:56:56 [async_omni_engine.py:607] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-09 21:56:56 [async_omni_engine.py:607] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-09 21:56:56 [async_omni_engine.py:607] [ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
ERROR 04-09 21:56:56 [async_omni_engine.py:607] [PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
ERROR 04-09 21:56:56 [async_omni_engine.py:607]         Possible Cause: Available memory is insufficient.
ERROR 04-09 21:56:56 [async_omni_engine.py:607]         Solution: Close applications not in use.
ERROR 04-09 21:56:56 [async_omni_engine.py:607]         TraceBack (most recent call last):
ERROR 04-09 21:56:56 [async_omni_engine.py:607]         Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:56 [async_omni_engine.py:607]         Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
ERROR 04-09 21:56:56 [async_omni_engine.py:607]         [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]
ERROR 04-09 21:56:56 [async_omni_engine.py:607] 
INFO 04-09 21:56:56 [async_omni_engine.py:1133] [AsyncOmniEngine] Shutting down Orchestrator
Exception in thread orchestrator:
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/root/miniconda3/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 603, in _bootstrap_orchestrator
    loop.run_until_complete(_run_orchestrator())
  File "/root/miniconda3/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 588, in _run_orchestrator
    self._initialize_stages(stage_init_timeout)
  File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 490, in _initialize_stages
    stage_clients[stage_id] = initialize_diffusion_stage(
  File "/root/vllm-omni/vllm_omni/engine/stage_init_utils.py", line 460, in initialize_diffusion_stage
    return StageDiffusionClient(model, od_config, metadata, batch_size=batch_size)
  File "/root/vllm-omni/vllm_omni/diffusion/stage_diffusion_client.py", line 50, in __init__
    self._engine = AsyncOmniDiffusion(model=model, od_config=od_config, batch_size=batch_size)
  File "/root/vllm-omni/vllm_omni/entrypoints/async_omni_diffusion.py", line 142, in __init__
    self.engine: DiffusionEngine = DiffusionEngine.make_engine(od_config)
  File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 276, in make_engine
    return DiffusionEngine(config, scheduler=scheduler)
  File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 84, in __init__
    raise e
  File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 80, in __init__
    self._dummy_run()
  File "/root/vllm-omni/vllm_omni/diffusion/diffusion_engine.py", line 382, in _dummy_run
    raise RuntimeError(f"Dummy run failed: {output.error}")
RuntimeError: Dummy run failed: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
        Possible Cause: Available memory is insufficient.
        Solution: Close applications not in use.
        TraceBack (most recent call last):
        Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
        Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
        [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]

Traceback (most recent call last):
  File "/root/vllm-omni/examples/offline_inference/helios/end2end.py", line 348, in <module>
    main()
  File "/root/vllm-omni/examples/offline_inference/helios/end2end.py", line 207, in main
    omni = Omni(
  File "/root/vllm-omni/vllm_omni/entrypoints/omni_base.py", line 95, in __init__
    self.engine = AsyncOmniEngine(
  File "/root/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 266, in __init__
    startup_future.result(timeout=startup_timeout)
  File "/root/miniconda3/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/root/miniconda3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
RuntimeError: Orchestrator initialization failed: Dummy run failed: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Conv2D.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[ERROR] 2026-04-09-21:56:46 (PID:210907, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[PID: 210907] 2026-04-09-21:56:40.384.694 Memory_Allocation_Failure(EL0004): Failed to allocate memory requested by GE module.
        Possible Cause: Available memory is insufficient.
        Solution: Close applications not in use.
        TraceBack (most recent call last):
        Call rtMalloc fail, purpose: page caching, type = 2, size:3221291008, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
        Call rtMalloc fail, purpose: page caching, type = 2, size:1610678272, device_id:0[FUNC:Alloc][FILE:device_allocator.cc][LINE:72]
        [Exec][Op]Execute op failed. op type = Conv2D, ge result = 4294967295[FUNC:ReportCallError][FILE:acl_op_executor_log_inner.cpp][LINE:124]

[SHYuanBest]

你用的ascend哪款NPU？貌似他们环境更新了... 可能要在他们仓库提个issue。

[OrangePure]

你用的ascend哪款NPU？貌似他们环境更新了... 可能要在他们仓库提个issue。

用的910B 64G的，之前是0.16 or 0.17可以运行吗，我可以切到旧版试试

[SHYuanBest]

readme好像都是关于gpu的

实际步骤也比较简单：

1. 先安装https://github.com/PKU-YuanGroup/Helios/blob/main/requirements.txt （安装前把torch、torchvision、torchaudio、triton那几行注释掉）
2. 然后再安装 https://github.com/PKU-YuanGroup/Helios/blob/main/requirements_npu.txt

[OrangePure]

readme好像都是关于gpu的

实际步骤也比较简单：

1. 先安装https://github.com/PKU-YuanGroup/Helios/blob/main/requirements.txt （安装前把torch、torchvision、torchaudio、triton那几行注释掉）
2. 然后再安装 https://github.com/PKU-YuanGroup/Helios/blob/main/requirements_npu.txt

嗯嗯，这个仓库和sglang我都跑起来了，但是在npu上存在相同的问题，是否有解决方法：
1、t2v任务用832*480分辨率的时候会报错；

报错

``` [04-10 02:04:07] [HeliosChunkedDenoisingStage] Error during execution after 60539.9838 ms: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 Traceback (most recent call last): File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/stages/base.py", line 203, in __call__ result = self.forward(batch, server_args) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/helios_denoising.py", line 649, in forward latents, global_step_offset = self._denoise_one_chunk_stage2( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/helios_denoising.py", line 431, in _denoise_one_chunk_stage2 latents = self.scheduler.step( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_helios.py", line 717, in step return self.step_dmd( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_helios.py", line 661, in step_dmd pred_image_or_video = self.convert_flow_pred_to_x0( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_helios.py", line 645, in convert_flow_pred_to_x0 x0_pred = xt - sigma_t * flow_pred RuntimeError: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 [04-10 02:04:07] Error executing request c0192c86-5054-4867-91c4-d24a31a54eb2: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 Traceback (most recent call last): File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/managers/gpu_worker.py", line 230, in execute_forward result = self.pipeline.forward(req, self.server_args) File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py", line 622, in forward return self.executor.execute_with_profiling(self.stages, batch, server_args) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/executors/pipeline_executor.py", line 57, in execute_with_profiling batch = self.execute(stages, batch, server_args) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py", line 101, in execute batch = self._execute(stages, batch, server_args) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/executors/parallel_executor.py", line 92, in _execute batch = stage(batch, server_args) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/stages/base.py", line 203, in __call__ result = self.forward(batch, server_args) File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/helios_denoising.py", line 649, in forward latents, global_step_offset = self._denoise_one_chunk_stage2( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/helios_denoising.py", line 431, in _denoise_one_chunk_stage2 latents = self.scheduler.step( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_helios.py", line 717, in step return self.step_dmd( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_helios.py", line 661, in step_dmd pred_image_or_video = self.convert_flow_pred_to_x0( File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/models/schedulers/scheduling_helios.py", line 645, in convert_flow_pred_to_x0 x0_pred = xt - sigma_t * flow_pred RuntimeError: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 [04-10 02:04:07] Failed to generate output for prompt 1: Error executing request c0192c86-5054-4867-91c4-d24a31a54eb2: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 Traceback (most recent call last): File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/utils/logging_utils.py", line 626, in log_generation_timer yield timer File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/entrypoints/diffusion_generator.py", line 235, in generate raise Exception(f"{output_batch.error}") Exception: Error executing request c0192c86-5054-4867-91c4-d24a31a54eb2: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 [04-10 02:04:07] Generation failed for prompt 1/1: Error executing request c0192c86-5054-4867-91c4-d24a31a54eb2: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 Traceback (most recent call last): File "/root/autodl-tmp/sglang/python/sglang/multimodal_gen/runtime/entrypoints/diffusion_generator.py", line 235, in generate raise Exception(f"{output_batch.error}") Exception: Error executing request c0192c86-5054-4867-91c4-d24a31a54eb2: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3 [04-10 02:04:07] Completed batch processing. Generated 0 outputs in 64.25 seconds ```

2、使用竖屏640384分辨率，在t2v i2v任务下，每1s左右会抽帧、形变；384640没有问题

[SHYuanBest]

为了快速验证我们提出的方法的有效性，我们将训练数据的分辨率限制在 384 x 640。由于数据集规模有限（约 80 万个视频片段），我们使用同一批数据进行了多轮训练。遗憾的是，这削弱了预训练基础模型外推到其他分辨率的能力。但我们的dataloader原生支持多种分辨率和可变视频时长。如果您拥有充足的计算资源，我们强烈建议您使用更多样化的数据集来复现或微调 Helios 模型。

具体可以查看：#47