Bugs in Image-to-Video

[holgerpieta]

Hello Experts

I'm trying to run CogVideoX Image-to-Video, but I seem to be doing something wrong.
Here is what I do:

• Start SD.Next
• Select any base model (not important for the problem)
• Select "Image" Tab
• Input some prompt (again not important)
• Drag and drop an image file onto the left box
• Open "Resize": "Fixed", "Nearest", "Fixed", 512x512
• Script "Video CogVideoX"
• Model: "THUDM/CogVideoX-5b-I2V"
• Generate

The result in an error message: CogVideoX: model=THUDM/CogVideoX-5b-I2V image input not supported
I've traced the message to line 187 in file scripts/cogvideo.py
Looking at that I think the problem is that the run function is called with an empty image argument.

So I presume I did something wrong, clicked the wrong buttons or the didn't click the right buttons, causing the image input to be empty.

Any hints what I did wrong?

[vladmandic]

this was a bug, now fixed in dev branch.

[holgerpieta]

Thanks. Bugs seems fixed.
But the format string in line 183 doesn't work. The inner ' must be ", otherwise it won't work.

[vladmandic]

ah, silly me, i'll fix it.

[holgerpieta]

I've got it to run, but it seems there is something wrong:

Using CogVideoX from text input works somewhat better, at least without those block artifacts. So I'd say there is nothing fundamentally wrong with my setup.
But I'm not expert on that, so I don't even know where to look.

[vladmandic]

i got that as well, yeah, something is wrong, but will take a while to dig in - this is likely issue within diffusers.
i'll convert this into an issue so it doesn't fall off the radar.

[holgerpieta]

I've renamed the threat because I'm playing around with image to video and keep getting errors.
This one is the result of trying SVD:

20:14:10-380615 ERROR    Processing: RuntimeError
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ D:\SD.Next\automatic\modules\processing_diffusers.py:99 in process_base                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                      │
│    98 │   │   else:                                                                                                                                                                                                                                                                  │
│ ❱  99 │   │   │   output = shared.sd_model(**base_args)                                                                                                                                                                                                                              │
│   100 │   │   if isinstance(output, dict):                                                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\utils\_contextlib.py:115 in decorate_context                                                                                                                                                                                       │
│                                                                                                                                                                                                                                                                                      │
│   114 │   │   with ctx_factory():                                                                                                                                                                                                                                                    │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                                                                                                                                                                                                       │
│   116                                                                                                                                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\diffusers\pipelines\stable_video_diffusion\pipeline_stable_video_diffusion.py:510 in __call__                                                                                                                                            │
│                                                                                                                                                                                                                                                                                      │
│   509 │   │                                                                                                                                                                                                                                                                          │
│ ❱ 510 │   │   image_latents = self._encode_vae_image(                                                                                                                                                                                                                                │
│   511 │   │   │   image,                                                                                                                                                                                                                                                             │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\diffusers\pipelines\stable_video_diffusion\pipeline_stable_video_diffusion.py:239 in _encode_vae_image                                                                                                                                   │
│                                                                                                                                                                                                                                                                                      │
│   238 │   │   image = image.to(device=device)                                                                                                                                                                                                                                        │
│ ❱ 239 │   │   image_latents = self.vae.encode(image).latent_dist.mode()                                                                                                                                                                                                              │
│   240                                                                                                                                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\diffusers\utils\accelerate_utils.py:46 in wrapper                                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                                      │
│   45 │   │   │   self._hf_hook.pre_forward(self)                                                                                                                                                                                                                                     │
│ ❱ 46 │   │   return method(self, *args, **kwargs)                                                                                                                                                                                                                                    │
│   47                                                                                                                                                                                                                                                                                 │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\diffusers\models\autoencoders\autoencoder_kl_temporal_decoder.py:334 in encode                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                      │
│   333 │   │   """                                                                                                                                                                                                                                                                    │
│ ❱ 334 │   │   h = self.encoder(x)                                                                                                                                                                                                                                                    │
│   335 │   │   moments = self.quant_conv(h)                                                                                                                                                                                                                                           │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\module.py:1532 in _wrapped_call_impl                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                      │
│   1531 │   │   else:                                                                                                                                                                                                                                                                 │
│ ❱ 1532 │   │   │   return self._call_impl(*args, **kwargs)                                                                                                                                                                                                                           │
│   1533                                                                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\module.py:1541 in _call_impl                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                                      │
│   1540 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                                                                                                                                                                                       │
│ ❱ 1541 │   │   │   return forward_call(*args, **kwargs)                                                                                                                                                                                                                              │
│   1542                                                                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\diffusers\models\autoencoders\vae.py:143 in forward                                                                                                                                                                                      │
│                                                                                                                                                                                                                                                                                      │
│   142 │   │                                                                                                                                                                                                                                                                          │
│ ❱ 143 │   │   sample = self.conv_in(sample)                                                                                                                                                                                                                                          │
│   144                                                                                                                                                                                                                                                                                │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\module.py:1532 in _wrapped_call_impl                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                      │
│   1531 │   │   else:                                                                                                                                                                                                                                                                 │
│ ❱ 1532 │   │   │   return self._call_impl(*args, **kwargs)                                                                                                                                                                                                                           │
│   1533                                                                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\module.py:1541 in _call_impl                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                                      │
│   1540 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                                                                                                                                                                                       │
│ ❱ 1541 │   │   │   return forward_call(*args, **kwargs)                                                                                                                                                                                                                              │
│   1542                                                                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\conv.py:460 in forward                                                                                                                                                                                                  │
│                                                                                                                                                                                                                                                                                      │
│    459 │   def forward(self, input: Tensor) -> Tensor:                                                                                                                                                                                                                               │
│ ❱  460 │   │   return self._conv_forward(input, self.weight, self.bias)                                                                                                                                                                                                              │
│    461                                                                                                                                                                                                                                                                               │
│                                                                                                                                                                                                                                                                                      │
│ D:\SD.Next\automatic\venv\Lib\site-packages\torch\nn\modules\conv.py:456 in _conv_forward                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                                      │
│    455 │   │   │   │   │   │   │   _pair(0), self.dilation, self.groups)                                                                                                                                                                                                             │
│ ❱  456 │   │   return F.conv2d(input, weight, bias, self.stride,                                                                                                                                                                                                                     │
│    457 │   │   │   │   │   │   self.padding, self.dilation, self.groups)                                                                                                                                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Input type (float) and bias type (struct c10::BFloat16) should be the same

I've tried finding the root cause, but that's too deep into the woods for me to understand easily.
Any hints what I can try?
Or do to help you to debug it?

[vladmandic]

that one seems like a big in svd diffusers code, it encodes image using default precision instead of model precision, so when model tries to use encoded image, it doesn't match. reality is that video model support is far more flaky in general as there are less ppl using it so less interest so less work going into it.

[holgerpieta]

Ah, OK. I've tried to understand all the various bits and pieces where I could set anything about precision, but I'm not yet able to keep them apart.
Any hint if I could set something to make the precisions match, even if it wouldn't be optimal for performance?

[vladmandic]

not from the top of my head.
can you pls create an issue for this svd i2v problem?