NaN values appearing in generated video frames

[YuCheng-Chang]

While using Text2Video-Zero, I've encountered a critical issue where NaN (Not a Number) values are appearing in some of the generated video frames, resulting in partial black screen output in the final video.

Problem Description:

• During video generation using the Text2Video-Zero pipeline, NaN values are occurring in certain frames.
• This issue manifests as black screen sections in the output video, significantly impacting the quality and usability of the generated content.
• The black screen appears to correspond directly to the frames where NaN values are present.
• I have checked the pretrained weights downloaded from Hugging Face, and they do not contain any NaN values, suggesting the issue arises during the generation process.

My inference code:
import torch
from model import Model
print(torch.cuda.is_available()) # True
print(torch.version.cuda) # 11.3
print(torch.cuda.get_device_name(torch.cuda.current_device())) # NVIDIA GeForce RTX 2080
model = Model(device = "cuda", dtype = torch.float16)
from hf_utils import get_model_list
model_list = get_model_list()

print(f'model name={model_list[3]}')# model name=CompVis/stable-diffusion-v1-4
prompt = "A chinese landscape painting of a boat drifting in a river stream at the foot of mist-surronded green moutains. The with fluffy clouds floats by with elegant birds taking flight."
params = {"t0": 44, "t1": 47 , "motion_field_strength_x" : 1, "motion_field_strength_y" : 1, "video_length": 8, "chunk_size": 4, "model_name": model_list[3]}
out_path, fps = f"./text2video_landscape.mp4", 4
model.process_text2video(prompt, fps = fps, path = out_path, **params);

Additional Information:

• Python version: 3.8.12
• Operating System: Windows 11
• GPU model (if applicable): NVIDIA GeForce RTX 2080

Questions:

1. Is this a known issue with the current version of Text2Video-Zero?
2. Are there any workarounds or solutions to prevent NaN values from appearing in the generated frames?
3. Could this be related to specific input parameters or the content of the prompt?

I would appreciate any insights or assistance in resolving this issue. Thank you for your time and support.