Error when runing main_v2

[zhewei-mt]

Hello,
I am trying to run you great work with the command:
python main_v2.py -c config/multimodal_context_v2.yml
but got an error:
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/torch/nn/functional.py", line 1237, in leaky_relu
result = torch._C.nn.leaky_relu(input, negative_slope)
RuntimeError: CUDA error: no kernel image is available for execution on the device

I googled it downgrade my pytorch version to 1.5.0 but the error still persists.
Do you know how to fix this error? Any help will be appreciate!!

My torch version:
pytorch 1.5.0 py3.7_cuda10.2.89_cudnn7.6.5_0 pytorch
torchvision 0.6.0 py37_cu102 pytorch

[UttaranB127]

Unfortunately, I cannot help much with versioning or cuda backend issues. If it helps, these are the configs I have:

• python version: running python --version gives 3.7.13
• torch version: running pip freeze gives torch==1.12.0+cu116
• torchvision version: torch version: running pip freeze gives torchvision==0.13.0+cu116
• cuda version: running torch.version.cuda gives 11.6
• cudnn version: running torch.backends.cudnn.version() gives 8302

[zhewei-mt]

Thank you for your reply!
I upgrade all related version to be consistent with cuda 11.7 and the problem is gone.
But at the every last step, another error occurs when executing the following code:
ani.save(video_path, fps=15, dpi=80)
And here is the error message:
MovieWriter stderr:
[libopenh264 @ 0x55eec1dec300] Incorrect library version loaded
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

Traceback (most recent call last):
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 234, in saving
yield self
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 1093, in save
writer.grab_frame(**savefig_kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 352, in grab_frame
dpi=self.dpi, **savefig_kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/figure.py", line 3058, in savefig
self.canvas.print_figure(fname, **kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 2325, in print_figure
**kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 1648, in wrapper
return func(*args, **kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/_api/deprecation.py", line 415, in wrapper
return func(*inner_args, **inner_kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py", line 486, in print_raw
fh.write(renderer.buffer_rgba())
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main_v2.py", line 149, in
s2ag_epoch=290, samples=samples, make_video=True, save_pkl=True)
File "/home/zhewei.qiu/audio2body/3d/speech2affective_gestures/processor_v2.py", line 1564, in generate_gestures_by_dataset
make_video=make_video, save_pkl=save_pkl)
File "/home/zhewei.qiu/audio2body/3d/speech2affective_gestures/processor_v2.py", line 1408, in render_clip
delete_audio_file=False)
File "/home/zhewei.qiu/audio2body/3d/speech2affective_gestures/utils/gen_utils.py", line 128, in create_video_and_save
ani.save(video_path, fps=15, dpi=80) # dpi 150 for a higher resolution
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 1093, in save
writer.grab_frame(**savefig_kwargs)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 236, in saving
self.finish()
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 342, in finish
self._cleanup() # Inline _cleanup() once cleanup() is removed.
File "/home/zhewei.qiu/anaconda3/envs/s2ag-env/lib/python3.7/site-packages/matplotlib/animation.py", line 374, in _cleanup
self._proc.returncode, self._proc.args, out, err)
subprocess.CalledProcessError: Command '['ffmpeg', '-f', 'rawvideo', '-vcodec', 'rawvideo', '-s', '960x320', '-pix_fmt', 'rgba', '-r', '15', '-loglevel', 'error', '-i', 'pipe:', '-vcodec', 'h264', '-pix_fmt', 'yuv420p', '-y', '/home/zhewei.qiu/audio2body/3d/speech2affective_gestures/outputs/genea_challenge_2020/videos_trimodal_style/temp_TestSeq001_s0_0.00_128.83_290_0.mp4']' returned non-zero exit status 1.

Is that because the version of matplotlib? And can you share which version of matplotlib you use?

[UttaranB127]

Yes, seems like another version issue. I am using matplotlib==3.5.2.

[zhewei-mt]

There is something weird happening. I downgrade matplotlib from 3.5.3 to 3.5.2 and still get same error. But when I try using debug mode is vs code, the script seems to run successfully and I can get the .wav and .mp4 file.
Also, can you provide pure inference script, e.g., the input is audio, text, identity and the output is the predict 3d position of all joint?

[UttaranB127]

It's non-trivial to prepare a "pure" inference script because of all the library, path, and data dependencies. But if you run generate_gestures_by_dataset (beginning line 1441) in processor_v2.py with the dataset set to 'ted_db', that's essentially the default inference script.

[zhewei-mt]

Got you. And is there a way to get the rotation information from the 3d joint position output by your model or do you plan to use rotation information as a supervisor since 3d joint position is not friendly in existing render engine such as UE and blender?

[UttaranB127]

A good question! I would recommend checking out our blender files inside the blender folder, where we essentially use a form of IK to obtain the rotations and render the motions on 3D meshes. Our approach here is similar to that of https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context, so you can check out their paper and blender project as well for additional context.

[zhewei-mt]

I tried the repo you mentioned. I got the same error when making the result video. Seems that your repo and his are highly related. I tried different version of matplotlib but the error is not gone.

[UttaranB127]

Is it the same situation where things run in debug mode but not otherwise? Are you able to generate the pose outputs? I'm not sure what the exact issue could be (and I can't seem to replicate it), but maybe you can try other animation scripts with the generated poses outputs.