Implement Sadtalker

[Koichi73]

#1203 のPRです。
opset = 20

エラー報告

ailia v1.5.0でSegmentation faultが発生したため、onnxruntimeモードオプションを実装しました。

該当箇所

animation_generator.onnx.prototxt
sadtalker.py 142行目
∟MODEL_ANIMATION_GENERATOR_PATHをNoneにすると動作する

animation_generator.onnxの出力コード

https://gist.github.com/Koichi73/f93e58d2599bcd0e21dc91ba9dec1020

オリジナルモデルとの差異

• Enhance Mode：gfpganのみ実装
• Background Enhancer：非実装
• 3D Mode：非実装

※パラメータの参考

ailia-modelsとの依存関係

• retinaface
• GFPGAN

[kyakuno]

モデルアップロード済み
https://storage.googleapis.com/ailia-models/sadtalker/animation_generator.onnx.prototxt

[kyakuno]

問題の起きるprototxtを再生成。

https://storage.googleapis.com/ailia-models/sadtalker/animation_generator.old.onnx.prototxt
->
https://storage.googleapis.com/ailia-models/sadtalker/animation_generator.onnx.prototxt

[kyakuno]

保存時にオプションが必要そう。

kyakuno@mbakk sadtalker % python3 sadtalker.py 
 INFO arg_utils.py (13) : Start!
 INFO arg_utils.py (163) : env_id: 2
 INFO arg_utils.py (166) : MPSDNN-Apple M2
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
3DMM Extraction for source image
landmark Det: 100%|███████████████████████████████| 1/1 [00:00<00:00,  9.64it/s]
3DMM Extraction In Video: 100%|███████████████████| 1/1 [00:00<00:00, 16.33it/s]
mel: 100%|██████████████████████████████████| 84/84 [00:00<00:00, 149225.56it/s]
audio2exp: 100%|██████████████████████████████████| 9/9 [00:00<00:00, 65.20it/s]
Face Renderer: 100%|████████████████████████████| 42/42 [01:58<00:00,  2.81s/it]
Traceback (most recent call last):
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/face_recognition/sadtalker/sadtalker.py", line 298, in <module>
    main()
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/face_recognition/sadtalker/sadtalker.py", line 281, in main
    result = generate_animation(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/face_recognition/sadtalker/sadtalker.py", line 212, in generate_animation
    result = animate_from_coeff.generate(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/face_recognition/sadtalker/animation/animate.py", line 60, in generate
    imageio.mimsave(path, result, fps=float(25))
  File "/opt/homebrew/lib/python3.12/site-packages/imageio/v2.py", line 495, in mimwrite
    return file.write(ims, is_batch=True, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/imageio/plugins/pyav.py", line 634, in write
    self.init_video_stream(codec, fps=fps, pixel_format=out_pixel_format)
  File "/opt/homebrew/lib/python3.12/site-packages/imageio/plugins/pyav.py", line 845, in init_video_stream
    stream = self._container.add_stream(codec, fps)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "av/container/output.pyx", line 63, in av.container.output.OutputContainer.add_stream
ValueError: needs one of codec_name or template