Added super-resolution using GFPGAN and CodeFormer

[A1Kumari]

Enhanced Pipeline with Super-Resolution

How to Run

1. Set up the environment:

   bash setup_env.sh

2. Download the required checkpoints:
   GFPGAN: GFPGANv1.3.pth
   CodeFormer: codeformer.pth

   Place the checkpoints in the checkpoints/ directory.

3. bash inference.sh --superres [GFPGAN/CodeFormer]

---

** Summary of Changes**

Here’s a brief explanation of what changes I made to each file:

File: latentsync/pipelines/lipsync_pipeline.py

• Changes:
  • Added a helper function apply_superres to apply GFPGAN or CodeFormer.
  • Modified the __call__ method to calculate the resolution ratio and apply super-resolution if necessary.

File: inference.py

• Changes:
  • Added a new argument --superres to allow users to choose between GFPGAN and CodeFormer.

File: inference.sh

• Changes:
  • Updated the script to include the --superres parameter.

[chunyu-li]

Good work! 👍
I've been quite busy recently. I'll check this and merge it once I have some free time.

[SantiagoSchez]

I think some more changes are needed in this PR, as running gradio_app.py won't work anymore. Also, what codeformer package are you using for Python? None has the CodeFormerRestorer class as used in code. And GFPGAN 1.3.0 throws some errors due to conflicts with other dependencies; as a workaround, a version that worked for me was installing via pip install git+https://github.com/TencentARC/GFPGAN.git@8d2447a2d918f8eba5a4a01463fd48e45126a379 --prefer-binary.

EDIT: this is just an AI hallucinating. Probably the author of the PR didn't even test it, as it doesn't work.

[romain130492]

Is implementing Codeformer in the roadmap @chunyu-li ?

[zouzhekang]

why don't you use gfpganv1.4? it can be download from: https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth