Skip to content

Add supports for Mac M1#227

Open
tomy128 wants to merge 3 commits into
OpenTalker:mainfrom
tomy128:m1
Open

Add supports for Mac M1#227
tomy128 wants to merge 3 commits into
OpenTalker:mainfrom
tomy128:m1

Conversation

@tomy128

@tomy128 tomy128 commented Apr 1, 2024

Copy link
Copy Markdown

No description provided.

@tomy128

tomy128 commented Apr 1, 2024

Copy link
Copy Markdown
Author

Tested on Mac m1 silicon ship. Below is the output of test command:

python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/1_1.mp4
show output logs

(venv) ops@arm64:~/www/video-retalking $ PYTORCH_ENABLE_MPS_FALLBACK=1 python3 inference.py
--face examples/face/1.mp4
--audio examples/audio/1.wav
--outfile results/1_1.mp4
[Info] Using mps for inference.
[Step 0] Number of frames available for inference: 135
[Step 1] Using saved landmarks.
[Step 2] 3DMM Extraction In Video:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 135/135 [00:02<00:00, 47.97it/s]
using expression center
Load checkpoint from: checkpoints/DNet.pt
Load checkpoint from: checkpoints/LNet.pth
Load checkpoint from: checkpoints/ENet.pth
[Step 3] Using saved stabilized video.
[Step 4] Load audio; Length of mel chunks: 109
[Step 5] Reference Enhancement: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [02:07<00:00, 1.17s/it]
landmark Det:: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:47<00:00, 2.29it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 39592.89it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 997.79it/s]
FaceDet:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:20<00:00, 1.35it/s]
[Step 6] Lip Synthesis:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [16:40<00:00, 142.87s/it]
outfile: results/1_1.mp4

@tomy128

tomy128 commented Apr 1, 2024

Copy link
Copy Markdown
Author

The PYTORCH_ENABLE_MPS_FALLBACK=1 variable is solve Error of pytorch below:

    ....
    ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm). 
NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device.   
 If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764.   
As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op.   
WARNING: this will be slower than running natively on MPS.

@tomy128

tomy128 commented Apr 1, 2024

Copy link
Copy Markdown
Author

completion for #217 and #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant