Skip to content

Conversation

@tomy128
Copy link

@tomy128 tomy128 commented Apr 1, 2024

No description provided.

@tomy128
Copy link
Author

tomy128 commented Apr 1, 2024

Tested on Mac m1 silicon ship. Below is the output of test command:

python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/1_1.mp4
show output logs

(venv) ops@arm64:~/www/video-retalking $ PYTORCH_ENABLE_MPS_FALLBACK=1 python3 inference.py
--face examples/face/1.mp4
--audio examples/audio/1.wav
--outfile results/1_1.mp4
[Info] Using mps for inference.
[Step 0] Number of frames available for inference: 135
[Step 1] Using saved landmarks.
[Step 2] 3DMM Extraction In Video:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 135/135 [00:02<00:00, 47.97it/s]
using expression center
Load checkpoint from: checkpoints/DNet.pt
Load checkpoint from: checkpoints/LNet.pth
Load checkpoint from: checkpoints/ENet.pth
[Step 3] Using saved stabilized video.
[Step 4] Load audio; Length of mel chunks: 109
[Step 5] Reference Enhancement: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [02:07<00:00, 1.17s/it]
landmark Det:: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:47<00:00, 2.29it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 39592.89it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 997.79it/s]
FaceDet:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:20<00:00, 1.35it/s]
[Step 6] Lip Synthesis:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [16:40<00:00, 142.87s/it]
outfile: results/1_1.mp4

@tomy128
Copy link
Author

tomy128 commented Apr 1, 2024

The PYTORCH_ENABLE_MPS_FALLBACK=1 variable is solve Error of pytorch below:

    ....
    ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm). 
NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device.   
 If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764.   
As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op.   
WARNING: this will be slower than running natively on MPS.

@tomy128
Copy link
Author

tomy128 commented Apr 1, 2024

completion for #217 and #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant