Skip to content

README.md

Latest commit

 

History

History
236 lines (177 loc) · 9.25 KB

README.md

File metadata and controls

236 lines (177 loc) · 9.25 KB
Raw


EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generation

Xiangyue Zhang*, Jianfang Li*, Jiaxu Zhang, Jianqiang Ren, Liefeng Bo, Zhigang Tu†

✨ACM MM 2025✨

EchoMask teaser
Motivation

📣 Updates

  • [2026.07.28] Released the 25-speaker EchoMask checkpoint, reusable All-Speaker representation weights, evaluation metrics, and paired NPZ results.
  • [2025.09.11] 🔥 Release: - [✔] Inference code - [✔] Training code

💖 Inference Data

The two releases use different training protocols and are provided separately:

  • Speaker 2 (paper protocol): download the original test .npz files from Google Drive.
  • All Speakers (25 English BEAT2 speakers): download the paired test .npz files from Google Drive.

📊 Results

Training protocol Speakers FGD ↓ (×10-1) BC ↑ (×10-1) DIV ↑ MSE ↓ (×10-8) LVD ↓ (×10-5)
Speaker 2 (paper) 1 4.623 7.738 13.370 6.761 7.290
All Speakers (released checkpoint) 25 5.656 4.951 9.299 4.700 6.090

The Speaker 2 row is copied from Table 1 of the paper. The All-Speaker row is measured on the released checkpoint selected by a coarse 10-epoch search followed by a local 1-epoch search. Raw metrics and checkpoint metadata are in results/all_speakers. The two rows use different training protocols and should not be interpreted as a controlled single-speaker versus multi-speaker ablation.

⚒️ Quick Start

Build Environtment

We Recommend a python version =3.8 and cuda version >=12.1. Then build environment as follows:

# [Optional] Create a virtual env
conda create -n EchoMask python=3.8
conda activate EchoMask
# pytorch, torchvison
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia
# Install with pip:
pip install -r requirements.txt
sudo apt-get update && sudo apt-get install -y ffmpeg  # if you don't have ffmpeg

Download Data

please refer to EMAGE and download datasets from BEAT2 for datasets and place the dataset inside the EchoMask folder, i.e. path-to-EchoMask/BEAT2

Download Weights

Download hubert-large-ls960-ft(used for extracting audio features) weights and place itside in the EchoMask folder, i.e. path-to-EchoMask/facebook. If you are in China, you can use hf-mirror for faster and more reliable downloads. The process may take some time, so please be patient.

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download facebook/hubert-large-ls960-ft --local-dir facebook/hubert-large-ls960-ft

Choose the weights that match the intended protocol:

Keep the original Speaker 2 archive in its documented locations. Install the All-Speaker archive without overwriting those files:

unzip EchoMask_all_speakers_weights_25spk.zip
cp -R EchoMask_all_speakers_weights/weights/all_speakers ./weights/

Finally, these EchoMask folder should be orgnized as follows:

.
├── BEAT2
│   └── beat_english_v2.0.0
├── configs
├── dataloaders
├── datasets   (this folder will appear after you generate the datasets)
│   ├── beat2_cache2
│   ├── beat2_echomask_train
│   └── echomask_dataloader.py
├── facebook
│   └── hubert-large-ls960-ft
├── models
├── optimizers
├── src
├── utils
├── weights
│   ├── pretrained_vq
│   ├── smplx_models
│   ├── best_echomask.bin
│   └── all_speakers
│       ├── pretrained_vq
│       └── best_echomask.bin
├── ae_trainer.py
├── aeface_trainer.py
├── aelower_trainer.py
├── aelowerfoot_trainer.py
├── requirements.txt
├── echomask_trainer.py
├── echomask_trainer.py
└── train.py

Generate Dataset

Notice: Please make sure you are in the root directory, i.e. path-to-EchoMask.

  1. To generate the training dataset, run:
python dataloaders/save_train_dataset.py # generate train dataset

This process may take some time, so please be patient.

  1. To generate the test dataset, run:
python dataloaders/save_test_dataset.py # generate test dataset

🚀Training, Testing, and Inference

Training of EchoMask

The commands below preserve the original Speaker 2 protocol. For the separate 25-speaker preprocessing, training, checkpoint search, and evaluation workflow, see scripts/all_speakers/README.md.

Train RVQ-VAE

The commands below reproduce the original Speaker 2 RVQ-VAE protocol. You can train the models yourself and place them under path-to-EchoMask/weights, or download weights that match your training protocol:

  • Speaker 2 (paper protocol): use the original pretrained weights.
  • All Speakers (25 English BEAT2 speakers): use the All-Speaker weights. They contain the five shared representation checkpoints under weights/all_speakers/pretrained_vq and the selected EchoMask checkpoint. The representation models do not need to be retrained for EchoMask.
torchrun --nproc_per_mode=1 train.py --train_rvq --config configs/cnn_vqvae_face_30.yaml # face
torchrun --nproc_per_mode=1 train.py --train_rvq --config configs/cnn_vqvae_hands_30.yaml # hands
torchrun --nproc_per_mode=1 train.py --train_rvq --config configs/cnn_vqvae_upper_30.yaml # upper body
torchrun --nproc_per_mode=1 train.py --train_rvq --config configs/cnn_vqvae_lower_foot_30.yaml # lower foot
torchrun --nproc_per_mode=1 train.py --train_rvq --config configs/cnn_vqvae_lower_30.yaml # lower body

Training of EchoMask

torchrun --nproc_per_mode=1 train.py --config configs/echomask.yaml

Testing of EchoMask

Notice: Before running the test code, make sure the load_ckpt and base_ckpt paths in configs/echomask.yaml are set correctly.

torchrun --nproc_per_mode=1 train.py --test_state --config configs/echomask.yaml

Inference

you can put your inference wav format aduio on ./demo path, for example, you can run:

torchrun --nproc_per_mode=1 train.py --inference --config configs/echomask.yaml --audio_infer_path ./demo/2_scott_0_1_1_test.wav

📺 Visualize

Following EMAGE, you can download SMPLX blender addon, and install it in your blender 3.x or 4.x. Click the button Add Animation to visualize the generated smplx file (like xxx.npz).

🙏 Acknowledgments

Thanks to EMAGE, DiffSHEG, our code is partially borrowing from them. Please check these useful repos.

📖 Citation

If you find our code or paper helps, please consider citing:

@inproceedings{zhang2025echomask,
  title={EchoMask: Speech-Queried Attention-based Mask Modeling for Holistic Co-Speech Motion Generation},
  author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Ren, Jianqiang and Bo, Liefeng and Tu, Zhigang},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  pages={10827--10836},
  year={2025}
}