Skip to content

README.md

Latest commit

 

History

History
229 lines (152 loc) · 7.97 KB

README.md

File metadata and controls

229 lines (152 loc) · 7.97 KB
Raw


Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints

Xiangyue Zhang*, Jianfang Li*†, Jianqiang Ren, Jiaxu Zhang

✨AAAI 2026✨

GlobalDiff is developed by Alibaba Cloud and released under the Apache License 2.0.

GlobalDiff Overview

🚧 Code Release Plan

  • Training: ✅ 2025.12.29
  • Testing: ✅ 2025.12.29

💖 Inference Data

If you would like to compare your paper’s results with GlobalDiff but find it too difficult to run the repository, you can simply download the test .npz file. Results for person-2 are provided in best_pid_2.zip.

If you want to compare your results with our method, you could just simply download all test results without reproducing the codes.

⚡ Quick Start

GlobalDiff now includes two helper entrypoints:

  • python tools/check_env.py: verify whether your local environment is ready
  • python tools/run.py ...: unified commands for preprocessing, training, and inference

The fastest path to a runnable setup is:

conda create -n globaldiff python=3.10 -y
conda activate globaldiff
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
python tools/check_env.py

If you already have BEAT2 and the required checkpoints prepared, you can run inference with:

python tools/run.py infer \
  ckpt/split/DiffusionDITNetPartsFixedExpressions2PostNorm_LL_split_HorizonFlip_MaskedVAE3_W02_BoneDirLoss \
  --data-root /path/to/beat_v2.0.0/beat_english_v2.0.0

🛠️ Environment Setup

The repository now provides a lightweight requirements.txt, a unified CLI under tools/run.py, and an environment checker under tools/check_env.py to make the project easier to reproduce.

Recommended Setup

  • OS: Linux
  • Python: 3.10
  • CUDA: 11.8 or a version compatible with your local PyTorch build
  • GPU: at least 1 NVIDIA GPU for inference, 4 GPUs are recommended for the training command used below

Create a conda environment:

conda create -n globaldiff python=3.10 -y
conda activate globaldiff

Install PyTorch first. Please choose the command that matches your CUDA version from the official PyTorch website. For CUDA 11.8, you can use:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Then install the remaining Python dependencies:

pip install -r requirements.txt

External Resources

GlobalDiff also relies on two external resources:

  1. SMPL-X model file
    Download SMPLX_NEUTRAL_2020.npz from the official SMPL-X release, then either:

    • place it at Data/SMPLX_NEUTRAL_2020.npz, or
    • set GLOBALDIFF_SMPLX_MODEL_PATH to its absolute path

  2. WavLM checkpoint
    By default, the code now uses the Hugging Face model id patrickvonplaten/wavlm-libri-clean-100h-large.
    If you already have the model cached locally, you can point the code to that directory with:

export GLOBALDIFF_SMPLX_MODEL_PATH=/path/to/SMPLX_NEUTRAL_2020.npz
export GLOBALDIFF_WAVLM_PATH=/path/to/wavlm-libri-clean-100h-large

If you do not set GLOBALDIFF_WAVLM_PATH, Transformers will download the model automatically from Hugging Face the first time it is used.

Quick Check

You can run:

python tools/check_env.py

or, if you want to verify the BEAT2 path as well:

python tools/check_env.py --data-root /path/to/beat_v2.0.0/beat_english_v2.0.0

The checker validates Python, core packages, PyTorch/CUDA, SMPL-X, WavLM source, and required checkpoints.

Before training or inference, make sure the following are ready:

  • Data/BEAT2/create_lmdb.py can access your downloaded BEAT2 data
  • Scripts/VAE/ckpt/split/global/MaskedVAE2-HorizonFlip/best.pt exists
  • Scripts/VAE/ckpt/split/global/MaskedVAE3-HorizonFlip/best.pt exists
  • Scripts/FM/ckpt/split/SimpleSpeechModel/best.pt is available after unzipping best.zip

Download Data

please refer to EMAGE and download datasets from BEAT2 for datasets. If you are in China, you can use hf-mirror for faster and more reliable downloads. The process may take some time, so please be patient.

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --repo-type dataset --resume-download H-Liu1997/BEAT2 --local-dir H-Liu1997/BEAT2

1) Create Training Data

Create LMDB with the helper CLI:

python tools/run.py preprocess-lmdb /path/to/beat_v2.0.0/beat_english_v2.0.0 --split train

This may take a while. Then extract HuBERT features:

python tools/run.py preprocess-wavlm Data/BEAT2/train_seq_size_60_stride_size_20_global.lmdb

This takes about 1 hour.

You will get two folders:

  • train_seq_size_60_stride_size_20_global.lmdb
  • train_seq_size_60_stride_size_20_global_wavlm.lmdb

2) Training

Unzip best.zip to:

  • Scripts/FM/ckpt/split/SimpleSpeechModel/

Then run training:

python tools/run.py train-fm --nproc-per-node 4

If you need custom arguments, append them after --, for example:

python tools/run.py train-fm --nproc-per-node 4 -- --batch_size 64 --epoch 500

3) Inference

python tools/run.py infer \
  ckpt/split/DiffusionDITNetPartsFixedExpressions2PostNorm_LL_split_HorizonFlip_MaskedVAE3_W02_BoneDirLoss \
  --data-root /path/to/beat_v2.0.0/beat_english_v2.0.0

This will run all checkpoints sequentially. The FID record is saved to:

  • ckpt/split/DiffusionDITNetPartsFixedExpressions2PostNorm_LL_split_HorizonFlip_MaskedVAE3_W02_BoneDirLoss/record_2.txt

The results with the best FID are saved to:

  • ckpt/split/DiffusionDITNetPartsFixedExpressions2PostNorm_LL_split_HorizonFlip_MaskedVAE3_W02_BoneDirLoss/best_pid_2/

Note: This FID is computed using our internal evaluation implementation.
For fair comparison, please use the official evaluation from EMAGE or SemTalk:

4) Legacy Scripts

The original research scripts are still preserved under Data/BEAT2, Scripts/VAE, and Scripts/FM. If you prefer the old workflow, you can still call them directly. The new tools/run.py wrapper is only a convenience layer for easier reproduction.

5) 📺 Visualization

Following EMAGE, you can download SMPLX blender addon, and install it in your blender 3.x or 4.x. Click the button Add Animation to visualize the generated smplx file (like xxx.npz).

🙏 Acknowledgments

Thanks to EMAGE, SemTalk, our code is partially borrowing from them. Please check these useful repos.

📚 Citation

If you find our code or paper helps, please consider citing:

    @article{zhang2025mitigating,
      title={Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints},
      author={Zhang, Xiangyue and Li, Jianfang and Ren, Jianqiang and Zhang, Jiaxu},
      journal={arXiv preprint arXiv:2511.10076},
      year={2025}
    }