HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos

HDMI is a novel framework that enables humanoid robots to acquire diverse whole-body interaction skills directly from monocular RGB videos of human demonstrations.

This repository contains the official training code of HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos.

🚀 Quick Start

Setup virtual environment with uv sync and apply mjlab patch (venv files)

patch --forward -p0 < patches/mjlab_local.patch

Prepare Data

AMASS data: refer to https://github.com/Axellwppr/gentle-humanoid-training. use scripts/data_process/generate_amass_dataset.py to convert to HDMI format.

Lafan data: refer to https://github.com/EGalahad/lafan-process.

Train and Evaluate

Teacher policy

uv run scripts/train.py algo=ppo_roa_train task=G1/tracking/amass
uv run scripts/train.py algo=ppo_roa_train task=G1/tracking/lafan

Student policy

uv run scripts/train.py algo=ppo_roa_finetune task=G1/tracking/lafan

Play policy

uv run scripts/play.py algo=ppo_roa_play task=G1/tracking/lafan checkpoint_path=run:<wandb_run_path>

add export_policy=true to export onnx model.

Sim2Real

Please see github.com/EGalahad/sim2real for details.