humanoid-amp-ppo

AMP-based humanoid RL training for tasks:

• Mjlab-G1-Locomotion
• Mjlab-G1-DualArm

An AMP discriminator is trained on human motion data to help the humanoid RL policy learn with fewer task-specific rewards while encouraging more human-like behavior.

Locomotion policy:

MP4 version

Dual-arm policy:

MP4 version

Dual-arm training (ResMimic-style residual learning)

The dual-arm task trains a residual policy on top of a frozen locomotion actor (following ResMimic): the base policy supplies balance and whole-body coordination, and the RL policy learns task-specific corrections (a = a_base + 0.3 * a_residual). The base checkpoint (models/locomotion.pt) is part of the task's default env config, so training and play compose actions identically.

Supporting mechanisms:

• Success-adaptive curricula — the virtual object controller's assistance and the object's reset-height bootstrap decay only while the lift-success EMA stays above threshold, so difficulty self-regulates instead of following a blind step schedule.
• Assistance-force penalty — the policy is increasingly penalized for the virtual-PD force actually used, paying it to take over the lift itself.
• AMP as a style regularizer — the AMP reward coefficient is capped near the task reward's standing baseline so imitating the mocap never outpays completing the task.

Train:

MUJOCO_GL=egl PYTHONPATH=src uv run python train_dualarm_from_locomotion.py

Codebase Structure

• src/mjlab_g1/
  • task configs, envs, wrappers, viewers
• rsl_rl/
  • RSL-RL 5.2 with migrated AMP-PPO code
• dataset/
  • AMP motion datasets

Install

This repo is developed with uv.

Install uv, clone the repository together with the DeFM submodule, and install the project:

curl -LsSf https://astral.sh/uv/install.sh | sh

git clone --recurse-submodules https://github.com/shuayyy/humanoid-amp-ppo.git
cd humanoid-amp-ppo

uv sync

Installed packages:

• mjlab-g1
• rsl-rl-lib from rsl_rl
• defm from third_party/defm

uv sync installs DeFM and its runtime dependencies. Do not install DeFM with --no-deps; dual-arm needs those dependencies for the depth encoder.

Task IDs

Registered tasks:

• Mjlab-G1-Locomotion
• Mjlab-G1-DualArm

Training

Locomotion:

MUJOCO_GL=egl PYTHONPATH=src uv run python src/mjlab_g1/scripts/train.py \
Mjlab-G1-Locomotion \
--video False

Dual arm:

MUJOCO_GL=egl PYTHONPATH=src uv run python src/mjlab_g1/scripts/train.py \
Mjlab-G1-DualArm \
--video False

Dual-arm uses a depth camera and frozen DeFM features. On smaller GPUs, reduce the environment count, for example:

MUJOCO_GL=egl PYTHONPATH=src uv run python src/mjlab_g1/scripts/train.py \
Mjlab-G1-DualArm \
--env.scene.num_envs 64 \
--video False

Play

Trained Policies:

• models/locomotion.pt
• models/dual_arm.pt

Locomotion:

PYTHONPATH=src uv run python src/mjlab_g1/scripts/play.py \
Mjlab-G1-Locomotion \
--checkpoint-file models/locomotion.pt

Dual arm:

PYTHONPATH=src uv run python src/mjlab_g1/scripts/play.py \
Mjlab-G1-DualArm \
--checkpoint-file models/dual_arm.pt

AMP datasets

Current task-specific AMP dataset paths:

• locomotion: dataset/locomotion
• dualarm: dataset/dualarm