Mimicboy

Overview

This repository is designed to enable the creation, management, and training of teleoperation datasets for the Unitree G1 robot. By leveraging cutting-edge tools such as IsaacSim, Unity, ROS2, and VIVE Pro, we facilitate the collection of high-quality data for robot teleoperation tasks.

Key Features

• Dataset Creation: Generate teleoperation datasets in HuggingFace and LeRobot formats.
• Integration with Modern Tools:
  • Use IsaacSim for realistic robot simulation.
  • Leverage Unity for VR-based teleoperation using the VIVE Pro headset.
  • Utilize ROS2 for data communication between the simulation and VR interface.
• Policy Training: Train advanced policies such as ACT (Action Chunking with Transformer), Diffusion Policy, and π0 on the collected datasets.

Structure

The repository, Mimicboy, consists of mimicboy, mimicboy_assets, mimicboy_ros2, and mimicboy_tasks:

.
├── mimicboy
│   ├── config
│   ├── docs
│   ├── mimicboy
│   ├── pyproject.toml
│   └── setup.py
├── mimicboy_assets
│   ├── config
│   ├── data
│   ├── docs
│   ├── mimicboy_assets
│   ├── pyproject.toml
│   └── setup.py
├── mimicboy_ros2
│   ├── config
│   ├── docs
│   ├── mimicboy_ros2
│   ├── pyproject.toml
│   └── setup.py
└── mimicboy_tasks
    ├── config
    ├── docs
    ├── mimicboy_tasks
    ├── pyproject.toml
    └── setup.py

Installation

To install this repository, follow these steps:

1. Connect to the server.

ssh -L <WEBPORT>:localhost:<WEBPORT> <server_name>

Note

Please use the WEBPORT specified in the .env.sample file. The available server_name options are: robot_dev2, robot_dev4, robot_dev5, and robot_dev6.

2. Clone the repository.

git clone https://github.com/makolon/Mimicboy.git

3. Build docker container.

cp .env.sample .env
docker compose build isaac-lab-base
docker compose build isaac-lab-ros2
docker compose build isaac-lab-mimicboy

4. Enter mimicboy container.

docker compose -p <container_name> run isaac-lab-mimicboy

Note

If you encounter an error indicating that the WEBPORT is already in use and cannot run docker, please change the WEBPORT and DISPLAY numbers in the .env.sample file, then start again from step 1. You can check if the WEBPORT is already in use by running docker ps.

Note

Update the DISPLAY environment in the .env file using free display. (The display free if it is not in the /tmp/.X11-unix/ folder of the host machine) Also, change the WEBPORT to enable the first free port (Get it by calculating DISPLAY + 6080).

# If there is no file `/tmp/.X11-unix/X20`
DISPLAY=:20
WEBPORT=6100

Example

You can simulate the Isaac-Kitchen-CoffeeMake-G1-29DoF-Inspire-v0 environment by running the following command.

cd ./scripts/examples/
python env_example.py --task Isaac-Kitchen-CoffeeMake-G1-29DoF-Inspire-v0 --enable_cameras

Data Collection

Here is an explanation of how to set up and run the sample code for teleoperating the Unitree G1 robot.

1. Setup Unity on Your Windows PC

First, you need to set up Unity on your Windows PC. For detailed instructions, please refer to this guide.

2. Execute the Data Collection Code

After setting up Unity, follow these steps to run the data collection script.

a. Start the ROS Bridge

Open a separate terminal and run the following command to start the ROS bridge:

ros2 launch rosbridge_server rosbridge_websocket_launch.xml

b. Start the Unity ROS# Project

Open Unity and press the ▶ (Play) button at the top of the Unity screen.

c. Run the Data Collection Script

In your terminal, navigate to the scripts directory and execute the following command:

cd ./scripts/
python collect_dataset.py --task Isaac-BlockWorld-Stack-G1-29DoF-Dex3-v0 --enable_cameras --episode_index 0

Note

Ensure that all required dependencies are installed before running the script. Replace Isaac-BlockWorld-Stack-G1-29DoF-Dex3-v0 with your specific task name if needed.

Important

If you stop the simulation using Ctrl+C or a similar method, you must restart the ROS2 launch server to prevent high latency in ROS2 communication.

Teleoperation Interface Guide

The table below summarizes the teleoperation controls, including how different buttons and triggers on the VR controller interact with the robotic system.

---

Table: Teleoperation Controls for VR Interface

Button/Trigger	Function	Description
Main Button (Press Once)	Start Teleoperation	Starts teleoperation mode when pressed for the first time.
Main Button (Press Again)	Stop Teleoperation	Stops teleoperation and exits the mode.
Left Trigger	Left Gripper Control	Controls the left gripper’s opening and closing based on trigger pressure.
Right Trigger	Right Gripper Control	Controls the right gripper’s opening and closing based on trigger pressure.
Squeeze Left Button	Save Dataset	Saves the collected episode data to the Zarr dataset.
Squeeze Right Button	Reset Episode	Resets the episode without saving the data.
Hold Main Button (Long Press)	Exit the Simulation	Ends teleoperation and exits the simulation.

---

Additional Notes

• The environment will not start teleoperation until the main button is pressed.
• The dataset is only saved when the left squeeze button is pressed.
• If the right squeeze button is pressed, the episode resets without saving.
• The triggers allow precise gripper control by detecting gradual pressure.
• Pressing the main button repeatedly allows switching between starting and stopping teleoperation.

This table provides a structured guide to controlling the robot using the VR interface in Isaac Sim.
Let me know if you need additional modifications! 🚀

Train and Test Your Policy

Here is an explanation of how to train a policy using the datasets collected with the Unitree G1 robot.

1. Prepare the Dataset

Before starting the training, make sure to convert your dataset into the LeRobot format using the following code:

cd ./scripts/
python convert_dataset.py

Note

Before executing the above command, you must modify the video codec library from libsvtav1 to libx265 in /root/lerobot/lerobot/common/datasets/video_utils.py inside the Docker container. You can do this using a text editor or DevContainer.

2. Execute the Training Script

Once your dataset is ready, follow these steps to train your policy:

cd ./scripts/

# Diffusion Policy
python train_diffusion_policy.py

# Action Chunking with Transformers
python train_act_policy.py

# VQ-BeT
python train_vqbet_policy.py

# Pi0
python -c "from huggingface_hub import login; login()" #  Please enter your token.
python train_pi0_policy.py

Please run this if you want to train with multiple GPUs.

cd ./scripts/

# Diffusion Policy
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -m accelerate.commands.launch --num_processes=4 train_diffusion_policy_multi.py

# ACT
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -m accelerate.commands.launch --num_processes=4 train_act_policy_multi.py

# VQ-BeT
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -m accelerate.commands.launch --num_processes=4 train_vqbet_policy_multi.py

# Pi0
python -c "from huggingface_hub import login; login()" #  Please enter your token.
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -m accelerate.commands.launch --num_processes=4 train_pi0_policy_multi.py

3. Test Your Policy

After you have trained a policy (e.g., Diffusion Policy, ACT, etc.), you can evaluate its performance in the simulation environment.

cd ./scripts/

# Diffusion Policy
python inference_diffusion_policy.py --task Isaac-BlockWorld-Stack-G1-29DoF-Dex3-v0 --enable_cameras --episode_id latest

# Action Chunking with Transformers
python inference_act_policy.py --task Isaac-BlockWorld-Stack-G1-29DoF-Dex3-v0 --enable_cameras --episode_id latest

# VQ-BeT
python inference_vqbet_policy.py --task Isaac-BlockWorld-Stack-G1-29DoF-Dex3-v0 --enable_cameras --episode_id latest

# Pi0
python inference_pi0_policy.py --task Isaac-BlockWorld-Stack-G1-29DoF-Dex3-v0 --enable_cameras --episode_id latest

Future Extensions

• Improve training and inference scripts for stable ACT and Diffusion Policy.
• Integrate the MimicGen data generation pipeline.
• Add kitchen tasks.
• Add the Inspire-Hand model and a teleoperation interface for dexterous manipulation.
• Enable whole-body teleoperation using human pose estimation.