Lerobot SO-101 Teleop Isaac Lab environment

lerobot_so101_teleop.mp4

example.webm

Sample Environment for the LeRobot SO-101 Robot in Isaac Lab to collect demonstrations in a simulation. This can be utilized to later procedurally scale up datasets using various methods of domain randomization and style transfer techniques ✨ 🤖

Episoded are recorded directly to Lerobot Dataset, and can quickly be uploaded to HuggingFace Hub 🚀

Installation

• Create a Conda environment

  conda create -y -n lerobot-isaac-lab python=3.11

• In your the new environment, Install the following:

  conda install ffmpeg=7.1.1 -c conda-forge
  
  pip install uv
  
  # install Isaac Sim
  uv pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com
  # run isaac sim and accept the EULA
  isaacsim
  
  # Install Isaac Lab
  git clone https://github.com/isaac-sim/IsaacLab.git
  cd IsaacLab
  git checkout v2.3.0
  ./isaaclab.sh -i
  cd ..
  
  # Install this repo!
  git clone https://github.com/liorbenhorin/lerobot_so101_teleop.git
  cd lerobot_so101_teleop
  git lfs fetch # assets
  uv pip install -e source/lerobot_so101_teleop
  
  # Fix numpy dependency
  uv pip install numpy==1.26.0 lxml==4.9.4 packaging==23.2

• Make sure your Lerobot Leader arm has been calibrated.

• Edit source/lerobot_so101_teleop/scripts/lerobot_agent.py by changing lerobot_cfg to match your port and id of your setup. (You have this information from the calibration step)

• Verify that the extension is correctly installed by listing the available tasks:

  list_envs

  You should see a several environments named like Lerobot-So101-Teleop-*

• Run the environment

  lerobot_agent --task Lerobot-So101-Teleop-Rock-A-Stack-Simple

• Get familiar with the teleop feeling

Record a dataset

• Run the environment and include dataset repo-id and repo-root (They will get created if not already exists)

  lerobot_agent --task Lerobot-So101-Teleop-Rock-A-Stack-Hard \
  --repo_id ${HF_USER}/so101_teleop \
  --repo_root $(pwd)/datasets/so101_teleop \
  --task_name "Pick up the yellow ring and put it on the pole"

  • Click S to start/stop the recording.
  • Click C to cancel current recording (Very useful!).
  • Reset the environment R will also stop the recording.
  • Episodes are queued for processing, while you work.
  • When you are done recodring, look for this massage in the console: [INFO]: No more episodes in queue. Stopping processor thread...
  • Exit the simulation with Ctrl+C.

Playback & Upload

• To playback dataset episodes, use lerobot rerun visualizer

  lerobot-dataset-viz \
  --repo-id ${HF_USER}/so101_teleop \
  --root $(pwd)/datasets/so101_teleop \
  --episode-index 0 # or other episode

• To push your dataset to HuggingFace Hub (Optional)

  lerobot_push_dataset \
  --repo-id hf-repo-id/so101_teleop \
  --root $(pwd)/datasets/so101_teleop \
  --tags robotics teleop rock-a-stack \ # separate by spaces

Evaluate policy

We assume you already trained a model based on the data collected (either in simulation or in real) and have a model ready to use

    lerobot_eval \
    --task Lerobot-So101-Teleop-Rock-A-Stack-Hard-Eval \
    --policy_path ${HF_USER}/your_policy

If your policy has differnt camera names, you can also provide a mapping. The key is the simulated camera name and the value is the policy camera name.

    --rename_map '{"ego":"front", "external":"top"}'

Creating new Tasks

To learn how to add new tasks, follow this guide

Contributors

• Thank you LycheeAI for making the SO101 arm available in USD format 💚 https://github.com/MuammerBay/so-arm101-ros2-bridge/tree/main/IsaacSim_USD

• Contributions are welcome via pull requests

Limitations

• Simulated teleoperation can be hard - but training makes perfect 🏅