Our project aims to apply hand tracking to the robot teleoperation task by translating physical hand positions to a simulated Panda robot arm in Robosuite. This project was primarily tested on the lift task in robosuite but due to the nature of teleoperation other tasks could be successfully completed without additional adaptation.
Note
Mileage may vary with different package versions. We highly recommended venv or conda virtual environments. The packages listed below are the versions we used and are not representative of all verions that may work.
| Package | Version |
|---|---|
| Python | 3.10.13 |
| Mediapipe | 0.10.21 |
| Robosuite | 1.5.1 |
| Mujoco | 3.3.0 |
| OpenCV | 4.11.0.86 |
| Numpy | 1.26.4 |
| FilterPy | 1.4.5 |
- Create a new conda environment
$ conda create -n teleoperate python=3.10.13- Activate environment
$ conda activate teleoperate- Clone the Repository
$ git clone https://github.com/charlenehuang1/CS188FinalProject.git
$ cd CS188FinalProject- Install Required Packages
$ pip install -r requirements.txt- Begin Hand Tracking
$ python handtracking.pyWarning
In order for the socket connection to be established correctly you must wait for the handtracking script to fully initialize opencv and mediapipe. This can be seen by the message INFO: Created TensorFlow Lite XNNPACK delegate for CPU. or something similar appearing in the console.
- Begin Robosuite Simulation
On Windows and Linux, run:
$ python robosuite_test.pyOn macOS, due to Mujoco requirements, run:
$ mjpython robosuite_test.pyWarning
The initial position of your hand when this script initalizes is used as a reference to compute displacement from the original position so we highly recommend to place your hand somewhere close to the middle of the camera view as possible to allow for as much movement in x, y, z as possible