Skip to content

README.md

Latest commit

 

History

History
94 lines (70 loc) · 3.6 KB

README.md

File metadata and controls

94 lines (70 loc) · 3.6 KB
Raw

Finetuning SO100 Model

This guide shows how to finetune dataset collected from SO100 robot, and evaluate the model on the real robot.

Dataset

To collect the dataset via teleoperation, please refer to the official documentation in lerobot: https://huggingface.co/docs/lerobot/il_robots?teleoperate_so101=Command

Dataset Path: izuluaga/finish_sandwich

Visualize it with this link

Handling the dataset

uv run --project scripts/lerobot_conversion \
  python scripts/lerobot_conversion/convert_v3_to_v2.py \
  --repo-id izuluaga/finish_sandwich \
  --root examples/SO100/finish_sandwich_lerobot \
  --in-place

--in-place replaces the downloaded v3.0 dataset with the converted v2.1 one, so the paths below stay the same. The original v3.0 dataset is kept in a sibling finish_sandwich_backup_v3.0 directory.

Then copy the modality.json file into the dataset's meta/ directory:

cp examples/SO100/modality.json examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich/meta/modality.json

Finetuning

Run the shared finetune launcher directly, using absolute joint positions (feel free to experiment with relative positions):

CUDA_VISIBLE_DEVICES=0 NUM_GPUS=1 uv run bash examples/finetune.sh \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich \
  --modality-config-path examples/SO100/so100_config.py \
  --embodiment-tag NEW_EMBODIMENT \
  --output-dir /tmp/so100_finetune

Open-Loop Evaluation

Evaluate the finetuned model with the following command:

uv run python gr00t/eval/open_loop_eval.py \
  --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich/ \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --traj-ids 0 \
  --execution-horizon 16 \
  --steps 400

Evaluation Results

The evaluation produces visualizations comparing predicted actions against ground truth trajectories:

Open-loop evaluation results showing predicted vs ground truth trajectories

To read these numbers and decide whether your fine-tune is working, see Interpreting the Result: Is My Fine-tune Working?.

Closed-Loop Evaluation

Please refer to eval_so100.py for how to write SO100 deployment code using Policy API.

  1. set up client side deps
cd gr00t/eval/real_robot/SO100
uv venv
source .venv/bin/activate
uv pip install -e . --verbose
uv pip install --no-deps -e ../../../../
  1. Start policy server from the repository root in a separate terminal:
uv run python gr00t/eval/run_gr00t_server.py \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --embodiment-tag NEW_EMBODIMENT
  1. Run the eval script as the client from the gr00t/eval/real_robot/SO100 environment created above:
uv run python eval_so100.py \
  --robot.type=so101_follower --robot.port=/dev/ttyACM2 \
  --robot.id=orange_follower \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}, front: {type: opencv, index_or_path: 6, width: 640, height: 480, fps: 30}}" \
  --policy-host=localhost --policy-port=5555 --lang-instruction="finish the ham cheese olives sandwich"