feat(policies): add NVIDIA gr00t N1.6 VLA policy

[AravindhShan-nv]

feat(policies): add GR00T N1.6 policy support in LeRobot

Type / Scope

• Type: Feature
• Scope: policies/gr00t_n1d6, policy factory, training pipeline, docs

Summary / Motivation

• This PR adds first-class gr00t_n1d6 support to LeRobot, including model/config/processor integration, policy factory wiring, and docs for installation, training, and evaluation. Original Isaac Groot Repo
• The goal is to enable GR00T N1.6 workflows in LeRobot while matching core N1.6 design elements and preserving compatibility with supported embodiments (SO100/SO101 and libero_panda).

Related issues

• Fixes / Closes: #
• Related: #

What changed

• Added a new policy package at src/lerobot/policies/gr00t_n1d6/:
  • policy config (configuration_gr00t_n1d6.py)
  • model + wrapper (gr00t_n1d6.py, modeling_gr00t_n1d6.py)
  • processor + collator + pre/post processor steps (processor_gr00t_n1d6.py)
  • N1.6 modules/utilities and Eagle3 backbone assets (modules/*, eagle3_model/*, utils.py)
• Updated policy registration and factory plumbing in src/lerobot/policies/factory.py:
  • recognize policy.type=gr00t_n1d6
  • build/load GR00T N1.6 pre/post processors
  • support processor config loading from pretrained checkpoints
  • compute and inject relative-action stats for supported embodiments during processor construction
• Updated training entrypoint in src/lerobot/scripts/lerobot_train.py to pass dataset context into processor creation (needed for relative-action statistics computation).
• Added GR00T N1.6 documentation in docs/source/gr00tn1d6.mdx:
  • dependency setup (flash-attn, torchcodec, albumentations, etc.)
  • train/eval examples for SO100 and Libero
  • embodiment extension notes and config pointers
• Added GR00T N1.6 tests:
  • tests/policies/groot/test_gr00t_n1d6_lerobot.py
  • tests/policies/groot/test_gr00t_n1d6_standalone.py
  • tests/policies/groot/test_gr00t_n1d6_state_dict.py
• Breaking changes: None.

How was this tested (or how to run locally)

• Tests added:
  • tests/policies/groot/test_gr00t_n1d6_lerobot.py
  • tests/policies/groot/test_gr00t_n1d6_standalone.py
  • tests/policies/groot/test_gr00t_n1d6_state_dict.py
• Manual checks / dataset runs performed.
  • Policy Trained on SO-100 Dataset (aravindhs-NV/pickpen-50
  • Visualize Eval Dataset -
    aravindhs-NV/eval_pickpens50_0220_ah8

file-000_3min.mp4

• Libero Evaluation Tests were run

Benchmark	LeRobot N1.6 Implementation	GR00T N1.6 Reference
Libero Spatial	95.5%	97.65%
Libero Object	100.0%	98.45%
Libero Goal	96.5%	97.5%
Libero 10	90.5%	94.35%

• Instructions for the reviewer

SO100 Training Command

DATASET_REPO_ID=<TRAINING_DATASET_REPO_ID>  # change dataset if needed
OUTPUT_DIR=<CHECKPOINT_OUTPUT_DIR>  # change output path
POLICY_REPO_ID=<POLICY_REPO_ID>  # change HF repo for checkpoints

nohup lerobot-train \
    --policy.type=gr00t_n1d6 \
    --dataset.repo_id=$DATASET_REPO_ID \
    --batch_size=32 \
    --steps=10000 \
    --log_freq=100 \
    --save_freq=5000 \
    --output_dir=$OUTPUT_DIR \
    --policy.repo_id=$POLICY_REPO_ID \
    > so100_train_10k.log 2>&1 &

SO100 Real Robot Eval

TELEOP_PORT=/dev/ttyUSB0  # change leader serial port
TELEOP_ID=leader_arm  # change leader arm ID
ROBOT_PORT=/dev/ttyUSB1  # change follower serial port
ROBOT_ID=follower_arm  # change follower arm ID
CAMERAS_JSON='{"wrist": {"type": "opencv", "index_or_path": 4, "width": 640, "height": 480, "fps": 30}, "front": {"type": "opencv", "index_or_path": 6, "width": 640, "height": 480, "fps": 30}}'  # update camera indices/paths
TASK_TEXT="Pick up the vial and place it in the yellow rack"  # update for your task
DATASET_REPO_ID=<EVAL_DATASET_REPO_ID>  # change eval dataset repo
POLICY_PATH=<TRAINED_POLICY_REPO_ID>  # change policy checkpoint/repo

lerobot-record \
  --teleop.type=so101_leader \
  --teleop.port=$TELEOP_PORT \
  --teleop.id=$TELEOP_ID \
  --robot.type=so101_follower \
  --robot.port=$ROBOT_PORT \
  --robot.id=$ROBOT_ID \
  --robot.cameras="$CAMERAS_JSON" \
  --dataset.single_task="$TASK_TEXT" \
  --dataset.repo_id=$DATASET_REPO_ID \
  --dataset.episode_time_s=30 \
  --dataset.num_episodes=3 \
  --policy.path=$POLICY_PATH \
  --policy.n_action_steps=8 \
  --dataset.push_to_hub=true

Checklist (required before merge)

• Linting/formatting run (pre-commit run -a)
• All tests pass locally (pytest)
• Documentation updated
• CI is green

Reviewer notes

• Important change in common file: src/lerobot/utils/control_utils.py (99-128) — this PR adds a GR00T N1.6-specific fast inference path that reuses cached preprocessed batches when the action queue is non-empty (bypassing repeated preprocessing). Please verify correctness, cache lifecycle/reset behavior, and that this path is only active for gr00t_n1d6.
• Please focus review on src/lerobot/policies/gr00t_n1d6/processor_gr00t_n1d6.py (state/action normalization, relative-action handling, processor state load/save, pre/post integration).
• Please focus review on src/lerobot/policies/factory.py (processor loading path for pretrained checkpoints and relative-action stats flow).
• Please focus review on src/lerobot/policies/gr00t_n1d6/utils.py embodiment modality/stat mappings (MODALITY_CONFIGS, EMBODIMENT_STAT_CONFIGS).
• Verify docs and commands in docs/source/gr00tn1d6.mdx match expected runtime dependencies (flash-attn, torchcodec, albumentations) and supported embodiments.
• Anyone in the community is free to review the PR.