Groot freezing and Pi0.5 accuracy issues (v0.5.0)

[SlenderMongoose]

Ticket Type

🐛 Bug Report (Something isn't working)

Environment & System Info

- LeRobot version: 0.5.0
- Platform: Linux-6.17.0-14-generic-x86_64-with-glibc2.39
- Python version: 3.12.12
- Huggingface Hub version: 1.6.0
- Datasets version: 4.7.0
- Numpy version: 2.2.6
- FFmpeg version: 7.1.1
- PyTorch version: 2.10.0+cu128
- Is PyTorch built with CUDA support?: True
- Cuda version: 12.8
- GPU model: NVIDIA GeForce RTX 4090
- Using GPU in script?: <fill in>
- lerobot scripts: ['lerobot-calibrate', 'lerobot-dataset-viz', 'lerobot-edit-dataset', 'lerobot-eval', 'lerobot-find-cameras', 'lerobot-find-joint-limits', 'lerobot-find-port', 'lerobot-imgtransform-viz', 'lerobot-info', 'lerobot-record', 'lerobot-replay', 'lerobot-setup-can', 'lerobot-setup-motors', 'lerobot-teleoperate', 'lerobot-train', 'lerobot-train-tokenizer']

Description

I’ve been testing v0.5.0 (libero scenario) and encountered two specific issues:

Groot freezing: The process stalls indefinitely during evaluation.

Pi0.5 accuracy: Output values are lower than expected.

Context & Reproduction

for groot:

--policy.path=liorbenhorin-nv/groot-libero_spatial-128_20000
--env.type=libero
--env.task=libero_spatial
--eval.batch_size=1
--eval.n_episodes=10
--policy.n_action_steps=50
--env.max_parallel_tasks=1
--output_dir=./evals/groot_libero_spatial

for pi0.5: (only got 0.88, should be ~0.98)

--policy.path=lerobot/pi05_libero_finetuned_v044
--env.type=libero
--env.task=libero_spatial
--eval.batch_size=1
--eval.n_episodes=10
--policy.n_action_steps=50
--env.max_parallel_tasks=1
--output_dir=./evals/pi05_libero_spatial

Relevant logs or stack trace

Traceback (most recent call last):
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/scripts/lerobot_eval.py", line 813, in <module>
    main()
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/scripts/lerobot_eval.py", line 809, in main
    eval_main()
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/configs/parser.py", line 233, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/scripts/lerobot_eval.py", line 528, in eval_main
    policy = make_policy(
             ^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/factory.py", line 491, in make_policy
    policy = policy_cls.from_pretrained(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/groot/modeling_groot.py", line 175, in from_pretrained
    return super().from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/pretrained.py", line 107, in from_pretrained
    instance = cls(config, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/groot/modeling_groot.py", line 66, in __init__
    self._groot_model = self._create_groot_model()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/groot/modeling_groot.py", line 82, in _create_groot_model
    model = GR00TN15.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/groot/groot_n1.py", line 368, in from_pretrained
    pretrained_model = super().from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swm/SWM_Environments/envs/lerobot-055/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4094, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/groot/groot_n1.py", line 218, in __init__
    self.action_head = FlowmatchingActionHead(action_head_cfg)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/workspace_2t/SWM_Academic/_None_Paper_Project/lerobot/lerobot-0.5.0/src/lerobot/policies/groot/action_head/flow_matching_action_head.py", line 208, in __init__
    self.beta_dist = Beta(config.noise_beta_alpha, config.noise_beta_beta)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swm/SWM_Environments/envs/lerobot-055/lib/python3.12/site-packages/torch/distributions/beta.py", line 59, in __init__
    self._dirichlet = Dirichlet(
                      ^^^^^^^^^^
  File "/home/swm/SWM_Environments/envs/lerobot-055/lib/python3.12/site-packages/torch/distributions/dirichlet.py", line 73, in __init__
    super().__init__(batch_shape, event_shape, validate_args=validate_args)
  File "/home/swm/SWM_Environments/envs/lerobot-055/lib/python3.12/site-packages/torch/distributions/distribution.py", line 77, in __init__
    if not torch._is_all_true(valid):
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swm/SWM_Environments/envs/lerobot-055/lib/python3.12/site-packages/torch/utils/_device.py", line 109, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/swm/SWM_Environments/envs/lerobot-055/lib/python3.12/site-packages/torch/_meta_registrations.py", line 7779, in meta_local_scalar_dense
    raise RuntimeError("Tensor.item() cannot be called on meta tensors")

Checklist

• I have searched existing tickets to ensure this isn't a duplicate.
• I am using the latest version of the main branch.
• I have verified this is not an environment-specific problem.

Additional Info / Workarounds

No response

[wangerforcs]

Maybe you should try setting pi0.5's n_action_steps to 5 or 10 instead of 50.

[SlenderMongoose]

Maybe you should try setting pi0.5's n_action_steps to 5 or 10 instead of 50.

Thanks for the suggestions! The numbers have improved, though they still haven't quite reached the original performance levels. I’m comfortable with the current state of pi0.5, but I’m having significant trouble with gr00t—it seems to be broken after the latest update.

I’ve tried modifying the code related to the beta distribution to fix this, but it consistently triggers a flood of errors. This wasn't an issue in version 0.4.4. Do you have any insights or a potential workaround for this?

[CarolinePascal]

Hi @SlenderMongoose,

Thank you for reporting on the v0.5.0 right after the release ! Regarding your observations on Libero with pi0.5, this is indeed a side effect of the latest release.
In v0.5.0, we had to fix two issues :

• Removing the padding of images stored as float32 in [0,1] before applying the signup norm;
• Removing the padding of the state in the pi0.5 processor.

As a consequence, the checkpoints and fine-tuned weights obtained in v.04.4 will no longer work as expected in v.0.5.0. New checkpoints are available here : https://huggingface.co/lerobot/pi05-libero and legacy checkpoints are still available here : https://huggingface.co/lerobot/pi0fast-libero-v044. You will likely need to fine-tune again to reach the same performances.

Regarding Gr00t, I'll have a look at our implementation asap !

Best,

Caroline