[Question] Collecting Recovery Demonstrations for Fine-Tuning GR00T

[edisimo]

Collecting Failed / Recovery Demonstrations for Fine-Tuning GR00T

Hi everyone,

First of all, thank you for the great work you’re doing with Isaac Lab and GR00T — it’s been really exciting to experiment with.

I’ve been fine-tuning GR00T on my own dataset of a G1 robot collected with AVP, and I noticed something: when an object (for example, one held in hand) accidentally falls, the model keeps trying to complete the task without the object instead of recovering or retrying.

I was therefore wondering if it makes sense to also include failed demonstrations, or even better, recovery demonstrations (the object slips at first, then I regrasp it) into the dataset for finetuning GR00T ?

If so, could anyone please point me to the direction of how many recovery examples you would recommend collecting to meaningfully improve the model’s behavior?

I was also wondering, will I be able to use these demonstrations in connection with GR00T-Mimic for synthetic dataset generation, or do you think some issues will arise?

[RandomOakForest]

Thank you for posting this. This question is great for our Discussions section. I'll move the post there. I think probably it makes better sense to reset after the object is no longer in play for the robot, as there's nothing to learn in subsequent frames. Sensors should help.