Investigate available models that can be run to get actions for the lekiwi

[francocipollone]

Summary

Eventually, we will want to use a model to retrieve actions.

• These actions can be directly joint information or (most probably according our discussions) trajectories for both the arm and the mobile base.

Fine tunning is not necessary but eventually could be a plus if we find this is necessary.

Models

Hugging face/lerobot provides several models already that can be tested: like smolvla

There are also other like: https://robo-point.github.io/

Opening the investigation to more massive models like isaac groot would be nice

[olmerg]

I just add this proposal that I was creating about serving VLAs where can be easier to gather links . VLA proposals

[francocipollone]

SmolVLA

https://huggingface.co/blog/smolvla

Can be tested and deployed on very affordable hardware (SO-100, SO-101, LeKiwi, etc.)

I have my doubts about how easy is to use in the lekiwi.

Trained on SO ARM 100 datasets (5DoF arms).
We could use the model as it is (for only the arm); however, fine-tuning is recommended.

Naturally, the action space will be different from the one we need for the lekiwi, so we would need:

• Replace the action head with the one that matches the lekiwi action space.
• Collect dataset for our mobile base + arm.
• Fine-tuning.

It would be nice to test this in a Lekiwi, as I haven't found info on the community doing so yet. We will have the sim for easy recording of datasets.