Discussion: Unexpected performance difference between SmolVLA and Pi0.5 with the same real robot dataset

[skkkkk0609]

Unexpected real robot performance difference between SmolVLA and Pi0.5

Hi everyone,

I recently conducted a real robot deployment comparison between SmolVLA and Pi0.5 using exactly the same dataset, robot platform, and task description.

Video:
https://youtu.be/RMFLhEMqKlE?si=ExSwCahcE2FY30mR

I observed an interesting result:

The full-parameter SmolVLA model achieved better real robot performance than the full-parameter Pi0.5 model, even though both models were trained on the same imperfect dataset.

I would like to understand the possible reasons behind this phenomenon.

---

Robot setup

Robot:

• REALMAN NERO dual-arm robot

Task:

• Left arm places the box at the target position.
• Right arm places the black sponge into the box.

Task instruction:
Put the black sponge into the box

---

Dataset

The same dataset was used for all experiments.

Dataset details:

• 50 episodes
• Real robot teleoperation data
• Action chunk size: 50

The dataset is not perfect.

One possible data quality issue is that after the right arm grasps the sponge, the following actions are performed too continuously:

• Moving to the box
• Adjusting the position above the box
• Releasing the sponge

There are not enough intermediate pauses or stable visual observations between these stages.

Because of this, the observation-action alignment may not be ideal, and the model may need to learn from demonstrations where different manipulation stages are not clearly separated.

However, despite this imperfect dataset, SmolVLA full fine-tuning was still able to achieve reliable execution.

---

Training settings

SmolVLA

Full fine-tuning:

• Parameters: ~450M
• Backbone: not frozen
• Batch size: 32
• Training steps: 20k

Action expert:

• Backbone frozen

Pi0.5

Full fine-tuning:

• Parameters: ~3B+
• Backbone: not frozen
• Batch size: 64
• Training steps: 10k

Action expert:

• Backbone frozen

All models reached stable training loss.

Interestingly:

• Pi0.5 full fine-tuning achieved the lowest training loss.
• However, the lowest training loss did not lead to the best real robot performance.

---

Deployment observations

SmolVLA full fine-tuning

Results:

• Best performance among all tested models.
• Stable execution.
• Successfully completed the task multiple times.
• Could recover better when the scene was slightly disturbed.

SmolVLA action expert

Results:

• Performance dropped significantly compared with full fine-tuning.
• The model seemed to learn only a rough action pattern.

Pi0.5 full fine-tuning

Results:

• Although the training loss was very low, deployment performance was worse.
• When the scene was disturbed, the predicted actions sometimes became inconsistent with the visual input.
• The robot sometimes appeared to follow previously learned action sequences instead of adapting to the current observation.

Pi0.5 action expert

Results:

• Almost failed on this task.

---

Additional language generalization test

I also tested the models with modified language instructions.

Results:

• Pi0.5 full fine-tuning showed much stronger language generalization ability.
• SmolVLA action expert almost completely failed.

Therefore, I do not think Pi0.5 has weaker capability overall. Instead, the result may be related to how different VLA architectures utilize imperfect real robot demonstrations.

---

Questions

I would like to understand:

1. Why can a smaller model (SmolVLA) achieve better real robot performance than a larger model (Pi0.5) on the same imperfect dataset?

2. Could this be related to:

• Different VLA architectures?
• Different visual encoders?
• Different action prediction mechanisms?
• Different pretraining strategies?
• Different sensitivity to noisy observation-action alignment?

3. Does a larger VLA model require higher-quality demonstrations to fully utilize its capacity?

4. Could a smaller model sometimes perform better on specific robot tasks because it is less sensitive to imperfect demonstrations?

5. Are there recommended methods to analyze whether a VLA model is learning visual grounding or simply memorizing action sequences?

I would appreciate any insights or suggestions.

Thanks!