Expected zero shot performance on custom robot

[pepisg]

Hi! Thanks a lot for open-sourcing this awesome work.

I’ve been testing OmniVLA on a custom wheeled robot in outdoor environments, and I’m seeing mixed performance, as shown in the videos below (video speed is 3x):

vokoscreenNG-2026-01-15_10-54-24.mp4

acceptable performance

vokoscreenNG-2026-01-15_10-53-56.mp4

bad performance

The green line is generated from OmniVLA’s output waypoints, scaled by a metric resolution of 0.2 m. I tried both the satellite and prompt modalities, and the behavior is very similar.

I’m running inference on a desktop RTX 5090. The forward pass takes ~100 ms, but I’m only receiving images at 4 Hz. I also tried all released checkpoints (omnivla-original, omnivla-original-balance, omnivla-finetuned-cast) with similar results.

For inference, I wrote a ROS wrapper around the official script:

https://github.com/NHirose/OmniVLA/blob/main/inference/run_omnivla.py

The wrapper fills in the robot state and produces a cmd_vel output, while keeping the core inference code unchanged. I limited the command speeds to:

linear velocity ≤ 0.3 m/s

angular velocity ≤ 0.75 rad/s

Here is a sample video where the model fails to keep the robot on the sidewalk. The prompt is: "navigate on the center of the sidewalk".

camera_color_image_raw.mp4

I was wondering:

• What's the expected zero shot performance of the OmniVLA on unseen robots?
• What's the amount of data you think it would be needed for fine tuning the model to reach the same performance as the one you get in the training platforms?
• What's the best format to store the dataset for finetuning?

Thanks!

[NHirose]

Thank you for sharing your results with us.
As we mentioned in our paper, there are space to improve the performance especially in the language-conditioned navigation. Specifically navigation with out-of-distribution (OOD) language prompt such as "navigate on the center of the sidewalk" is still challenging task for OmniVLA.

But I think that the pose-conditioned navigation does work well in our cases. And our policy nicely performs on not only the wheeled robot but also the quadruped robot. If your videos are the pose-conditioned navigation, we may be able to debug.

I am not certain the required amount of data for fine-tuning. It depends on your task and evaluation setup. But I believe that the additional curated dataset helps a lot. When I tried to use the CAST dataset, the CAST dataset nicely helps to improve the performance for the OOD language prompt.

I gave the sample dataloader in our repository. I think that you can use it. But you can use any data format if you write the dataloader by yourself.

[NHirose]

In addition, can you try the wide FOV camera? It may be able to help a bit.

[pepisg]

Hi!

Thanks a lot for your response and clarifications. Did a couple more experiments:

1. Tried the pose conditioned navigation with the same camera and also with a fisheye camera, and found that the robot stll struggles to keep on the sidewalk. On the video, inference is run on the fisheye camera above but I still kept the original camera for viewing the path projection. I tried the same setup with the camera on the bottom and observed similar results

omnivla.mp4

2. I noted that when being further from the goal on the pose conditioned modality, the model's predicted waypoints seem to fall in a smaller space span closer together, and the output path does not seem to start on the robot's position. This does not happen when using satellite conditioned modality.

I will try to produce a small test datset with the images, robot and goal pose, and other model inputs to more easily reproduce the output I'm observing.

I'm also going to try to finetune on my data and come back with the results.

Thanks again!

[NHirose]

Sorry for my late response.

Hmm... it is weird for me. I have never seen this kind of outputs in my experiments.
Could you try my sample inference code?? In my sample current image, I think that you can not see the above behaviors. If you see the similar behavior in my sample inference code, I think that the conda environment may be weird. (Maybe...)

If you can not see the above behaviors in my sample code, please replace the current image with yours. I think that my sample inference code can help you debug. Thank you.