Evaluating in the closed-loop simulator HUGSIM

[TheNeeloy]

Hi there! I'm trying to evaluate Alpamayo in HUGSIM: https://github.com/hyzhou404/HUGSIM
Would anyone else have any experience with the same?

I've updated the camera intrinsics and extrinsics of the simulator to match the expectations of Alpamayo, and I feed the pose of the vehicle and the images to the model to sample a trajectory with high probability and low temperature. Usually, I find the model starts by going straight, and then it inevitably turns right.

One potential issue is there may be a mismatch between the frame conventions that the model expects to receive vs. what the simulator expects for the predicted trajectory. I believe that Alpamyo expects the pose to be x forward, y left, and z up. And hugsim expects the predicted trajectory to be x right, y forward, and z up.

Another issue could be that the frequency of the simulator and the model predictions do not align. I have set it so that the simulator expects the predicted trajectory to be at 10Hz, but even so, when I feed the trajectory to the simulator, the vehicle moves very slowly. As such, I have been feeding the simulator every 8th waypoint, and it moves faster.

In the videos below, I've projected the predicted trajectory into the front view with the camera intrinsics (after sampling every 8th point).

https://github.com/user-attachments/assets/73274397-b1cf-4a83-b4d7-0edebab2e92b
https://github.com/user-attachments/assets/3354d7ba-c524-48d2-8bba-45b9d1a1f376
https://github.com/user-attachments/assets/b6750548-18ae-4f5d-80f4-b7100b883bea
https://github.com/user-attachments/assets/f67b35a2-8029-4c5a-8a34-9d6d6d540075

[TheNeeloy]

I also tried the model in Alpasim recently and have found that it can ignore the instruction goal and instead merge unsafely at times. For example, in the video below, the ego car is told to go straight but it weaves right and left, while colliding into surrounding vehicles. I'm assuming that the RL post-training assists with these issues.

clipgt-clipgt-09a95ffa-37b7-486d-9404-ac8cf1c3e045_0_ed509e04-20d4-11f1-9e9c-f5e669405d6f_camera_front_wide_120fov_default.mp4

[bhanu-iav]

@TheNeeloy Hey Neeloy, nice example. I have a few questions.

• "the ego car is told to go straight" -> you mean from COT?
• why HUGSIM and not Alpasim?

[TheNeeloy]

Hi!

1. I was referring to the command visualized at the top left of the video rendered in Alpasim (eg. straight, left, right), but I realize that Alpamayo ignores that command. [Alpamayo 1 Critical VLA Alignment Failure: Reasoning & Trajectory Heads Decoupled alpasim#20 (comment)
2. I am attempting to also incorporate perturbations into the simulator sensors and control other vehicles to generate safety critical scenarios where a model fails. These tasks are easier to implement in HUGSIM at the moment with how the simulator was built, but as discussed in this linked issue, the Alpasim team is adding features that would make this easier in the future. Stress testing policies in Alpasim alpasim#47 Also, I have been able to get other policies working in HUGSIM like DrivoR, so it would be nice to be able to compare them in the same environment. I believe Alpasim has more realistic renderings and a physics engine that tops the simpler approach used in HUGSIM.

[maximilianigl]

This is for AlpaSim, not alpamayo:
If you have feedback on the new architecture and in particular on how we could make it simpler to add new features, please let us know. Our goal is to make extending AlpaSim as simple as possible, so we're happy for suggestions.

One thing we added recently is the plugin system. It's currently only used in a few places, but if there are additional places that would be helpful to use it, let us know.