[DOC] Evaluation Configuration of Alpamayo 1.5 on AlpaSim

[sangohkim]

Report needed documentation

First of all, thank you for open-sourcing this great project.

Report needed documentation
The Alpamayo-1.5-10B model card on Hugging Face (https://huggingface.co/nvidia/Alpamayo-1.5-10B) reports closed-loop performance using the AlpaSim score (1.37 ± 0.10 on 913 scenarios), but the evaluation setup behind the reported numbers is not documented. In particular, the number of rollouts per scene and the exact metric definition are unclear, which makes it difficult to reproduce the reported results or compare against them in a consistent manner.

Describe the documentation you'd like
It would be very helpful if the evaluation setup for the reported AlpaSim scores could be documented, specifically:

• Number of rollouts per scene: When measuring the AlpaSim performance reported on the model card, how many rollouts were performed per scene? If multiple rollouts were used, how were the reported numbers aggregated (e.g., mean across rollouts)?
• Metric definition: In the Alpamayo-R1 paper, the AlpaSim score is defined as the average distance driven between events, where events correspond to offroad or close encounter occurrences. However, the public AlpaSim github codebase does not appear to implement close encounter detection; instead, it implements the avg_dist_between_incidents_at_fault metric based on offroad and collision events. Is the AlpaSim score reported on the model card the avg_dist_between_incidents_at_fault value as computed by the public codebase?

Steps taken to search for needed documentation

• Reviewed the Alpamayo-1.5-10B model card on Hugging Face, which links to the AlpaSim repository but does not specify the evaluation configuration or metric definition used for the reported scores.
• Reviewed the AlpaSim metric definitions in the Alpamayo-R1 paper (Sec. 6.1), which define the AlpaSim score based on offroad and close encounter events.
• Inspected the public AlpaSim codebase and found that close encounter detection is not implemented; the composite metric implemented is avg_dist_between_incidents_at_fault, based on offroad and collision events.

Thank you in advance :)

[super-anova]

Hi, thanks for reaching out! See the clarification below. Let us know if anything else is unclear.

• We had 3 rollouts per clip/scene. The reported metrics are mean +/- std of the 3 rollouts.
• Yes, avg_dist_between_incidents_at_fault is what was reported.

[sangohkim]

Thank you very much :)