Questions regarding Bench2Drive-VL-baselines-Table3 Dev10

[Xiaolong-RRL]

Dear Authors~

Thanks for your great work!

In this document [https://thinklab-sjtu.github.io/Bench2Drive-VL/docs/references/baselines#data-analysis], , the Dev10 data access method used by Table3 is not provided, making it impossible to compare it with the methods in Table3.

Therefore, I would like to ask if you could provide the access method for Dev10.

Looking forward to your reply~

Best regards!

[MeteorCollector]

Hi, Thank you for your interest in our project~

Actually, both the closed-loop performance in Table 2 and the VQA performance in Table 3 are collected from the same run on the Dev10 scenarios. They just focus on different aspects.

For Table 3, we evaluated the answers to various questions given by the VLM while it was controlling the ego vehicle in the Dev10 scenarios. Most of the scores were obtained by comparing the VLM’s answers with the ground-truth answers from the expert model using an LLM-based scoring method. For several key questions, we designed stricter scoring criteria, which are described in detail here.

Meanwhile, the closed-loop performance of the VLM-controlled agent in the same Dev10 run was summarized into Table 2.

Therefore, the two tables are not really meant to be directly comparable, since they target different aspects—Table 2 emphasizes planning, while Table 3 emphasizes reasoning.

[Xiaolong-RRL]

Thanks for your kindly reply~

I still have three questions:

1. Is the data in Dev10 routes from this link? https://github.com/Thinklab-SJTU/Bench2Drive/blob/main/docs/bench2drive_mini_10.json

2. Were the models Qwen2.4-VL, Gemma3, and InternVL3 used in the evaluation not trained on the bench2drive-base training set, but instead directly used for zero-shot inference?

3. How is the use of COT controlled in the evaluation code? And what is the format of COT?

[MeteorCollector]

1. No, Dev10 is from DriveTranformer. In Bench2Drive repo, it's here. It is also in this repo: drivetransformer_bench2drive_dev10.xml

2. Yes, they're not trained. We directly used official checkpoints.

3. They can be found here. Each question has a QID, and they are used in configuration. For example, in Table3, CoT-style evaluation framework asks the VLM questions, say, 1 -> 2 -> 3 -> 4 -> 5 and their contexts are inherited. Non-CoT ones also answer 1, 2, 3, 4, 5 but the contexts are not inherited.