Training Data 10Hz vs CARLA Simulation 20Hz

[sshouhua]

Hi, thanks for the great work on Bench2Drive!

I have a question regarding the temporal frequency mismatch between training data and the benchmark environment.
As far as I understand, the training dataset is recorded at 10 Hz, while the CARLA benchmark simulation runs at 20 Hz by default (as seen in the evaluation settings).

My question is:

Would this frequency difference cause any inconsistency or potential problems during evaluation?

For example:

• Are observations duplicated/interpolated internally?
• Does the agent receive 20Hz simulation steps but perception is only updated at 10Hz?

Any clarification on the intended design assumptions or recommended settings would be very helpful.

Thanks again for the fantastic benchmark and simulator integration!

[PDillis]

I'm not an author, but I agree that this is a mismatch. Note two things: one is that all benchmarks fix the frequency at 20 Hz (NoCrash, Longest6, Bench2Drive, etc.), and the second is that this lets big models run at a higher frequency than they should in real life. Take DriveTransformer: their calculated latency in an A6000 for the Large variant is 211.7 ms or a frequency of 4.72 Hz. If we standardize the hardware, then most likely than not all models should perform worse due to this gap. IMO, more should be look into this.