GenEval score discrepancy on VGT (Qwen2.5VL): 0.80 vs reported 0.83

[DingShizhe]

I tested VGT (Qwen2.5VL) on GenEval using the standard generation settings (cfg_scale = 4.5, num_steps = 30, acc_ratio = 1, seed = 42). On AMD MI325x with ROCm 6.2, the best score I can get is around 0.80, and I’m not able to reproduce the reported 0.83.

I’m wondering whether the 0.83 result can be reproduced on NVIDIA GPUs (CUDA) with the same settings, or if it requires some special setup such as a particular seed, multi-seed averaging, or other generation/inference tweaks. Also curious whether there is any known difference between ROCm and CUDA that could affect this result.

Thanks!

[gjhhust]

Thank you for following VGT! You can try setting cfg_scale to 7.5！

[DingShizhe]

Thanks! Increasing cfg_scale to 7.5 does improve the result — I can reach about 0.807 on GenEval — but there’s still some gap to 0.83.

Are there any other parameters or evaluation details I should pay attention to (e.g. steps, seeds, or inference settings)?

[gjhhust]

accelerate launch --config_file "$ACCEL_CFG" \
        "$ROOT_DIR/scripts/evaluation/gen_eval.py" \
        "$CONFIG_PATH" \
        --checkpoint "$CHECKPOINT_PATH" \
        --batch_size 8 \
        --output "$OUTPUT_DIR" \
        --height 448 --width 448 \
        --cfg_scale 7.5 \
        --seed 42

It represents cfg_scale 7.5, num_steps=30, seed=42, I get the result:

Task breakdown
==============
position         = 85.00% (340 / 400)
colors           = 90.69% (341 / 376)
color_attr       = 71.75% (287 / 400)
counting         = 62.19% (199 / 320)
single_object    = 100.00% (320 / 320)
two_object       = 92.68% (367 / 396)

Overall score (avg. over tasks): 0.83718

and cfg_scale 4.5:

Task breakdown
==============
position         = 81.50% (326 / 400)
colors           = 90.43% (340 / 376)
color_attr       = 69.25% (277 / 400)
counting         = 62.50% (200 / 320)
single_object    = 99.38% (318 / 320)
two_object       = 91.67% (363 / 396)

Overall score (avg. over tasks): 0.82453