The benchmark results are not reproducible

[shuizhonghaitong]

Hello, I re-ran Qwen 2.5 VL downloaded from ModelScope without modifying any configurations, but I cannot reproduce the results reported in the paper. Could you help me identify the issue? Below are the results from the OCR benchmark.
[
{
"blue": 15.26,
"chrf": 49.87,
"ed": 103.0,
"cer": 0.27,
"wer": 0.64,
"meteor": 0.35,
"dataset": "historicalbooks"
},
{
"blue": 2.7,
"chrf": 23.73,
"ed": 27.83,
"cer": 0.58,
"wer": 0.94,
"meteor": 0.11,
"dataset": "muharaf"
},
{
"blue": 61.37,
"chrf": 84.59,
"ed": 64.8,
"cer": 0.19,
"wer": 0.33,
"meteor": 0.8,
"dataset": "arabicocr"
},
{
"blue": 17.12,
"chrf": 49.15,
"ed": 21.8,
"cer": 0.35,
"wer": 0.72,
"meteor": 0.31,
"dataset": "khatt"
},
{
"blue": 48.11,
"chrf": 72.62,
"ed": 159.59,
"cer": 0.18,
"wer": 0.35,
"meteor": 0.65,
"dataset": "hindawi"
},
{
"blue": 20.65,
"chrf": 50.84,
"ed": 13.84,
"cer": 0.33,
"wer": 0.66,
"meteor": 0.34,
"dataset": "onlinekhatt"
},
{
"blue": 7.74,
"chrf": 40.76,
"ed": 31.29,
"cer": 0.68,
"wer": 1.05,
"meteor": 0.19,
"dataset": "historyar"
},
{
"blue": 20.19,
"chrf": 80.09,
"ed": 0.91,
"cer": 0.19,
"wer": 0.36,
"meteor": 0.36,
"dataset": "evarest"
},
{
"blue": 73.86,
"chrf": 86.57,
"ed": 12.11,
"cer": 0.15,
"wer": 0.23,
"meteor": 0.86,
"dataset": "patsocr"
},
{
"blue": 4.56,
"chrf": 34.45,
"ed": 274.89,
"cer": 4.04,
"wer": 4.35,
"meteor": 0.31,
"dataset": "khattparagraph"
},
{
"blue": 43.98,
"chrf": 70.29,
"ed": 16.97,
"cer": 0.42,
"wer": 0.63,
"meteor": 0.54,
"dataset": "isippt"
},
{
"blue": 1.88,
"chrf": 31.17,
"ed": 5.58,
"cer": 0.68,
"wer": 1.08,
"meteor": 0.1,
"dataset": "adab"
},
{
"blue": 55.08,
"chrf": 78.06,
"ed": 6.8,
"cer": 0.17,
"wer": 0.35,
"meteor": 0.67,
"dataset": "synthesizear"
}
]

[shuizhonghaitong]

or maybe did you set the random seed? And if yes, what value did you set?

[maximazzik]

For khattparagraph, the wrong ground truth is being used: the annotation for the full page contains only the first line, which causes the CER to shoot above 400%. More details here: #5