The measured metrics for Qwen-Image, evaluated using gpt-4o, show significant discrepancies.

[zzzandyx]

Hello, I retested the Qwen-Image metrics according to your method and found a significant discrepancy. What could be the reason? Model Cultural Time Space Biology Physics Chemistry Overall Qwen 0.62 0.63 0.77 0.57 0.75 0.4 0.62 Qwen 0.56 0.57 0.65 0.48 0.55 0.32 0.54 The metrics above are those you tested; the metrics below are those I actually tested. I'm using the internal API interface for testing.

[Purshow]

Hello, we are using https://huggingface.co/Qwen/Qwen-Image, and all use the default inference settings. The api uses GPT-4o-2024-05-13. Please check if there are any problems.

[zzzandyx]

Is your GPT-4o-2024-05-13 version a special version of GPT? What are the differences between it and the regular GPT-4o? I am using GPT-4o, but I don't have a specific version number or date.

[Purshow]

Yes, please refer to openai’s official api document. GPT-4o-2024-05-13 is a designated stable version

[zzzandyx]

I checked the official OpenAI API documentation and found that GPT-4o has three versions: 2024-05-13, 2024-08-06, and 2024-11-20. I'm currently using the latest version. Could you please provide the benchmark metrics for the latest version of GPT-4o? I noticed that version GPT-4o-2024-05-13 will be discontinued soon, and the metrics might not match up. Thank you very much.

[Purshow]

Hi,

Since version 2024-05-13 has become the standard benchmark for WISE evaluation in the community—especially when involving closed-source models that are inaccessible to us—it is challenging for us to re-run all evaluations using the latest versions.

Furthermore, based on our previous testing, while different GPT versions do introduce some fluctuations in results, the variance is typically not substantial. If you observe an abnormal discrepancy, we suggest prioritizing a check on whether the correct model was used or if there are issues with the inference settings.

[zekaiblog]

Hi, I wonder how to use the benchmark once GPT-4o-2024-05-13 is discontinued (and I want to test my model on WISE). Can you provide the image results before evaluation for reproduction? Thanks.

[Purshow]

Hi,

Thanks for your suggestion. We have already taken this issue into consideration. To ensure the long-term reproducibility of the benchmark, we are planning to release a new version of WISE, which will utilize the open-source model Qwen3.5 as the primary evaluator instead of relying solely on closed-source APIs.

If everything goes as planned, we expect to open-source this implementation and update the evaluation results within the next two weeks.

Regarding the performance of Qwen-Image: we strictly followed the official implementation provided by the Qwen team. You should be able to reproduce our results easily by using the code from the official Qwen-Image repository with default inference settings.

[zekaiblog]

hi, is the tested data same as current data (just change the API or evaluation)? or newly built ones?

[Purshow]

We will keep the current data for everyone's reproduction, and at the same time, we will update a version of the data for a more rigorous and strict evaluation, which will change about 200-300 entries.

[Purshow]

hi, is the tested data same as current data (just change the API or evaluation)? or newly built ones?

Hi, we have updated the repo!