Questions about the T2I-CompBench results

[SparklingH]

Thanks for your excellent work!

I tried to evaluate RPG on the T2I-CompBench but got different results than those in Table 1. So, I want to ask for the details about the evaluation of RPG on the T2I-CompBench.

1. About the MLLMs: GPT-4o or other MLLMs.
2. About the Base Ratio: As you said in your paper, the Base Ratio is important in RPG which affects the generation results. So I want to know the value of the Base Ratio you use to evaluate.
3. About the results: I use RPG to generate one image for each prompt in T2I-CompBench with GPT-4o-mini (300 images for each metric). However, the results are quite different from those in Table 1. Some results are as follows:

• color -- 0.4917, shape -- 0.3864, texture -- 0.4573, spatial -- 0.1042, non-spatial -- 0.2877, complex -- 0.2997 (Base Ratio = 0.2)
• color -- 0.4464, shape -- 0.3863, texture -- 0.4289, spatial -- 0.1132, non-spatial -- 0.2950, complex -- 0.3012 (Base Ratio = 0.5)
  I also evaluate SD v2 on T2I-CompBench and the difference between 300 images and 3000 images is around 0.05. So I don't think it is caused by the number of generated images. Moreover, I use GPT-4o as MLLMs and the results are similar to those using GPT-4o-mini.

4. About the code: When I use RPG on the T2I-CompBench, some bugs also happen.

• When the 'Final split ratio' is '1': An error occurred: local variable 'split_ratio2' referenced before assignment.

Hope to see your reply.

Best regards.