Sharing our benchmarks and metrics!

[linzhiqiu]

I am Zhiqiu Lin, a final-year PhD student at Carnegie Mellon University working with Prof. Deva Ramanan. I came across your paper and like it a lot! Congratulations!

I also wanted to share some of our recent work on benchmarking generative VLMs:

1. VQAScore (ECCV'24): A simple but effective alignment score for text-to-image/video/3D generation, strongly agreeing with human judgments. VQAScore can be run using one-line Python code here! Google's Imagen3 used VQAScore as the strongest replacement for CLIPScore.
2. GenAI-Bench (CVPR'24 SynData Workshop): A benchmark with 1,600 prompts from professional designers for compositional visual generation. We also show VQAScore can serve as a strong reward metric to re-rank the DALLE-3 generated images. GenAI-Bench was awarded the Best Short Paper at the SynData@CVPR24 workshop and adopted in Imagen3's report.

Hope you find them useful!

Best,
Zhiqiu