Hello, thank you for releasing this benchmark and the leaderboard — it is very helpful for the community.
I am currently trying to reproduce the Janus Pro results reported on the leaderboard. However, I am running into difficulties reproducing the evaluation scores and generation quality.
Specifically:
I am using the official Janus Pro inference code released by the authors.
With this setup, I am unable to reproduce the reported leaderboard metrics.
The generated images also appear noticeably worse than the examples or results implied by the leaderboard scores.
Because of this, I suspect there may be additional evaluation code, inference configuration, or generation settings (e.g., sampling strategy, guidance scale, number of steps, resolution, post-processing, etc.) that are not fully specified in the public inference scripts.
I would like to ask:
Is the Janus Pro evaluation code used for this benchmark available?
If not, would it be possible to share the exact inference and evaluation configuration used to produce the leaderboard results?
Hello, thank you for releasing this benchmark and the leaderboard — it is very helpful for the community.
I am currently trying to reproduce the Janus Pro results reported on the leaderboard. However, I am running into difficulties reproducing the evaluation scores and generation quality.
Specifically:
I am using the official Janus Pro inference code released by the authors.
With this setup, I am unable to reproduce the reported leaderboard metrics.
The generated images also appear noticeably worse than the examples or results implied by the leaderboard scores.
Because of this, I suspect there may be additional evaluation code, inference configuration, or generation settings (e.g., sampling strategy, guidance scale, number of steps, resolution, post-processing, etc.) that are not fully specified in the public inference scripts.
I would like to ask:
Is the Janus Pro evaluation code used for this benchmark available?
If not, would it be possible to share the exact inference and evaluation configuration used to produce the leaderboard results?