Skip to content

Question about reproducing Janus Pro evaluation results (request for eval config / code) #13

Description

@tls0523

Hello, thank you for releasing this benchmark and the leaderboard — it is very helpful for the community.
I am currently trying to reproduce the Janus Pro results reported on the leaderboard. However, I am running into difficulties reproducing the evaluation scores and generation quality.
Specifically:
I am using the official Janus Pro inference code released by the authors.
With this setup, I am unable to reproduce the reported leaderboard metrics.
The generated images also appear noticeably worse than the examples or results implied by the leaderboard scores.
Because of this, I suspect there may be additional evaluation code, inference configuration, or generation settings (e.g., sampling strategy, guidance scale, number of steps, resolution, post-processing, etc.) that are not fully specified in the public inference scripts.
I would like to ask:
Is the Janus Pro evaluation code used for this benchmark available?
If not, would it be possible to share the exact inference and evaluation configuration used to produce the leaderboard results?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions