Questions about TokLIP Reconstruction and Generative Capabilities

[yfChang-cv]

Hi,
The reconstruction part of TokLIP (codebook & decoder) wasn't trained. Why do VQGAN and TokLIP have different FID scores in Table 4? If it was trained, what would the rFID score be? Also, why not include generative capabilities in the multimodal model? Could you explain the reasoning behind this?

[FelixMessi]

Hi, thank you for your questions.

In the experiments presented in Table 4, we concatenate low-level tokens from VQGAN with high-level features from the causal encoder as input to the GPT models. The results suggest that incorporating high-level semantics improves generation quality. Note that the VQGAN reconstruction module was not trained in our setup, so the rFID remains unchanged.

As for the generative capabilities of the multimodal model, we are actively working on this direction and plan to release updates soon. Please stay tuned to our repository.

[yfChang-cv]

Thank you for your response. Great work!