Training Code for MLLM understanding and visual generation

[zgzxy001]

Hi, thank you for your great work! Do you have plan to release the training code for MLLM understanding and visual generation? Like the ones in TokenFlow (MLLM understanding, visual generation). Thanks!

[FelixMessi]

Hi, thanks for your interest in our work!

At the moment, we’re not ready to release the corresponding training code. In our setup, the LLM or GPT input is formed by concatenating the VQGAN features with the causal encoder, while the supervision signal comes from VQGAN.

For now, we’ve provided pseudocode that shows how to adapt TokLIP to MLLM, which you can use as a reference.

continuous_feat = visual.encode(image)
discrete_feat = visual.quantize.get_codebook_entry(image_codes)

continuous_embed = Linear(continuous_feat) # continuous_dim -> embed_dim
discrete_embed = Linear(discrete_feat) # discrete_dim -> embed_dim

## for discrete

discrete_embed = GELU(discrete_embed)
discrete_embed = Linear(discrete_embed) # embed_dim -> embed_dim//2

## for continuous

continuous_embed = RMSNorm(continuous_embed)
continuous_embed = GELU(continuous_embed)
continuous_embed = Linear(continuous_embed) # embed_dim -> embed_dim//2

embed = torch.cat([cont_embed, embed]