mllm作用疑问？

[zhangquanwei962]

作者你好，mllm微调之后看起来是可以理解初始图像和编辑文本的gap 但代码看起来只是用mllm的prefix encoder hidden state 这样子能用到mllm的理解能力吗？又没有用到答案的我有点儿不太理解

[JiayiGuo821]

Hi, this design follows several existing implementations that use the hidden states of LLMs to guide diffusion generation, such as LaVi-Bridge and ELLA. In the prefill stage, the MLLM receives all visual and textual tokens and constructs the kv cache that stores contextual representations. The subsequent decoding stage then generates outputs conditioned on this cache. From this perspective, although the decoding stage could further interpret the cached representations, the prefilled hidden states still to a considerable extent reflect the model’s internal understanding of the input information.

[JiayiGuo821]

Also see Qwen-Image.