Where is the linear projection reflected in the code?

[jinqia0]

The linear projection layer between the Visual Encoder and the face tokens does not seem to be reflected in the SkyReelsA1ImagePoseToVideoPipeline class. Did you define a new linear projection layer in this intermediate step, or directly fine-tune some of the linear layers within the Visual Encoder?

[qiudi0127]

@jinqia0 The linear projection is written directly to CogVideoXPatchEmbed class in transformer3d.py.

[jinqia0]

The code I found regarding the projection layer in CogVideoXPatchEmbed is as follows:

        self.proj = nn.Conv2d(
            in_channels, embed_dim, kernel_size=(patch_size, patch_size), stride=patch_size, bias=bias
        )

        self.text_proj = nn.Linear(text_embed_dim, embed_dim)

        self.pose_proj = torch.nn.Sequential(
           torch.nn.Linear(1152, 1152*3),
           torch.nn.SiLU(),
           torch.nn.Linear(1152*3, text_embed_dim),
        #    torch.nn.LayerNorm(text_embed_dim)
        )
        self.pose_proj = zero_module(self.pose_proj)

Are the projection layer and convolutional layer in the first two stages of training mentioned in your paper implemented here? Do they correspond to self.proj and self.pose_proj respectively?