ff_layer weights sharing

[RispaXyl]

Hi! It seems that in TransformerEncoder each TransformerLayer uses the same ff_layer weights that are instantiated once. Is this intentional?

        if ff_layer is None:
            ff_layer =  torch.nn.Sequential(torch.nn.Linear(model_dim, ff_dim),
                                            torch.nn.ReLU(),
                                            torch.nn.Linear(ff_dim, model_dim))
        else:
            ff_layer = ff_layer()
        if cross_attention_layer is not None:
            cross_attention_layer = cross_attention_layer()
        self.encoder_layers = torch.nn.ModuleList([TransformerLayer(model_dim, 
                                                    attention_layer(), 
                                                    ff_layer, 
                                                    norm_layer, 
                                                    norm_type, 
                                                    cross_attention_layer,
                                                    drop_path,
                                                    init_values) for i in range(num_layers)])