Problems about groups

[liqi0126]

Should not the key_conv, query_conv and value_conv be defined with group parameters? I am not clear what group parameters do in the following lines.

Stand-Alone-Self-Attention/attention.py

Lines 43 to 50 in a983f0f

	k_out = k_out.contiguous().view(batch, self.groups, self.out_channels // self.groups, height, width, -1)
	v_out = v_out.contiguous().view(batch, self.groups, self.out_channels // self.groups, height, width, -1)
	q_out = q_out.view(batch, self.groups, self.out_channels // self.groups, height, width, 1)
	out = q_out * k_out
	out = F.softmax(out, dim=-1)
	out = torch.einsum('bnchwk,bnchwk -> bnchw', out, v_out).view(batch, -1, height, width)

[mrngbaeb]

The 'groups' are not like the groups for group convolutions. In this case, each of the multi-attention heads takes in the full input dimension (not 1/n_groups of it). Computationally, this is done as one convolution and then reshaped to separate the heads.

[liqi0126]

However, it seems like not to separate k_out, v_out and q_out into groups gives the same out results in the above codes. So what does the group parameter exactly do?

[zenghui420]

Seems like q_out and k_out should be 'matrix-multiplied' at dim=2 (the one with the size self.out_channels // self.groups).

[xyecoding]

According to the listed code lines, it seems that self.groups has no effect on the result. I beleive no matter how to set self.groups, the code can only realize the case "group=1".