additional parameter matrix factorization

[ShusenTang]

Hello, there seem to be two differences from the paper:

1. In the paper, the Qi and Ri matrices are factorized as products of low-rank matrices,
2. and for different rounds, the mogrifier parameter matrices are different, ie., Q1, Q3, R2, R4.

[RMichaelSwan]

It's been a while, so I will take a deeper look as soon as I have time (maybe a week or two). My hunch is that you are probably right as I remember having trouble understanding the exact set of equations that were being used. Feel free to add a pull request if you get a more proper version working!

[fawazsammani]

@RMichaelSwan and @ShusenTang perhaps you can do something like this for point 2?

To Define the layers:

self.mogrifier_list = nn.ModuleList([nn.Linear(hidden_size, input_size)])  # start with q
for i in range(1, mogrify_steps):
   if i%2 == 0:
        self.mogrifier_list.extend([nn.Linear(hidden_size, input_size)])  # q
   else:
        self.mogrifier_list.extend([nn.Linear(input_size, hidden_size)])  # r

and in the mogrify function (called every timestep):

def mogrify(self, x, h):
    for i in range(self.mogrify_steps):
        if (i+1) % 2 == 0: 
            h = (2*torch.sigmoid(self.mogrifier_list[i](x))) * h
        else:
            x = (2*torch.sigmoid(self.mogrifier_list[i](h))) * x
    return x, h