Replication of Mogrifier LSTM paper

[qlan3]

Does your code replicate the results in Mogrifier LSTM paper?

[RMichaelSwan]

No. I attempted to implement it and test it in a basic RNN network for text prediction using the Brown corpus but didn't get very promising results compared to a regular LSTM. Perhaps with a larger dataset, more training time, better network architecture, or better hyperparameter tuning I could have come up with something more definitive. I believe my implementation is correct, but it's hard to verify without spending more time with the aforementioned criteria (this was a weekend project).

Looking at the paper, it seems they did plan to release the code (they even put a link that does not work: https://github.com/deepmind/lamb ), but something has been holding the writer up. I scoured the internet and found a couple comments (that I can't seem to find any more) between the author and another researcher that seemed to indicate the author was having trouble getting their official implementation working properly.

[qlan3]

Thank you for your detailed reply! I've emailed the author.

[RMichaelSwan]

Welcome! I'd be interested to hear if they tell you anything new.

[RMichaelSwan]

@qlan3 thanks to #2 I seem to have something approaching the results of the paper. Would need more testing to fully verify, but it seems promising. Hope you find what you need!

[qlan3]

Thanks!