A simple implementation of Transformer using Pytorch
Much thanks to the repo : https://github.com/jadore801120/attention-is-all-you-need-pytorch/tree/master !
It's a clear and detailed implementation, from which I've learnt a lot . And my code is mainly refered to the repo , I added some annotations, added some type hints and renamed some variables to make better understanding.
And also much thanks to https://github.com/sooftware/attentions
The repo provides a variety of implementations of attention with different types , I also get inspired a lot !
run :
python3 model.py
can get the very very start of the model and check if the model is built successfully , I'll use this model to try to train with some datasets , and I'll update soon.