transformer

Transformer

Useful links:

• https://towardsdatascience.com/transformers-141e32e69591
• https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04
• https://arxiv.org/abs/1706.03762
• https://www.youtube.com/watch?v=rURRYI66E54&ab_channel=Computerphile
• https://www.youtube.com/watch?v=U0s0f995w14&ab_channel=AladdinPersson

Multi-Head Attention

• values (V)
• keys (K)
• queries (Q)

Formulas

$$attention(Q, K, V) = softmax(Q*K**T) / (sqrt(d_k)) * V$$

More complex ideas / words / stuff

• permutationally invariant