|
93 | 93 | "source": [ |
94 | 94 | "This tutorial demonstrates how to create and train a [sequence-to-sequence](https://developers.google.com/machine-learning/glossary#sequence-to-sequence-task) [Transformer](https://developers.google.com/machine-learning/glossary#Transformer) model to translate [Portuguese into English](https://www.tensorflow.org/datasets/catalog/ted_hrlr_translate#ted_hrlr_translatept_to_en). The Transformer was originally proposed in [\"Attention is all you need\"](https://arxiv.org/abs/1706.03762) by Vaswani et al. (2017).\n", |
95 | 95 | "\n", |
96 | | - "Transformers are deep neural networks that replace CNNs and RNNs with [self-attention](https://developers.google.com/machine-learning/glossary#self-attention). Self attention allows Transformers to easily transmit information across the input sequences.\n", |
| 96 | + "Transformers are deep neural networks that replace CNNs and RNNs with [self-attention](https://developers.google.com/machine-learning/glossary#self-attention). Self-attention allows Transformers to easily transmit information across the input sequences.\n", |
97 | 97 | "\n", |
98 | 98 | "As explained in the [Google AI Blog post](https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html):\n", |
99 | 99 | "\n", |
|
138 | 138 | "To get the most out of this tutorial, it helps if you know about [the basics of text generation](./text_generation.ipynb) and attention mechanisms. \n", |
139 | 139 | "\n", |
140 | 140 | "A Transformer is a sequence-to-sequence encoder-decoder model similar to the model in the [NMT with attention tutorial](https://www.tensorflow.org/text/tutorials/nmt_with_attention).\n", |
141 | | - "A single-layer Transformer takes a little more code to write, but is almost identical to that encoder-decoder RNN model. The only difference is that the RNN layers are replaced with self attention layers.\n", |
| 141 | + "A single-layer Transformer takes a little more code to write, but is almost identical to that encoder-decoder RNN model. The only difference is that the RNN layers are replaced with self-attention layers.\n", |
142 | 142 | "This tutorial builds a 4-layer Transformer which is larger and more powerful, but not fundamentally more complex." |
143 | 143 | ] |
144 | 144 | }, |
|
186 | 186 | "## Why Transformers are significant\n", |
187 | 187 | "\n", |
188 | 188 | "- Transformers excel at modeling sequential data, such as natural language.\n", |
189 | | - "- Unlike the [recurrent neural networks (RNNs)](./text_generation.ipynb), Transformers are parallelizable. This makes them efficient on hardware like GPUs and TPUs. The main reasons is that Transformers replaced recurrence with attention, and computations can happen simultaneously. Layer outputs can be computed in parallel, instead of a series like an RNN.\n", |
190 | | - "- Unlike [RNNs](https://www.tensorflow.org/guide/keras/rnn) (like [seq2seq, 2014](https://arxiv.org/abs/1409.3215)) or [convolutional neural networks (CNNs)](https://www.tensorflow.org/tutorials/images/cnn) (for example, [ByteNet](https://arxiv.org/abs/1610.10099)), Transformers are able to capture distant or long-range contexts and dependencies in the data between distant positions in the input or output sequences. Thus, longer connections can be learned. Attention allows each location to have access to the entire input at each layer, while in RNNs and CNNs, the information needs to pass through many processing steps to move a long distance, which makes it harder to learn.\n", |
| 189 | + "- Unlike [recurrent neural networks (RNNs)](./text_generation.ipynb), Transformers are parallelizable. This makes them efficient on hardware like GPUs and TPUs. The main reasons is that Transformers replaced recurrence with attention, and computations can happen simultaneously. Layer outputs can be computed in parallel, instead of a series like an RNN.\n", |
| 190 | + "- Unlike [RNNs](https://www.tensorflow.org/guide/keras/rnn) (such as [seq2seq, 2014](https://arxiv.org/abs/1409.3215)) or [convolutional neural networks (CNNs)](https://www.tensorflow.org/tutorials/images/cnn) (for example, [ByteNet](https://arxiv.org/abs/1610.10099)), Transformers are able to capture distant or long-range contexts and dependencies in the data between distant positions in the input or output sequences. Thus, longer connections can be learned. Attention allows each location to have access to the entire input at each layer, while in RNNs and CNNs, the information needs to pass through many processing steps to move a long distance, which makes it harder to learn.\n", |
191 | 191 | "- Transformers make no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects (for example, [StarCraft units](https://www.deepmind.com/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii)).\n", |
192 | 192 | "\n", |
193 | 193 | "\u003cimg src=\"https://www.tensorflow.org/images/tutorials/transformer/encoder_self_attention_distribution.png\" width=\"800\" alt=\"Encoder self-attention distribution for the word it from the 5th to the 6th layer of a Transformer trained on English-to-French translation\"\u003e\n", |
|
1007 | 1007 | }, |
1008 | 1008 | "outputs": [], |
1009 | 1009 | "source": [ |
1010 | | - "embed_pt = PositionalEmbedding(vocab_size=tokenizers.pt.get_vocab_size(), d_model=512)\n", |
1011 | | - "embed_en = PositionalEmbedding(vocab_size=tokenizers.en.get_vocab_size(), d_model=512)\n", |
| 1010 | + "embed_pt = PositionalEmbedding(vocab_size=tokenizers.pt.get_vocab_size().numpy(), d_model=512)\n", |
| 1011 | + "embed_en = PositionalEmbedding(vocab_size=tokenizers.en.get_vocab_size().numpy(), d_model=512)\n", |
1012 | 1012 | "\n", |
1013 | 1013 | "pt_emb = embed_pt(pt)\n", |
1014 | 1014 | "en_emb = embed_en(en)" |
|
1340 | 1340 | "id": "J6qrQxSpv34R" |
1341 | 1341 | }, |
1342 | 1342 | "source": [ |
1343 | | - "### The global self attention layer" |
| 1343 | + "### The global self-attention layer" |
1344 | 1344 | ] |
1345 | 1345 | }, |
1346 | 1346 | { |
|
1360 | 1360 | "source": [ |
1361 | 1361 | "\u003ctable\u003e\n", |
1362 | 1362 | "\u003ctr\u003e\n", |
1363 | | - " \u003cth colspan=1\u003eThe global self attention layer\u003c/th\u003e\n", |
| 1363 | + " \u003cth colspan=1\u003eThe global self-attention layer\u003c/th\u003e\n", |
1364 | 1364 | "\u003ctr\u003e\n", |
1365 | 1365 | "\u003ctr\u003e\n", |
1366 | 1366 | " \u003ctd\u003e\n", |
|
1378 | 1378 | "source": [ |
1379 | 1379 | "Since the context sequence is fixed while the translation is being generated, information is allowed to flow in both directions. \n", |
1380 | 1380 | "\n", |
1381 | | - "Before Transformers and self attention, models commonly used RNNs or CNNs to do this task:" |
| 1381 | + "Before Transformers and self-attention, models commonly used RNNs or CNNs to do this task:" |
1382 | 1382 | ] |
1383 | 1383 | }, |
1384 | 1384 | { |
|
1415 | 1415 | "- The RNN allows information to flow all the way across the sequence, but it passes through many processing steps to get there (limiting gradient flow). These RNN steps have to be run sequentially and so the RNN is less able to take advantage of modern parallel devices.\n", |
1416 | 1416 | "- In the CNN each location can be processed in parallel, but it only provides a limited receptive field. The receptive field only grows linearly with the number of CNN layers, You need to stack a number of Convolution layers to transmit information across the sequence ([Wavenet](https://arxiv.org/abs/1609.03499) reduces this problem by using dilated convolutions).\n", |
1417 | 1417 | "\n", |
1418 | | - "The global self attention layer on the other hand lets every sequence element directly access every other sequence element, with only a few operations, and all the outputs can be computed in parallel. \n", |
| 1418 | + "The global self-attention layer on the other hand lets every sequence element directly access every other sequence element, with only a few operations, and all the outputs can be computed in parallel. \n", |
1419 | 1419 | "\n", |
1420 | 1420 | "To implement this layer you just need to pass the target sequence, `x`, as both the `query`, and `value` arguments to the `mha` layer: " |
1421 | 1421 | ] |
|
1470 | 1470 | "source": [ |
1471 | 1471 | "\u003ctable\u003e\n", |
1472 | 1472 | "\u003ctr\u003e\n", |
1473 | | - " \u003cth colspan=1\u003eThe global self attention layer\u003c/th\u003e\n", |
| 1473 | + " \u003cth colspan=1\u003eThe global self-attention layer\u003c/th\u003e\n", |
1474 | 1474 | "\u003ctr\u003e\n", |
1475 | 1475 | "\u003ctr\u003e\n", |
1476 | 1476 | " \u003ctd\u003e\n", |
|
1499 | 1499 | "source": [ |
1500 | 1500 | "\u003ctable\u003e\n", |
1501 | 1501 | "\u003ctr\u003e\n", |
1502 | | - " \u003cth colspan=1\u003eThe global self attention layer\u003c/th\u003e\n", |
| 1502 | + " \u003cth colspan=1\u003eThe global self-attention layer\u003c/th\u003e\n", |
1503 | 1503 | "\u003ctr\u003e\n", |
1504 | 1504 | "\u003ctr\u003e\n", |
1505 | 1505 | " \u003ctd\u003e\n", |
|
1515 | 1515 | "id": "Yq4NtLymD99-" |
1516 | 1516 | }, |
1517 | 1517 | "source": [ |
1518 | | - "### The causal self attention layer" |
| 1518 | + "### The causal self-attention layer" |
1519 | 1519 | ] |
1520 | 1520 | }, |
1521 | 1521 | { |
|
1524 | 1524 | "id": "VufkgF7caLze" |
1525 | 1525 | }, |
1526 | 1526 | "source": [ |
1527 | | - "This layer does a similar job as the global self attention layer, for the output sequence:" |
| 1527 | + "This layer does a similar job as the global self-attention layer, for the output sequence:" |
1528 | 1528 | ] |
1529 | 1529 | }, |
1530 | 1530 | { |
|
1535 | 1535 | "source": [ |
1536 | 1536 | "\u003ctable\u003e\n", |
1537 | 1537 | "\u003ctr\u003e\n", |
1538 | | - " \u003cth colspan=1\u003eThe causal self attention layer\u003c/th\u003e\n", |
| 1538 | + " \u003cth colspan=1\u003eThe causal self-attention layer\u003c/th\u003e\n", |
1539 | 1539 | "\u003ctr\u003e\n", |
1540 | 1540 | "\u003ctr\u003e\n", |
1541 | 1541 | " \u003ctd\u003e\n", |
|
1551 | 1551 | "id": "0AtF1HYFEOYf" |
1552 | 1552 | }, |
1553 | 1553 | "source": [ |
1554 | | - "This needs to be handled differently from the encoder's global self attention layer. \n", |
| 1554 | + "This needs to be handled differently from the encoder's global self-attention layer. \n", |
1555 | 1555 | "\n", |
1556 | 1556 | "Like the [text generation tutorial](https://www.tensorflow.org/text/tutorials/text_generation), and the [NMT with attention](https://www.tensorflow.org/text/tutorials/nmt_with_attention) tutorial, Transformers are an \"autoregressive\" model: They generate the text one token at a time and feed that output back to the input. To make this _efficient_, these models ensure that the output for each sequence element only depends on the previous sequence elements; the models are \"causal\"." |
1557 | 1557 | ] |
|
1608 | 1608 | "id": "WLYfIa8eiYgk" |
1609 | 1609 | }, |
1610 | 1610 | "source": [ |
1611 | | - "To build a causal self attention layer, you need to use an appropriate mask when computing the attention scores and summing the attention `value`s.\n", |
| 1611 | + "To build a causal self-attention layer, you need to use an appropriate mask when computing the attention scores and summing the attention `value`s.\n", |
1612 | 1612 | "\n", |
1613 | 1613 | "This is taken care of automatically if you pass `use_causal_mask = True` to the `MultiHeadAttention` layer when you call it:" |
1614 | 1614 | ] |
|
1650 | 1650 | "source": [ |
1651 | 1651 | "\u003ctable\u003e\n", |
1652 | 1652 | "\u003ctr\u003e\n", |
1653 | | - " \u003cth colspan=1\u003eThe causal self attention layer\u003c/th\u003e\n", |
| 1653 | + " \u003cth colspan=1\u003eThe causal self-attention layer\u003c/th\u003e\n", |
1654 | 1654 | "\u003ctr\u003e\n", |
1655 | 1655 | "\u003ctr\u003e\n", |
1656 | 1656 | " \u003ctd\u003e\n", |
|
1679 | 1679 | "source": [ |
1680 | 1680 | "\u003ctable\u003e\n", |
1681 | 1681 | "\u003c/tr\u003e\n", |
1682 | | - " \u003cth colspan=1\u003eThe causal self attention layer\u003c/th\u003e\n", |
| 1682 | + " \u003cth colspan=1\u003eThe causal self-attention layer\u003c/th\u003e\n", |
1683 | 1683 | "\u003ctr\u003e\n", |
1684 | 1684 | "\u003ctr\u003e\n", |
1685 | 1685 | " \u003ctd\u003e\n", |
|
0 commit comments