DLUB_Seminar4_RNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DLUB_Seminar4_RNN.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNAMWeca3tyzNHQ4AquhYHS",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dl-ub-summer-school/2021/blob/main/DLUB_Seminar4_RNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6-o5Api-UYlF"
      },
      "source": [
        "Recurent Neural Network RNN\n",
        "==============================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dx3HxU3ZUg89"
      },
      "source": [
        "### 1.2 RNN\n",
        "\n",
        "**RNN** дараалалт датаны цаг хугацааны болон бусад хамаарлыг голлон анхааран үзсэн загварчлал модел юм.\n",
        "\n",
        "RNN-д оролтын датан дээр яг адил нетворк(адил парамерт бүхий) давтан ашиглах ч тэр дундаа өмнөх оролтын датаны бүх мэдээллийг багтаан оруулж ашиглана. \n",
        "\n",
        "Зураг харвал\n",
        "\n",
        "<img src=\"https://www.dropbox.com/s/vm8e8sy204erqj1/010.png?raw=1\" width=\"600\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OdBFWh64UgcT"
      },
      "source": [
        "Оролтын датаг цаг хугацааны хамааралтай цуваа дата гэж үзвэл  $(x_0,x_0,...,x_t,...)$ гэсэн оролтын датаны үр дүн болох гаралтын дата нь  $(h_0,h_0,...,h_t,...)$ хэлбэрээр илэрхийлэгдэнэ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0d1pVFSbUkqA"
      },
      "source": [
        "<img src=\"https://www.dropbox.com/s/ij5780tko623wgd/011.png?raw=1\" width=\"1000\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84nC-LkgUnYV"
      },
      "source": [
        "### Гаралтын хэмжээнээс шалтгаалан"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1kGmFDyUvPP"
      },
      "source": [
        "\n",
        "<img src=\"https://i.stack.imgur.com/b4sus.jpg\" width=\"800\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VB_fEmIiUy-2"
      },
      "source": [
        "### Model Definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmuUIBWEUzds"
      },
      "source": [
        "Тухайн үеийн RNN давхарга нь тухайн давхаргын оролт ба\n",
        "Өмнөх RNN давхаргаас гаралтыг хүлээн авна. Эдгээр хоёр мэдээлэл дээр үндэслэн\n",
        "дараах хугацааны гаралтыг тооцоолно. Энэ үед хийсэн тооцоог дараах байдлаар математик томъёогоор илэрхийлнэ.\n",
        "\n",
        "### $h_t = tanh(h_{t-1} W_h + x_t W_x + b)$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adoAFeG3WKwv"
      },
      "source": [
        "### Backpropagation Through Time"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VvlEteqeU3k_"
      },
      "source": [
        "<img src=\"https://www.dropbox.com/s/6pjzhxvcur3qd6u/012.png?raw=1\" width=\"1000\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z44uOJjYWM3v"
      },
      "source": [
        "### Truncated BTT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JlwMhc_pU6dU"
      },
      "source": [
        "<img src=\"https://www.dropbox.com/s/jblbrcxw0heru25/013.png?raw=1\" width=\"1000\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n8viQ7xIU7IA"
      },
      "source": [
        "<img src=\"https://www.dropbox.com/s/06wje1n82h0a7he/014.png?raw=1\" width=\"1000\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HpG1jwiLU_R5",
        "outputId": "a5a83de1-3068-4c3a-96df-be34b1c890ba"
      },
      "source": [
        "!wget  https://download.pytorch.org/tutorial/data.zip\n",
        "!unzip data.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-06-28 15:54:10--  https://download.pytorch.org/tutorial/data.zip\n",
            "Resolving download.pytorch.org (download.pytorch.org)... 13.226.50.24, 13.226.50.77, 13.226.50.63, ...\n",
            "Connecting to download.pytorch.org (download.pytorch.org)|13.226.50.24|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2882130 (2.7M) [application/zip]\n",
            "Saving to: ‘data.zip’\n",
            "\n",
            "data.zip            100%[===================>]   2.75M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2021-06-28 15:54:11 (21.3 MB/s) - ‘data.zip’ saved [2882130/2882130]\n",
            "\n",
            "Archive:  data.zip\n",
            "   creating: data/\n",
            "  inflating: data/eng-fra.txt        \n",
            "   creating: data/names/\n",
            "  inflating: data/names/Arabic.txt   \n",
            "  inflating: data/names/Chinese.txt  \n",
            "  inflating: data/names/Czech.txt    \n",
            "  inflating: data/names/Dutch.txt    \n",
            "  inflating: data/names/English.txt  \n",
            "  inflating: data/names/French.txt   \n",
            "  inflating: data/names/German.txt   \n",
            "  inflating: data/names/Greek.txt    \n",
            "  inflating: data/names/Irish.txt    \n",
            "  inflating: data/names/Italian.txt  \n",
            "  inflating: data/names/Japanese.txt  \n",
            "  inflating: data/names/Korean.txt   \n",
            "  inflating: data/names/Polish.txt   \n",
            "  inflating: data/names/Portuguese.txt  \n",
            "  inflating: data/names/Russian.txt  \n",
            "  inflating: data/names/Scottish.txt  \n",
            "  inflating: data/names/Spanish.txt  \n",
            "  inflating: data/names/Vietnamese.txt  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z0z3JjKnU_pn"
      },
      "source": [
        "import os\n",
        "import random\n",
        "from string import ascii_letters\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "import torch.nn.functional as F\n",
        "#from unidecode import unidecode\n",
        "\n",
        "_ = torch.manual_seed(42)\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aWq0LgGJVCAf"
      },
      "source": [
        "data_dir = \"./data/names\"\n",
        "\n",
        "lang2label = {\n",
        "    file_name.split(\".\")[0]: torch.tensor([i], dtype=torch.long)\n",
        "    for i, file_name in enumerate(os.listdir(data_dir))\n",
        "}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yU9cRpBIVD8Z",
        "outputId": "bf389d19-b319-4e5c-f274-e3b7cf9009de"
      },
      "source": [
        "lang2label"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'Arabic': tensor([17]),\n",
              " 'Chinese': tensor([16]),\n",
              " 'Czech': tensor([5]),\n",
              " 'Dutch': tensor([7]),\n",
              " 'English': tensor([3]),\n",
              " 'French': tensor([2]),\n",
              " 'German': tensor([0]),\n",
              " 'Greek': tensor([13]),\n",
              " 'Irish': tensor([10]),\n",
              " 'Italian': tensor([4]),\n",
              " 'Japanese': tensor([9]),\n",
              " 'Korean': tensor([6]),\n",
              " 'Polish': tensor([14]),\n",
              " 'Portuguese': tensor([15]),\n",
              " 'Russian': tensor([1]),\n",
              " 'Scottish': tensor([11]),\n",
              " 'Spanish': tensor([8]),\n",
              " 'Vietnamese': tensor([12])}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0uOOMsGvVF0M"
      },
      "source": [
        "num_langs = len(lang2label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3cQpGFpsVHVR",
        "outputId": "4d0387f6-3890-4d6a-c6d2-2caad8c68298"
      },
      "source": [
        "import unicodedata\n",
        "import string\n",
        "\n",
        "all_letters = string.ascii_letters + \" .,;'\"\n",
        "n_letters = len(all_letters)\n",
        "\n",
        "# Turn a Unicode string to plain ASCII\n",
        "def unicodeToAscii(s):\n",
        "    return ''.join(\n",
        "        c for c in unicodedata.normalize('NFD', s)\n",
        "        if unicodedata.category(c) != 'Mn'\n",
        "        and c in all_letters\n",
        "    )\n",
        "print(unicodeToAscii('Ślusàrski'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Slusarski\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LQ-NXiyln0wP"
      },
      "source": [
        "### One-hot encoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4jJvuNjZVLTb",
        "outputId": "bc4c7a79-c01d-4812-b5c7-150e4473fd58"
      },
      "source": [
        "# TODO\n",
        "# char2idx = ..."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "59"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ju5zxACwVMoz"
      },
      "source": [
        "# TODO\n",
        "def name2tensor(name):\n",
        "    return tensor"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EE5jH3EtVNl4",
        "outputId": "b9bc63a3-0881-424a-83a2-b9529dd1f087"
      },
      "source": [
        "name2tensor(\"abc\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0.]],\n",
              "\n",
              "        [[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0.]],\n",
              "\n",
              "        [[0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "          0., 0., 0., 0., 0., 0., 0., 0.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "unnVynJYVO6h"
      },
      "source": [
        "tensor_names = []\n",
        "target_langs = []\n",
        "\n",
        "for file in os.listdir(data_dir):\n",
        "    with open(os.path.join(data_dir, file)) as f:\n",
        "        lang = file.split(\".\")[0]\n",
        "        names = [unicodeToAscii(line.rstrip()) for line in f]\n",
        "        for name in names:\n",
        "            try:\n",
        "                tensor_names.append(name2tensor(name))\n",
        "                target_langs.append(lang2label[lang])\n",
        "            except KeyError:\n",
        "                pass"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YXjaSpZOVQhb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "22a263ec-57f1-4577-da23-bf2818055405"
      },
      "source": [
        "len(tensor_names)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "20074"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n7HnbCnhfm7w"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_idx, test_idx = train_test_split(\n",
        "    range(len(target_langs)), \n",
        "    test_size=0.1, \n",
        "    shuffle=True\n",
        ")\n",
        "\n",
        "train_dataset = [\n",
        "    (tensor_names[i], target_langs[i])\n",
        "    for i in train_idx\n",
        "]\n",
        "\n",
        "test_dataset = [\n",
        "    (tensor_names[i], target_langs[i])\n",
        "    for i in test_idx\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FONuOcb4fpWr",
        "outputId": "c92a9eee-b491-4dc8-9da7-958aecc0960c"
      },
      "source": [
        "print(f\"Train: {len(train_dataset)}\")\n",
        "print(f\"Test: {len(test_dataset)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train: 18066\n",
            "Test: 2008\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gEoILx4GBOWJ"
      },
      "source": [
        "class RNN(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(RNN, self).__init__()\n",
        "\n",
        "        self.rnn = nn.RNN(\n",
        "            input_size=INPUT_SIZE,\n",
        "            hidden_size=32,     # rnn hidden unit\n",
        "            num_layers=1,       # number of rnn layer\n",
        "        )\n",
        "        self.out = nn.Linear(32, 1)\n",
        "\n",
        "    def forward(self, x, h_state):\n",
        "        # x (batch, time_step, input_size)\n",
        "        # h_state (n_layers, batch, hidden_size)\n",
        "        # r_out (batch, time_step, hidden_size)\n",
        "        r_out, h_state = self.rnn(x, h_state)\n",
        "\n",
        "        return r_out"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yWiahOH4fvJk"
      },
      "source": [
        "<img src=\"https://www.dropbox.com/s/x2knav50tqsocxh/015.png?raw=1\" width=\"400\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p9Q9pnhbA5gV"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j3Nb5s1nf1RT"
      },
      "source": [
        "hidden_size = 256\n",
        "learning_rate = 0.001\n",
        "# input_size : 59\n",
        "\n",
        "class MyRNN(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size, output_size):\n",
        "        super(MyRNN, self).__init__()\n",
        "        #TODO\n",
        "        #Linear1\n",
        "        #Linear2\n",
        "        #Softmax\n",
        "        self.init_state()\n",
        "    \n",
        "    def forward(self, x):\n",
        "        combined = torch.cat((x, self.hidden_state), 1)\n",
        "        return output\n",
        "    \n",
        "    def init_state(self):\n",
        "        self.hidden_state = torch.zeros(1, self.hidden_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k4_wgWbvf2Y7"
      },
      "source": [
        "hidden_size = 256\n",
        "learning_rate = 0.001\n",
        "\n",
        "model = MyRNN(num_letters, hidden_size, num_langs)\n",
        "loss_func = nn.NLLLoss()\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cKeGaTogf9lV",
        "outputId": "13c2bd22-5e6b-4188-e068-122fc17a040b"
      },
      "source": [
        "print(model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "MyRNN(\n",
            "  (in2hidden): Linear(in_features=315, out_features=256, bias=True)\n",
            "  (in2output): Linear(in_features=315, out_features=18, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WkeGeeYogAMc",
        "outputId": "7c5049c8-01b6-4ba4-8191-90feade66f3a"
      },
      "source": [
        "!pip install torchinfo"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torchinfo\n",
            "  Downloading https://files.pythonhosted.org/packages/e7/d3/11f9901d75f4d105b2b1700c81f83579fd33c4cf0ec88bb7a165d96c7bb4/torchinfo-0.1.5-py3-none-any.whl\n",
            "Installing collected packages: torchinfo\n",
            "Successfully installed torchinfo-0.1.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WqJGu01RgCAg"
      },
      "source": [
        "from torchinfo import summary"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zEPEcCbXgCh3",
        "outputId": "1c52945e-7e03-4281-b8bf-fad8934dc9b5"
      },
      "source": [
        "stats = summary(model, (1, 59), verbose=2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==========================================================================================\n",
            "Layer (type:depth-idx)                   Output Shape              Param #\n",
            "==========================================================================================\n",
            "MyRNN                                    --                        --\n",
            "├─Linear: 1-1                            [1, 256]                  80,896\n",
            "│    └─weight                                                      80,640\n",
            "│    └─bias                                                        256\n",
            "├─Linear: 1-2                            [1, 18]                   5,688\n",
            "│    └─weight                                                      5,670\n",
            "│    └─bias                                                        18\n",
            "==========================================================================================\n",
            "Total params: 86,584\n",
            "Trainable params: 86,584\n",
            "Non-trainable params: 0\n",
            "Total mult-adds (M): 0.09\n",
            "==========================================================================================\n",
            "Input size (MB): 0.00\n",
            "Forward/backward pass size (MB): 0.00\n",
            "Params size (MB): 0.35\n",
            "Estimated Total Size (MB): 0.35\n",
            "==========================================================================================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XpTRSGzPgGQP",
        "outputId": "6a0ab65c-1a54-40b8-ee63-e9d5e57e2323"
      },
      "source": [
        "num_epochs = 2\n",
        "print_interval = 3000\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "    random.shuffle(train_dataset)\n",
        "    for i, (name, label) in enumerate(train_dataset):\n",
        "        model.init_state()\n",
        "        #TODO\n",
        "        \n",
        "        if (i + 1) % print_interval == 0:\n",
        "            print(\n",
        "                f\"Epoch [{epoch + 1}/{num_epochs}], \"\n",
        "                f\"Step [{i + 1}/{len(train_dataset)}], \"\n",
        "                f\"Loss: {loss.item():.4f}\"\n",
        "            )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch [1/2], Step [3000/18066], Loss: 0.1151\n",
            "Epoch [1/2], Step [6000/18066], Loss: 0.0900\n",
            "Epoch [1/2], Step [9000/18066], Loss: 0.8687\n",
            "Epoch [1/2], Step [12000/18066], Loss: 4.3885\n",
            "Epoch [1/2], Step [15000/18066], Loss: 0.0233\n",
            "Epoch [1/2], Step [18000/18066], Loss: 4.6316\n",
            "Epoch [2/2], Step [3000/18066], Loss: 1.5977\n",
            "Epoch [2/2], Step [6000/18066], Loss: 1.0086\n",
            "Epoch [2/2], Step [9000/18066], Loss: 0.0670\n",
            "Epoch [2/2], Step [12000/18066], Loss: 1.7392\n",
            "Epoch [2/2], Step [15000/18066], Loss: 0.0000\n",
            "Epoch [2/2], Step [18000/18066], Loss: 1.3911\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_CZWtxT-h-7d",
        "outputId": "2e9a98bb-a0ae-4118-b9ac-bdd9a7a41a34"
      },
      "source": [
        "correct = 0\n",
        "total = len(test_dataset)\n",
        "\n",
        "with torch.no_grad():\n",
        "    for name, label in test_dataset:\n",
        "        model.init_state()\n",
        "        for char in name:\n",
        "\n",
        "            output = model(char)\n",
        "            print(output)\n",
        "        pred = torch.max(output, dim=1)[1]\n",
        "        correct += (pred == label).sum().item()\n",
        "\n",
        "print(f\"Accuracy: {correct / total * 100:.4f}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 74.4522%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GDyN4NY-h_ss"
      },
      "source": [
        "label2lang = {label.item(): lang for lang, label in lang2label.items()}\n",
        "\n",
        "def myrnn_predict(name):\n",
        "    \n",
        "    tensor_name = name2tensor(name)\n",
        "    with torch.no_grad():\n",
        "        model.init_state()\n",
        "        for char in tensor_name:\n",
        "            output = model(char)\n",
        "        _, pred = torch.max(output, dim=1) \n",
        "    return label2lang[pred.item()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "qPOP6fCoiDTk",
        "outputId": "36616da6-10d9-457b-ef83-c36d4ec52506"
      },
      "source": [
        "myrnn_predict(\"Smith\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'English'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "_KV2n9CeiGCP",
        "outputId": "ee520886-c1c5-47eb-ef82-33b1065942f1"
      },
      "source": [
        "myrnn_predict(\"Lenin\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Russian'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "13eNrS6xiIHV",
        "outputId": "ea3ced73-aecc-4bee-9489-ba2dffe3ac92"
      },
      "source": [
        "myrnn_predict(\"Qin\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Chinese'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "W7rdcIVrlf8K",
        "outputId": "bbe542a6-a137-46d4-9ef2-f982cbcd17a4"
      },
      "source": [
        "myrnn_predict(\"Yamada\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Japanese'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D9tV4cW5rbw7"
      },
      "source": [
        "### Gated RNN : LSTM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pq4C_pcerXdO"
      },
      "source": [
        "<img src='https://miro.medium.com/max/700/1*84MseqCgGpdvAWf6YBjGgA.jpeg' width=600>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6w0e4b0dsOht"
      },
      "source": [
        "### LSTM Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kU5-RNFgryM-"
      },
      "source": [
        "<img src=\"https://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs11063-020-10319-3/MediaObjects/11063_2020_10319_Fig1_HTML.png\" width=\"800\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vyuNgDKiiMPI"
      },
      "source": [
        "#LSTM class\n",
        "class LSTM_net(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size, output_size):\n",
        "        super(LSTM_net, self).__init__()\n",
        "        #TODO\n",
        "\n",
        "    def forward(self, input):\n",
        "        #TODO\n",
        "\n",
        "    def init_state(self):\n",
        "        #TODO"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IwtzKRTqiLOa"
      },
      "source": [
        "hidden_size = 100\n",
        "learning_rate = 0.001\n",
        "\n",
        "lstm_net = LSTM_net(num_letters, hidden_size, num_langs)\n",
        "loss_func = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(lstm_net.parameters(), lr=learning_rate)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jWOIbqeLnkk8",
        "outputId": "1ed7474d-382a-4fd7-f132-30ed61ee7f44"
      },
      "source": [
        "print(lstm_net)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "LSTM_net(\n",
            "  (lstm_cell): LSTM(59, 100)\n",
            "  (h2o): Linear(in_features=100, out_features=18, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-bZ7lzOHnyCC"
      },
      "source": [
        "from torchinfo import summary"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k9ze47vann1J",
        "outputId": "929d621b-9453-446d-d985-d37fc209d8e7"
      },
      "source": [
        "stats = summary(lstm_net, (1, 59), verbose=2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==========================================================================================\n",
            "Layer (type:depth-idx)                   Output Shape              Param #\n",
            "==========================================================================================\n",
            "LSTM_net                                 --                        --\n",
            "├─LSTM: 1-1                              [1, 1, 100]               64,400\n",
            "│    └─weight_ih_l0                                                23,600\n",
            "│    └─weight_hh_l0                                                40,000\n",
            "│    └─bias_ih_l0                                                  400\n",
            "│    └─bias_hh_l0                                                  400\n",
            "├─Linear: 1-2                            [1, 1, 18]                1,818\n",
            "│    └─weight                                                      1,800\n",
            "│    └─bias                                                        18\n",
            "==========================================================================================\n",
            "Total params: 66,218\n",
            "Trainable params: 66,218\n",
            "Non-trainable params: 0\n",
            "Total mult-adds (M): 0.07\n",
            "==========================================================================================\n",
            "Input size (MB): 0.00\n",
            "Forward/backward pass size (MB): 0.00\n",
            "Params size (MB): 0.26\n",
            "Estimated Total Size (MB): 0.27\n",
            "==========================================================================================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eepOFD7Vns30",
        "outputId": "aad7b337-5617-436b-9162-8d487fa64327"
      },
      "source": [
        "num_epochs = 2\n",
        "print_interval = 3000\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "    random.shuffle(train_dataset)\n",
        "    for i, (name, label) in enumerate(train_dataset):\n",
        "        lstm_net.init_state()\n",
        "        #TODO\n",
        "        \n",
        "        if (i + 1) % print_interval == 0:\n",
        "            print(\n",
        "                f\"Epoch [{epoch + 1}/{num_epochs}], \"\n",
        "                f\"Step [{i + 1}/{len(train_dataset)}], \"\n",
        "                f\"Loss: {loss.item():.4f}\"\n",
        "            )"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch [1/2], Step [3000/18066], Loss: 2.9822\n",
            "Epoch [1/2], Step [6000/18066], Loss: 0.4768\n",
            "Epoch [1/2], Step [9000/18066], Loss: 0.0003\n",
            "Epoch [1/2], Step [12000/18066], Loss: 7.2584\n",
            "Epoch [1/2], Step [15000/18066], Loss: 7.8647\n",
            "Epoch [1/2], Step [18000/18066], Loss: 0.6150\n",
            "Epoch [2/2], Step [3000/18066], Loss: 4.7334\n",
            "Epoch [2/2], Step [6000/18066], Loss: 2.8736\n",
            "Epoch [2/2], Step [9000/18066], Loss: 0.0000\n",
            "Epoch [2/2], Step [12000/18066], Loss: 3.5216\n",
            "Epoch [2/2], Step [15000/18066], Loss: 0.0000\n",
            "Epoch [2/2], Step [18000/18066], Loss: 0.0023\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6vkl-7_MolKH",
        "outputId": "7dde7984-d621-4236-bddc-7d979f05b92f"
      },
      "source": [
        "correct = 0\n",
        "total = len(test_dataset)\n",
        "\n",
        "with torch.no_grad():\n",
        "    for name, label in test_dataset:\n",
        "        lstm_net.init_state()\n",
        "        for char in name:\n",
        "            output = lstm_net(char)\n",
        "        pred = torch.max(output, dim=1)[1]\n",
        "        correct += (pred == label).sum().item()\n",
        "\n",
        "print(f\"Accuracy: {correct / total * 100:.4f}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy: 80.0797%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kFogjgWWo-fd"
      },
      "source": [
        "label2lang = {label.item(): lang for lang, label in lang2label.items()}\n",
        "\n",
        "def lstm_predict(name):\n",
        "    \n",
        "    tensor_name = name2tensor(name)\n",
        "    with torch.no_grad():\n",
        "        lstm_net.init_state()\n",
        "        for char in tensor_name:\n",
        "            output = lstm_net(char)\n",
        "        _, pred = torch.max(output, dim=1) \n",
        "    return label2lang[pred.item()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "JqjCp8KgpRob",
        "outputId": "ebf200be-836d-4c53-a99b-c252b900e2d8"
      },
      "source": [
        "lstm_predict(\"John\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Russian'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "RE4Z_7_IpVKz",
        "outputId": "3e32b2d2-0806-4882-fe36-adccf74006ca"
      },
      "source": [
        "lstm_predict(\"Smith\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'English'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "TB2IIlaSpbl9",
        "outputId": "7603a5cf-404a-4286-9f8f-f19e8eac0e79"
      },
      "source": [
        "lstm_predict(\"Lenin\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Russian'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "h5NPe7RmpcMX",
        "outputId": "83a4a80c-ffff-482d-d972-999157a60439"
      },
      "source": [
        "lstm_predict(\"Qin\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Chinese'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "_oHNy0IJpkZ9",
        "outputId": "ee5b877b-d612-4ec3-c958-0224d59377bc"
      },
      "source": [
        "lstm_predict(\"Yamada\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Japanese'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "IKPWmTLcpnvV",
        "outputId": "daaa7ef8-fe4f-4252-c5ed-b0b5568f3f41"
      },
      "source": [
        "lstm_predict(\"Tanaka\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Japanese'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BEGWsENvpprn"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}