PyTorch_2/sentiment_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use some Huggingface libraries for downloading and processing the dataset, and therefore need to install them first.\\\n",
    "The first is the datasets library which allows you to download and manipulate datasets.\\\n",
    "See [their guide](https://huggingface.co/docs/datasets/installation) for an introduction to the datasets library if you want to know more.\\\n",
    "The other library helps us in downloading data from the Huggingface Hub.\n",
    "\n",
    "We can simple install them using pip:\n",
    "> Note that `%pip` is a magic command for use in Jupyter Notebooks as discussed in the tutorial on setting up Python!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install datasets huggingface_hub ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "from functools import partial\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from datasets import load_dataset\n",
    "from huggingface_hub import hf_hub_download\n",
    "from torch.utils.data import DataLoader\n",
    "from tqdm.auto import trange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis\n",
    "\n",
    "Sentiment Analysis (SA) determines the emotional tone of a text sequence (e.g., a sentence) and classifies it into predefined categories.\\\n",
    "Therefore, SA is a text classification task which assigns a single class to the whole input.\n",
    "\n",
    "In this part, we will use the SST2 dataset, which stands for Stanford Sentiment Treebank.\\\n",
    "This is a binary task where inputs are labeled as `positive` (`1`) or `negative` (`0`).\n",
    "\n",
    "There are several possibilites on how to 'solve' this task.\\\n",
    "Starting from sequences, there exist many ways to turn them into features.\\\n",
    "A common way is to split them into smaller units, called tokens, which could be words.\\\n",
    "For this, there are again many possibilities how this can be implemented.\\\n",
    "You can do it manually or use some library for it, and there are many NLP libraries.\\\n",
    "Also, the neural network architecture can be almost arbitrary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Loading\n",
    "\n",
    "There are several possibiliites for downloading datasets.\\\n",
    "You can read them from files, or directly use a library with an online repository of datasets, such as [Huggingface Datasets](https://huggingface.co/docs/datasets/index).\n",
    "\n",
    "> A note on the Huggingface Datasets library: You can decide where downloaded files are cached, see https://huggingface.co/docs/datasets/v3.2.0/en/cache#cache-directory.\\\n",
    "> The documentation is as always a good source of information!\n",
    "\n",
    "We will use this library in our example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sst2 = load_dataset(\"stanfordnlp/sst2\")\n",
    "# Print the dataset object to see the dataset's structure\n",
    "print(sst2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn more about using this library, there is a very useful [tutorial](https://huggingface.co/docs/datasets/v3.2.0/en/tutorial) available online."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are three splits, and how many samples are contained in each split.\\\n",
    "Let's take a look at an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_train = sst2[\"train\"]\n",
    "# Print the first sample in the training dataset\n",
    "print(dataset_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, each sample includes an index, the input sentence and a label.\\\n",
    "This sample was labeled `negative` (`0`; `1` stand for `positive` in this dataset)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "In order to turn words into features, we can use pre-trained embeddings.\\\n",
    "These have been trained to carry the semantics of the words.\n",
    "\n",
    "### GloVe\n",
    "In GloVe, differences of word pairs should be roughly equal:\\\n",
    "<img src=\"https://nlp.stanford.edu/projects/glove/images/man_woman.jpg\" width=500/>\\\n",
    "Source: https://nlp.stanford.edu/projects/glove/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GloVe embeddings exist trained on different data (varying in the number of tokens and vocabulary size), and with different embedding dimension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the GloVe embeddings\n",
    "glove = hf_hub_download(\"stanfordnlp/glove\", \"glove.6B.zip\")\n",
    "\n",
    "with zipfile.ZipFile(glove, \"r\") as f:\n",
    "    print(f.namelist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we use the smallest vocabulary size with 300-dimensional features (but feel free to experiment with other dimensions!).\\\n",
    "How does the file look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There are multiple files with different dimensionality of the features in the zip archive: 50d, 100d, 200d, 300d\n",
    "filename = \"glove.6B.300d.txt\"\n",
    "with zipfile.ZipFile(glove, \"r\") as f:\n",
    "    for idx, line in enumerate(f.open(filename)):\n",
    "        print(line)\n",
    "        if idx == 5:\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Unpack the downloaded file\n",
    "word_to_index = dict()\n",
    "embeddings = []\n",
    "\n",
    "with zipfile.ZipFile(glove, \"r\") as f:\n",
    "    for idx, line in enumerate(f.open(filename)):\n",
    "        values = line.split()\n",
    "        word = values[0].decode(\"utf-8\")\n",
    "        features = torch.tensor([float(value) for value in values[1:]])\n",
    "        word_to_index[word] = idx\n",
    "        embeddings.append(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "Play around and inspect the dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We bring the embeddings into a useful format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Last token in the vocabulary is '<unk>' which is used for out-of-vocabulary words\n",
    "# We also add a '<pad>' token to the vocabulary for padding sequences\n",
    "word_to_index[\"<pad>\"] = len(word_to_index)\n",
    "padding_token_id = word_to_index[\"<pad>\"]\n",
    "unk_token_id = word_to_index[\"<unk>\"]\n",
    "\n",
    "embeddings.append(torch.zeros(embeddings[0].shape))\n",
    "\n",
    "# Convert the list of tensors to a single tensor\n",
    "embeddings = torch.stack(embeddings)\n",
    "\n",
    "print(f\"Embedding shape: {embeddings.size(1)}\")\n",
    "print(f\"Padding token id: {padding_token_id}\")\n",
    "print(f\"Unknown token id: {unk_token_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Processing\n",
    "\n",
    "Starting from our input sentences, we need to 1) tokenize the sentences to get smaller units (words or tokens) and 2) convert these tokens into vector representations using the pre-trained embeddings.\n",
    "\n",
    "The datasets library provides support for processing datasets, for more information see https://huggingface.co/docs/datasets/process.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text: str):\n",
    "    # We lowercase the text and split it by whitespaces to extract the tokens\n",
    "    # Punctuation is treated as separate tokens\n",
    "    return text.lower().split()\n",
    "\n",
    "\n",
    "def map_token_to_index(token):\n",
    "    # Return the index of the token or the index of the '<unk>' token if the token is not in the vocabulary\n",
    "    return word_to_index.get(token, unk_token_id)\n",
    "\n",
    "\n",
    "def map_text_to_indices(text: str):\n",
    "    return [map_token_to_index(token) for token in tokenize(text)]\n",
    "\n",
    "\n",
    "def prepare_dataset(dataset):\n",
    "    return dataset.map(lambda x: {\"token_ids\": map_text_to_indices(x[\"sentence\"])}, num_proc=4)\n",
    "\n",
    "\n",
    "dataset_train_tokenized = prepare_dataset(dataset_train)\n",
    "\n",
    "# Print the first sample in the tokenized training dataset\n",
    "print(dataset_train_tokenized[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need a dataloader that takes care of batching our data.\\\n",
    "You have seen this before, but this time we need also take care of padding, since the length of the sentences varies in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_inputs(batch, keys_to_pad=[\"token_ids\"], padding_value=-1):\n",
    "    # Pad keys_to_pad to the maximum length in batch\n",
    "    padded_batch = {}\n",
    "    for key in keys_to_pad:\n",
    "        # Get maximum length in batch\n",
    "        max_len = max([len(sample[key]) for sample in batch])\n",
    "        # Pad all samples to the maximum length\n",
    "        padded_batch[key] = torch.tensor(\n",
    "            [\n",
    "                sample[key] + [padding_value] * (max_len - len(sample[key]))\n",
    "                for sample in batch\n",
    "            ]\n",
    "        )\n",
    "    # Add remaining keys to the batch\n",
    "    for key in batch[0].keys():\n",
    "        if key not in keys_to_pad:\n",
    "            padded_batch[key] = torch.tensor([sample[key] for sample in batch])\n",
    "    return padded_batch\n",
    "\n",
    "\n",
    "def get_dataloader(dataset, batch_size=32, shuffle=False):\n",
    "    # Create a DataLoader for the dataset\n",
    "    return DataLoader(\n",
    "        dataset,\n",
    "        batch_size=batch_size,\n",
    "        collate_fn=partial(pad_inputs, padding_value=padding_token_id),\n",
    "        shuffle=shuffle,\n",
    "    )\n",
    "\n",
    "\n",
    "# We select the columns that we want to keep in the dataset\n",
    "dataset_train_tokenized = dataset_train_tokenized.with_format(\n",
    "    columns=[\"token_ids\", \"label\"]\n",
    ")\n",
    "# Create a DataLoader for the training dataset\n",
    "dataloader_train = get_dataloader(dataset_train_tokenized, batch_size=8, shuffle=True)\n",
    "\n",
    "for batch in dataloader_train:\n",
    "    token_ids = batch[\"token_ids\"]\n",
    "    labels = batch[\"label\"]\n",
    "    print(token_ids)\n",
    "    print(labels)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "We will use a simple network here with the pre-trained embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleTextClassifier(torch.nn.Module):\n",
    "    def __init__(self, embeddings, hidden_size=128, padding_index=-1):\n",
    "        super().__init__()\n",
    "        self.embedding = torch.nn.Embedding.from_pretrained(\n",
    "            embeddings, freeze=True, padding_idx=padding_index\n",
    "        )\n",
    "        self.layer1 = torch.nn.Linear(embeddings.shape[1], hidden_size)\n",
    "        self.output_layer = torch.nn.Linear(hidden_size, 2)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.embedding(x)\n",
    "        # x is of dimension (batch_size, seq_len, embedding_dim)\n",
    "        # By summing the embeddings of all tokens in the sequence, we get a bag-of-words vector for each sample input\n",
    "        x = torch.sum(x, dim=1)\n",
    "        x = torch.relu(self.layer1(x))\n",
    "        x = self.output_layer(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We instantiate and verify that the model is working (no errors):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SimpleTextClassifier(embeddings, padding_index=padding_token_id)\n",
    "print(model(torch.tensor(dataset_train_tokenized[\"token_ids\"][:2])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "How would you use an RNN instead of this simple feed-forward network?\\\n",
    "See https://pytorch.org/docs/stable/nn.html#recurrent-layers for details on how to use RNNs in PyTorch.\n",
    "\n",
    "The following architecture can be used for inspiration:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![RNN for Sentence Classification](https://www.tensorflow.org/static/text/tutorials/images/bidirectional.png)\\\n",
    "Source: https://www.tensorflow.org/text/tutorials/text_classification_rnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNNTextClassifier(torch.nn.Module):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RNNTextClassifier(embeddings, padding_index=padding_token_id)\n",
    "print(model(torch.tensor(dataset_train_tokenized[\"token_ids\"][0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "To assess the performance of our model, we compute a metric that fits for our task.\\\n",
    "For pure classification like SA, this is often the accuracy and optionally also the loss on the provided dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_accuracy(predictions: torch.tensor, labels: torch.tensor):\n",
    "    return torch.sum(torch.argmax(predictions, dim=1) == labels).item() / len(labels)\n",
    "\n",
    "\n",
    "def evaluate_model(model, dataset, loss_fn=None):\n",
    "    # Compute the accuracy and optionally the loss of the model on the dataset\n",
    "    dataloader = get_dataloader(dataset, batch_size=32)\n",
    "    accuracies = []\n",
    "    losses = []\n",
    "    # We don't need to compute gradients for the evaluation\n",
    "    with torch.no_grad():\n",
    "        for batch in dataloader:\n",
    "            token_ids = batch[\"token_ids\"]\n",
    "            labels = batch[\"label\"]\n",
    "            predictions = model(token_ids)\n",
    "            if loss_fn:\n",
    "                loss = loss_fn(predictions, labels)\n",
    "                losses.append(loss.item())\n",
    "            accuracies.append(compute_accuracy(predictions, labels))\n",
    "    return sum(accuracies) / len(accuracies), (\n",
    "        (sum(losses) / len(losses)) if loss_fn else None\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to evaluate our model on a separate dataset, we will also have to process this similarly to our training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_val = sst2[\"validation\"]\n",
    "dataset_val_tokenized = prepare_dataset(dataset_val)\n",
    "dataset_val_tokenized = dataset_val_tokenized.with_format(\n",
    "    columns=[\"token_ids\", \"label\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can evaluate our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy, _ = evaluate_model(model, dataset_val_tokenized)\n",
    "print(f\"Accuracy on the validation dataset: {accuracy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With randomly initialized weights, we expect to end up at ~50% Accuracy (in average)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Now having processed the data and instantiated our model, it is time for training.\\\n",
    "Therefore, we need a loss function and an optimizer.\\\n",
    "The rest is just a simple training loop as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_fn = torch.nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses_train, losses_val = [], []\n",
    "accuracies_train, accuracies_val = [], []\n",
    "\n",
    "# Compute loss and accuracy on the training set\n",
    "accuracy, loss = evaluate_model(model, dataset_train_tokenized, loss_fn)\n",
    "losses_train.append(loss)\n",
    "accuracies_train.append(accuracy)\n",
    "\n",
    "# Compute loss and accuracy on the validation set\n",
    "accuracy, loss = evaluate_model(model, dataset_val_tokenized, loss_fn)\n",
    "losses_val.append(loss)\n",
    "accuracies_val.append(accuracy)\n",
    "\n",
    "NUM_EPOCHS = 10\n",
    "\n",
    "# A progress bar to visualize the training progress\n",
    "pbar = trange(NUM_EPOCHS)\n",
    "# Training loop\n",
    "for epoch in pbar:\n",
    "    # Do one epoch of training\n",
    "    for batch in dataloader_train:\n",
    "        # Extract the token ids and the labels from the batch\n",
    "        token_ids = batch[\"token_ids\"]\n",
    "        labels = batch[\"label\"]\n",
    "\n",
    "        # Forward pass\n",
    "        predictions = model(token_ids)\n",
    "        loss = loss_fn(predictions, labels)\n",
    "\n",
    "        # Backpropagation\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    # Calculate the loss and accuracy on the training set\n",
    "    acc_train, loss_train = evaluate_model(model, dataset_train_tokenized, loss_fn)\n",
    "    accuracies_train.append(acc_train)\n",
    "    losses_train.append(loss_train)\n",
    "\n",
    "    # Evaluate the model on the validation set\n",
    "    acc_val, loss_val = evaluate_model(model, dataset_val_tokenized, loss_fn)\n",
    "    accuracies_val.append(acc_val)\n",
    "    losses_val.append(loss_val)\n",
    "\n",
    "    pbar.set_postfix_str(\n",
    "        f\"Train loss: {losses_train[-1]} - Validation acc: {accuracies_val[-1]}\"\n",
    "    ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize the loss and accuracy\n",
    "plt.plot(losses_train, color=\"orange\", linestyle=\"-\", label=\"Train loss\")\n",
    "plt.plot(losses_val, color=\"orange\", linestyle=\"--\", label=\"Validation loss\")\n",
    "plt.plot(accuracies_train, color=\"steelblue\", linestyle=\"-\", label=\"Train accuracy\")\n",
    "plt.plot(accuracies_val, color=\"steelblue\", linestyle=\"--\", label=\"Validation accuracy\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactive Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'd like to perform some interactive testing with our trained model.\\\n",
    "Widgets allow for a simple text interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import widgets\n",
    "from IPython.display import display\n",
    "\n",
    "sentence_widget = widgets.Text(\n",
    "    value=\"This movie is terrible\",\n",
    "    placeholder=\"Type something\",\n",
    "    description=\"Sentence:\",\n",
    "    disabled=False,\n",
    ")\n",
    "display(sentence_widget)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "Now the task is to do a prediction for the provided sentence using our trained model.\n",
    "\n",
    "Some hints:\n",
    "* You need to convert the input sentence to token ids, using the same mapping as for training. Can you re-use something to accomplish this?\n",
    "* Since we only have a single input, we don't need batching nor a dataloader\n",
    "* We don't need the gradients from the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the input sentence from the widget\n",
    "sentence = sentence_widget.value"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}