natural_language_processing/02-text-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae7d075d",
   "metadata": {
    "papermill": {
     "duration": 0.023374,
     "end_time": "2021-08-23T03:02:24.873294",
     "exception": false,
     "start_time": "2021-08-23T03:02:24.849920",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Natural Language Processing](https://www.kaggle.com/learn/natural-language-processing) course.  You can reference the tutorial at [this link](https://www.kaggle.com/matleonard/text-classification).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2b76379",
   "metadata": {
    "papermill": {
     "duration": 0.022191,
     "end_time": "2021-08-23T03:02:24.920717",
     "exception": false,
     "start_time": "2021-08-23T03:02:24.898526",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Natural Language Classification\n",
    "\n",
    "You did such a great job for DeFalco's restaurant in the previous exercise that the chef has hired you for a new project.\n",
    "\n",
    "The restaurant's menu includes an email address where visitors can give feedback about their food. \n",
    "\n",
    "The manager wants you to create a tool that automatically sends him all the negative reviews so he can fix them, while automatically sending all the positive reviews to the owner, so the manager can ask for a raise. \n",
    "\n",
    "You will first build a model to distinguish positive reviews from negative reviews using Yelp reviews because these reviews include a rating with each review. Your data consists of the text body of each review along with the star rating. Ratings with 1-2 stars count as \"negative\", and ratings with 4-5 stars are \"positive\". Ratings with 3 stars are \"neutral\" and have been dropped from the data.\n",
    "\n",
    "Let's get started. First, run the next code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9f6a2159",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:24.977048Z",
     "iopub.status.busy": "2021-08-23T03:02:24.976333Z",
     "iopub.status.idle": "2021-08-23T03:02:26.372063Z",
     "shell.execute_reply": "2021-08-23T03:02:26.371433Z",
     "shell.execute_reply.started": "2021-08-23T02:03:33.472978Z"
    },
    "papermill": {
     "duration": 1.429308,
     "end_time": "2021-08-23T03:02:26.372228",
     "exception": false,
     "start_time": "2021-08-23T03:02:24.942920",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Setup complete\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Set up code checking\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.nlp.ex2 import *\n",
    "print(\"\\nSetup complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46529ea7",
   "metadata": {
    "papermill": {
     "duration": 0.022063,
     "end_time": "2021-08-23T03:02:26.417230",
     "exception": false,
     "start_time": "2021-08-23T03:02:26.395167",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 1: Evaluate the Approach\n",
    "\n",
    "Is there anything about this approach that concerns you? After you've thought about it, run the function below to see one point of view."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "130e054c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:26.469282Z",
     "iopub.status.busy": "2021-08-23T03:02:26.468486Z",
     "iopub.status.idle": "2021-08-23T03:02:26.474595Z",
     "shell.execute_reply": "2021-08-23T03:02:26.474126Z",
     "shell.execute_reply.started": "2021-08-23T02:07:10.084634Z"
    },
    "papermill": {
     "duration": 0.035125,
     "end_time": "2021-08-23T03:02:26.474724",
     "exception": false,
     "start_time": "2021-08-23T03:02:26.439599",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"1_EvaluateFeedbackFormApproach\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "Any way of setting up an ML problem will have multiple strengths and weaknesses.  So you may have thought of different issues than listed here.\n",
       "\n",
       "The strength of this approach is that it allows you to distinguish positive email messages from negative emails even though you don't have historical emails that you have labeled as positive or negative.\n",
       "\n",
       "The weakness of this approach is that emails may be systematically different from Yelp reviews in ways that make your model less accurate. For example, customers might generally use different words or slang in emails, and the model based on Yelp reviews won't have seen these words.\n",
       "\n",
       "If you wanted to see how serious this issue is, you could compare word frequencies between the two sources. In practice, manually reading a few emails from each source may be enough to see if it's a serious issue. \n",
       "\n",
       "If you wanted to do something fancier, you could create a dataset that contains both Yelp reviews and emails and see whether a model can tell a reviews source from the text content. Ideally, you'd like to find that model didn't perform well, because it would mean your data sources are similar. That approach seems unnecessarily complex here."
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "Any way of setting up an ML problem will have multiple strengths and weaknesses.  So you may have thought of different issues than listed here.\n",
       "\n",
       "The strength of this approach is that it allows you to distinguish positive email messages from negative emails even though you don't have historical emails that you have labeled as positive or negative.\n",
       "\n",
       "The weakness of this approach is that emails may be systematically different from Yelp reviews in ways that make your model less accurate. For example, customers might generally use different words or slang in emails, and the model based on Yelp reviews won't have seen these words.\n",
       "\n",
       "If you wanted to see how serious this issue is, you could compare word frequencies between the two sources. In practice, manually reading a few emails from each source may be enough to see if it's a serious issue. \n",
       "\n",
       "If you wanted to do something fancier, you could create a dataset that contains both Yelp reviews and emails and see whether a model can tell a reviews source from the text content. Ideally, you'd like to find that model didn't perform well, because it would mean your data sources are similar. That approach seems unnecessarily complex here."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "#step_1.solution()\n",
    "step_1.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1edd5ed",
   "metadata": {
    "papermill": {
     "duration": 0.023049,
     "end_time": "2021-08-23T03:02:26.521275",
     "exception": false,
     "start_time": "2021-08-23T03:02:26.498226",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 2: Review Data and Create the model\n",
    "\n",
    "Moving forward with your plan, you'll need to load the data. Here's some basic code to load data and split it into a training and validation set. Run this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9451fae1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:26.571789Z",
     "iopub.status.busy": "2021-08-23T03:02:26.570848Z",
     "iopub.status.idle": "2021-08-23T03:02:27.035362Z",
     "shell.execute_reply": "2021-08-23T03:02:27.035908Z",
     "shell.execute_reply.started": "2021-08-23T02:08:28.598552Z"
    },
    "papermill": {
     "duration": 0.49128,
     "end_time": "2021-08-23T03:02:27.036081",
     "exception": false,
     "start_time": "2021-08-23T03:02:26.544801",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def load_data(csv_file, split=0.9):\n",
    "    data = pd.read_csv(csv_file)\n",
    "    \n",
    "    # Shuffle data\n",
    "    train_data = data.sample(frac=1, random_state=7)\n",
    "    \n",
    "    texts = train_data.text.values\n",
    "    labels = [{\"POSITIVE\": bool(y), \"NEGATIVE\": not bool(y)}\n",
    "              for y in train_data.sentiment.values]\n",
    "    split = int(len(train_data) * split)\n",
    "    \n",
    "    train_labels = [{\"cats\": labels} for labels in labels[:split]]\n",
    "    val_labels = [{\"cats\": labels} for labels in labels[split:]]\n",
    "    \n",
    "    return texts[:split], train_labels, texts[split:], val_labels\n",
    "\n",
    "train_texts, train_labels, val_texts, val_labels = load_data('../input/nlp-course/yelp_ratings.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79227fd7",
   "metadata": {
    "papermill": {
     "duration": 0.023331,
     "end_time": "2021-08-23T03:02:27.083286",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.059955",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You will use this training data to build a model. The code to build the model is the same as what you saw in the tutorial. So that is copied below for you.\n",
    "\n",
    "But because your data is different, there are **two lines in the modeling code cell that you'll need to change.** Can you figure out what they are? \n",
    "\n",
    "First, run the cell below to look at a couple elements from your training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9871db96",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:27.133768Z",
     "iopub.status.busy": "2021-08-23T03:02:27.133145Z",
     "iopub.status.idle": "2021-08-23T03:02:27.138893Z",
     "shell.execute_reply": "2021-08-23T03:02:27.138279Z",
     "shell.execute_reply.started": "2021-08-23T02:09:21.413814Z"
    },
    "papermill": {
     "duration": 0.032083,
     "end_time": "2021-08-23T03:02:27.139043",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.106960",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Texts from training data\n",
      "------\n",
      "[\"Some of the best sushi I've ever had....and I come from the East Coast.  Unreal toro, have some of it's available.\"\n",
      " \"One of the best burgers I've ever had and very well priced. I got the tortilla burger and is was delicious especially with there tortilla soup!\"]\n",
      "\n",
      "Labels from training data\n",
      "------\n",
      "[{'cats': {'POSITIVE': True, 'NEGATIVE': False}}, {'cats': {'POSITIVE': True, 'NEGATIVE': False}}]\n"
     ]
    }
   ],
   "source": [
    "print('Texts from training data\\n------')\n",
    "print(train_texts[:2])\n",
    "print('\\nLabels from training data\\n------')\n",
    "print(train_labels[:2])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0a0412b",
   "metadata": {
    "papermill": {
     "duration": 0.023507,
     "end_time": "2021-08-23T03:02:27.186284",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.162777",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now, having seen this data, find the two lines that need to be changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dfc54fc2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:27.240970Z",
     "iopub.status.busy": "2021-08-23T03:02:27.240248Z",
     "iopub.status.idle": "2021-08-23T03:02:27.721054Z",
     "shell.execute_reply": "2021-08-23T03:02:27.721552Z",
     "shell.execute_reply.started": "2021-08-23T02:11:33.230308Z"
    },
    "papermill": {
     "duration": 0.510527,
     "end_time": "2021-08-23T03:02:27.721731",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.211204",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2_CreateTextCatModel\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# Create an empty model\n",
    "nlp = spacy.blank(\"en\")\n",
    "\n",
    "# Create the TextCategorizer with exclusive classes and \"bow\" architecture\n",
    "textcat = nlp.create_pipe(\n",
    "              \"textcat\",\n",
    "              config={\n",
    "                \"exclusive_classes\": True,\n",
    "                \"architecture\": \"bow\"})\n",
    "\n",
    "# Add the TextCategorizer to the empty model\n",
    "nlp.add_pipe(textcat)\n",
    "\n",
    "# Add labels to text classifier\n",
    "textcat.add_label(\"NEGATIVE\")\n",
    "textcat.add_label(\"POSITIVE\")\n",
    "\n",
    "# Check your answer\n",
    "step_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5ad838fd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:27.774848Z",
     "iopub.status.busy": "2021-08-23T03:02:27.773986Z",
     "iopub.status.idle": "2021-08-23T03:02:27.777451Z",
     "shell.execute_reply": "2021-08-23T03:02:27.777915Z",
     "shell.execute_reply.started": "2021-08-23T02:11:36.782277Z"
    },
    "papermill": {
     "duration": 0.03159,
     "end_time": "2021-08-23T03:02:27.778082",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.746492",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_2.hint()\n",
    "#step_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf89b83",
   "metadata": {
    "papermill": {
     "duration": 0.025329,
     "end_time": "2021-08-23T03:02:27.828287",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.802958",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 3: Train Function\n",
    "\n",
    "Implement a function `train` that updates a model with training data. Most of this is general data munging, which we've filled in for you. Just add the one line of code necessary to update your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "55e54c87",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:27.881748Z",
     "iopub.status.busy": "2021-08-23T03:02:27.880823Z",
     "iopub.status.idle": "2021-08-23T03:02:35.037388Z",
     "shell.execute_reply": "2021-08-23T03:02:35.037893Z",
     "shell.execute_reply.started": "2021-08-23T02:15:24.725462Z"
    },
    "papermill": {
     "duration": 7.184762,
     "end_time": "2021-08-23T03:02:35.038063",
     "exception": false,
     "start_time": "2021-08-23T03:02:27.853301",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"3_TrainFunction\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy.util import minibatch\n",
    "import random\n",
    "\n",
    "def train(model, train_data, optimizer):\n",
    "    losses = {}\n",
    "    random.seed(1)\n",
    "    random.shuffle(train_data)\n",
    "    \n",
    "    batches = minibatch(train_data, size=8)\n",
    "    for batch in batches:\n",
    "        # train_data is a list of tuples [(text0, label0), (text1, label1), ...]\n",
    "        # Split batch into texts and labels\n",
    "        texts, labels = zip(*batch)\n",
    "        \n",
    "        # Update model with texts and labels\n",
    "        model.update(texts, labels, sgd=optimizer, losses=losses)\n",
    "        \n",
    "    return losses\n",
    "\n",
    "# Check your answer\n",
    "step_3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fa9221fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:35.094854Z",
     "iopub.status.busy": "2021-08-23T03:02:35.094246Z",
     "iopub.status.idle": "2021-08-23T03:02:35.096111Z",
     "shell.execute_reply": "2021-08-23T03:02:35.096594Z",
     "shell.execute_reply.started": "2021-08-23T02:15:31.441357Z"
    },
    "papermill": {
     "duration": 0.032975,
     "end_time": "2021-08-23T03:02:35.096751",
     "exception": false,
     "start_time": "2021-08-23T03:02:35.063776",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_3.hint()\n",
    "#step_3.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "24ee270c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:02:35.151751Z",
     "iopub.status.busy": "2021-08-23T03:02:35.151119Z",
     "iopub.status.idle": "2021-08-23T03:04:31.977812Z",
     "shell.execute_reply": "2021-08-23T03:04:31.978342Z",
     "shell.execute_reply.started": "2021-08-23T02:16:03.583012Z"
    },
    "papermill": {
     "duration": 116.855794,
     "end_time": "2021-08-23T03:04:31.978528",
     "exception": false,
     "start_time": "2021-08-23T03:02:35.122734",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.701575470772399\n"
     ]
    }
   ],
   "source": [
    "# Fix seed for reproducibility\n",
    "spacy.util.fix_random_seed(1)\n",
    "random.seed(1)\n",
    "\n",
    "# This may take a while to run!\n",
    "optimizer = nlp.begin_training()\n",
    "train_data = list(zip(train_texts, train_labels))\n",
    "losses = train(nlp, train_data, optimizer)\n",
    "print(losses['textcat'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40da834e",
   "metadata": {
    "papermill": {
     "duration": 0.025918,
     "end_time": "2021-08-23T03:04:32.030589",
     "exception": false,
     "start_time": "2021-08-23T03:04:32.004671",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can try this slightly trained model on some example text and look at the probabilities assigned to each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0636fe82",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:32.089183Z",
     "iopub.status.busy": "2021-08-23T03:04:32.088452Z",
     "iopub.status.idle": "2021-08-23T03:04:32.091012Z",
     "shell.execute_reply": "2021-08-23T03:04:32.091493Z",
     "shell.execute_reply.started": "2021-08-23T02:18:49.336487Z"
    },
    "papermill": {
     "duration": 0.034897,
     "end_time": "2021-08-23T03:04:32.091664",
     "exception": false,
     "start_time": "2021-08-23T03:04:32.056767",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'NEGATIVE': 0.7731689214706421, 'POSITIVE': 0.22683113813400269}\n"
     ]
    }
   ],
   "source": [
    "text = \"This tea cup was full of holes. Do not recommend.\"\n",
    "doc = nlp(text)\n",
    "print(doc.cats)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6b775cb",
   "metadata": {
    "papermill": {
     "duration": 0.026209,
     "end_time": "2021-08-23T03:04:32.144559",
     "exception": false,
     "start_time": "2021-08-23T03:04:32.118350",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "These probabilities look reasonable. Now you should turn them into an actual prediction.\n",
    "\n",
    "# Step 4: Making Predictions\n",
    "\n",
    "Implement a function `predict` that predicts the sentiment of text examples. \n",
    "- First, tokenize the texts using `nlp.tokenizer()`. \n",
    "- Then, pass those docs to the TextCategorizer which you can get from `nlp.get_pipe()`. \n",
    "- Use the `textcat.predict()` method to get scores for each document, then choose the class with the highest score (probability) as the predicted class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c6e4d7ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:32.201020Z",
     "iopub.status.busy": "2021-08-23T03:04:32.200356Z",
     "iopub.status.idle": "2021-08-23T03:04:35.731640Z",
     "shell.execute_reply": "2021-08-23T03:04:35.730956Z",
     "shell.execute_reply.started": "2021-08-23T02:28:13.111874Z"
    },
    "papermill": {
     "duration": 3.560514,
     "end_time": "2021-08-23T03:04:35.731797",
     "exception": false,
     "start_time": "2021-08-23T03:04:32.171283",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4_PredictFunction\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def predict(nlp, texts): \n",
    "    # Use the model's tokenizer to tokenize each input text\n",
    "    docs = [nlp.tokenizer(text) for text in texts]\n",
    "    \n",
    "    # Use textcat to get the scores for each doc\n",
    "    textcat = nlp.get_pipe('textcat')\n",
    "    scores, _ = textcat.predict(docs)\n",
    "    \n",
    "    # From the scores, find the class with the highest score/probability\n",
    "    predicted_class = scores.argmax(axis=1)\n",
    "    \n",
    "    return predicted_class\n",
    "\n",
    "# Check your answer\n",
    "step_4.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "597824f6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:35.791136Z",
     "iopub.status.busy": "2021-08-23T03:04:35.790513Z",
     "iopub.status.idle": "2021-08-23T03:04:35.793035Z",
     "shell.execute_reply": "2021-08-23T03:04:35.793484Z",
     "shell.execute_reply.started": "2021-08-23T02:28:24.266528Z"
    },
    "papermill": {
     "duration": 0.034375,
     "end_time": "2021-08-23T03:04:35.793644",
     "exception": false,
     "start_time": "2021-08-23T03:04:35.759269",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_4.hint()\n",
    "#step_4.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3f91601e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:35.857393Z",
     "iopub.status.busy": "2021-08-23T03:04:35.856692Z",
     "iopub.status.idle": "2021-08-23T03:04:35.860826Z",
     "shell.execute_reply": "2021-08-23T03:04:35.861295Z",
     "shell.execute_reply.started": "2021-08-23T02:29:26.668639Z"
    },
    "papermill": {
     "duration": 0.040341,
     "end_time": "2021-08-23T03:04:35.861463",
     "exception": false,
     "start_time": "2021-08-23T03:04:35.821122",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "POSITIVE: Came over and had their \"Pick 2\" lunch combo and chose their best selling 1/2 chicken sandwich with quinoa.  Both were tasty, the chicken salad is a bit creamy but was perfect with quinoa on the side.  This is a good lunch joint, casual and clean! \n",
      "\n",
      "POSITIVE: Went here last night and got oysters, fried okra, fries, and onion rings. I cannot complain. The portions were great and tasty!!! I will definitely be back for more. I cannot wait to try the crawfish boudin and soft shell crab. \n",
      "\n",
      "POSITIVE: This restaurant was fantastic! \n",
      "The concept of eating without vision was intriguing. The dinner was filled with laughs and good conversation. \n",
      "\n",
      "We were lead in a line to our table and each person to their seat. This was not just dark but you could not see something right in front of your face. \n",
      "\n",
      "The waiters/waitresses were all blind and allowed us to see how aware you need to be without the vision. \n",
      "\n",
      "Taking away one sense is said to increase your other senses so as taste and hearing which I believed to be true in this experience. \n",
      "\n",
      "The meal was extremely delicious. I had the chicken and it was cooked to perfection. I also had a surprise beer which was a nice surprise. \n",
      "\n",
      "The whole experience was unlike anything I have ever done and I hope this spreads to other cities. \n",
      "\n",
      "A must do! \n",
      "\n",
      "NEGATIVE: They won't book new patients for same day appointments. My dog is sick but it's not necessarily urgent so I asked when I would be able to book an appointment and was told \"new patients book out at least 6 weeks in advance\" so just a heads up this seems like a great vet from other reviews but it'll be hard to get in their system to know \n",
      "\n"
     ]
    }
   ],
   "source": [
    "texts = val_texts[34:38]\n",
    "predictions = predict(nlp, texts)\n",
    "\n",
    "for p, t in zip(predictions, texts):\n",
    "    print(f\"{textcat.labels[p]}: {t} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07bf4267",
   "metadata": {
    "papermill": {
     "duration": 0.027566,
     "end_time": "2021-08-23T03:04:35.917137",
     "exception": false,
     "start_time": "2021-08-23T03:04:35.889571",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "It looks like your model is working well after going through the data just once. However you need to calculate some metric for the model's performance on the hold-out validation data.\n",
    "\n",
    "# Step 5: Evaluate The Model\n",
    "\n",
    "Implement a function that evaluates a `TextCategorizer` model. This function `evaluate` takes a model along with texts and labels. It returns the accuracy of the model, which is the number of correct predictions divided by all predictions.\n",
    "\n",
    "First, use the `predict` method you wrote earlier to get the predicted class for each text in `texts`. Then, find where the predicted labels match the true \"gold-standard\" labels and calculate the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "54bed190",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:35.988684Z",
     "iopub.status.busy": "2021-08-23T03:04:35.987663Z",
     "iopub.status.idle": "2021-08-23T03:04:35.992147Z",
     "shell.execute_reply": "2021-08-23T03:04:35.992759Z",
     "shell.execute_reply.started": "2021-08-23T02:41:04.7443Z"
    },
    "papermill": {
     "duration": 0.047789,
     "end_time": "2021-08-23T03:04:35.992954",
     "exception": false,
     "start_time": "2021-08-23T03:04:35.945165",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 0])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict(nlp, texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "73d930d4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:36.073213Z",
     "iopub.status.busy": "2021-08-23T03:04:36.067454Z",
     "iopub.status.idle": "2021-08-23T03:04:39.544477Z",
     "shell.execute_reply": "2021-08-23T03:04:39.544967Z",
     "shell.execute_reply.started": "2021-08-23T02:47:58.526597Z"
    },
    "papermill": {
     "duration": 3.514594,
     "end_time": "2021-08-23T03:04:39.545144",
     "exception": false,
     "start_time": "2021-08-23T03:04:36.030550",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"5_EvaluateFunction\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def evaluate(model, texts, labels):\n",
    "    \"\"\" Returns the accuracy of a TextCategorizer model. \n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        model: ScaPy model with a TextCategorizer\n",
    "        texts: Text samples, from load_data function\n",
    "        labels: True labels, from load_data function\n",
    "    \n",
    "    \"\"\"\n",
    "    # Get predictions from textcat model (using your predict method)\n",
    "    predicted_class = predict(model, texts)\n",
    "    \n",
    "    # From labels, get the true class as a list of integers (POSITIVE -> 1, NEGATIVE -> 0)\n",
    "    true_class = [int(label['cats']['POSITIVE']) for label in labels]\n",
    "    \n",
    "    # A boolean or int array indicating correct predictions\n",
    "    correct_predictions = true_class == predicted_class\n",
    "    \n",
    "    # The accuracy, number of correct predictions divided by all predictions\n",
    "    accuracy = correct_predictions.mean()\n",
    "    \n",
    "    return accuracy\n",
    "\n",
    "step_5.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6b24d6fa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:39.607698Z",
     "iopub.status.busy": "2021-08-23T03:04:39.606740Z",
     "iopub.status.idle": "2021-08-23T03:04:39.610067Z",
     "shell.execute_reply": "2021-08-23T03:04:39.610548Z",
     "shell.execute_reply.started": "2021-08-23T02:48:20.457325Z"
    },
    "papermill": {
     "duration": 0.036666,
     "end_time": "2021-08-23T03:04:39.610704",
     "exception": false,
     "start_time": "2021-08-23T03:04:39.574038",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_5.hint()\n",
    "#step_5.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d95ed342",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:39.674186Z",
     "iopub.status.busy": "2021-08-23T03:04:39.673211Z",
     "iopub.status.idle": "2021-08-23T03:04:42.121635Z",
     "shell.execute_reply": "2021-08-23T03:04:42.122094Z",
     "shell.execute_reply.started": "2021-08-23T02:48:22.16738Z"
    },
    "papermill": {
     "duration": 2.48251,
     "end_time": "2021-08-23T03:04:42.122267",
     "exception": false,
     "start_time": "2021-08-23T03:04:39.639757",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9486\n"
     ]
    }
   ],
   "source": [
    "accuracy = evaluate(nlp, val_texts, val_labels)\n",
    "print(f\"Accuracy: {accuracy:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9b26ef7",
   "metadata": {
    "papermill": {
     "duration": 0.029201,
     "end_time": "2021-08-23T03:04:42.180920",
     "exception": false,
     "start_time": "2021-08-23T03:04:42.151719",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "With the functions implemented, you can train and evaluate in a loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d2596af9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:04:42.242870Z",
     "iopub.status.busy": "2021-08-23T03:04:42.241901Z",
     "iopub.status.idle": "2021-08-23T03:16:44.267828Z",
     "shell.execute_reply": "2021-08-23T03:16:44.268401Z"
    },
    "papermill": {
     "duration": 722.058472,
     "end_time": "2021-08-23T03:16:44.268582",
     "exception": false,
     "start_time": "2021-08-23T03:04:42.210110",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 4.498 \t Accuracy: 0.945\n",
      "Loss: 3.108 \t Accuracy: 0.946\n",
      "Loss: 2.348 \t Accuracy: 0.944\n",
      "Loss: 1.929 \t Accuracy: 0.945\n",
      "Loss: 1.591 \t Accuracy: 0.945\n"
     ]
    }
   ],
   "source": [
    "# This may take a while to run!\n",
    "n_iters = 5\n",
    "for i in range(n_iters):\n",
    "    losses = train(nlp, train_data, optimizer)\n",
    "    accuracy = evaluate(nlp, val_texts, val_labels)\n",
    "    print(f\"Loss: {losses['textcat']:.3f} \\t Accuracy: {accuracy:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e225574",
   "metadata": {
    "papermill": {
     "duration": 0.030829,
     "end_time": "2021-08-23T03:16:44.330508",
     "exception": false,
     "start_time": "2021-08-23T03:16:44.299679",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Step 6: Keep Improving\n",
    "\n",
    "You've built the necessary components to train a text classifier with spaCy. What could you do further to optimize the model?\n",
    "\n",
    "Run the next line to check your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6322c7cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-23T03:16:44.399683Z",
     "iopub.status.busy": "2021-08-23T03:16:44.398730Z",
     "iopub.status.idle": "2021-08-23T03:16:44.404835Z",
     "shell.execute_reply": "2021-08-23T03:16:44.404240Z"
    },
    "papermill": {
     "duration": 0.042316,
     "end_time": "2021-08-23T03:16:44.405000",
     "exception": false,
     "start_time": "2021-08-23T03:16:44.362684",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"6_ModelOptimizationQuestion\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "Answer: There are various hyperparameters to work with here. The biggest one is the TextCategorizer architecture. You used the simplest model which trains faster but likely has worse performance than the CNN and ensemble models. "
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "Answer: There are various hyperparameters to work with here. The biggest one is the TextCategorizer architecture. You used the simplest model which trains faster but likely has worse performance than the CNN and ensemble models. "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "#step_6.solution()\n",
    "step_6.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74c52efc",
   "metadata": {
    "papermill": {
     "duration": 0.031643,
     "end_time": "2021-08-23T03:16:44.468693",
     "exception": false,
     "start_time": "2021-08-23T03:16:44.437050",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Keep Going\n",
    "\n",
    "The next step is a big one. See how you can **[represent tokens as vectors that describe their meaning](https://www.kaggle.com/matleonard/word-vectors)**, and plug those into your machine learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf607cec",
   "metadata": {
    "papermill": {
     "duration": 0.031554,
     "end_time": "2021-08-23T03:16:44.532099",
     "exception": false,
     "start_time": "2021-08-23T03:16:44.500545",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161466) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 869.043294,
   "end_time": "2021-08-23T03:16:45.979490",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-08-23T03:02:16.936196",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}