Ch_08_SentimentAnalysis_TopicModelling_LDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f9df862e-4562-451a-81f6-9ed171230e86",
   "metadata": {},
   "source": [
    "***Reference:***\n",
    "\n",
    "***Raschka, Sebastian; Liu, Yuxi (Hayden); Mirjalili, Vahid. Machine Learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python. Packt Publishing.*** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2b29bd3-05b1-4767-b5af-ec2c7611c9e9",
   "metadata": {},
   "source": [
    "# ***<u>Chapter 8</u> - Applying Machine Learning to Sentiment Analysis***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36972a52-3836-41d2-b0cb-4e7a4dfcf717",
   "metadata": {
    "tags": []
   },
   "source": [
    "## NLP: Sentiment Analysis using IMDb Dataset\n",
    "\n",
    "**Sentiment analysis**, sometimes also called **opinion mining**, is a popular subdiscipline of the broader field of NLP; it is concerned with analyzing the sentiment of documents. A popular task in sentiment analysis is the classification of documents based on the expressed opinions or emotions of the authors with regard to a particular topic.\n",
    "\n",
    "\n",
    "The movie review dataset consists of 50,000 polar movie reviews that are labeled as either positive or negative; here, positive means that a movie was rated with more than six stars on IMDb, and negative means that a movie was rated with fewer than five stars on IMDb."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe3e5ce0-9779-49b3-afbb-f2ea3401d6d7",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Preprocessing the movie dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e841d914-fbae-4ebb-829b-4806f2dab64d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pyprind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "76a5aa61-57be-4cad-b72f-26976769c833",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pyprind\n",
    "# import pandas as pd\n",
    "# import os\n",
    "# import sys\n",
    "# from packaging import version\n",
    "\n",
    "\n",
    "# # change the `basepath` to the directory of the\n",
    "# # unzipped movie dataset\n",
    "\n",
    "# basepath = 'm_db'\n",
    "\n",
    "# labels = {'pos': 1, 'neg': 0}\n",
    "\n",
    "# # if the progress bar does not show, change stream=sys.stdout to stream=2\n",
    "# pbar = pyprind.ProgBar(50000, stream=2)\n",
    "\n",
    "# df = pd.DataFrame()\n",
    "# for s in ('test', 'train'):\n",
    "#     for l in ('pos', 'neg'):\n",
    "#         path = os.path.join(basepath, s, l)\n",
    "#         for file in sorted(os.listdir(path)):\n",
    "#             with open(os.path.join(path, file), \n",
    "#                       'r', encoding='utf-8') as infile:\n",
    "#                 txt = infile.read()\n",
    "                \n",
    "#             if version.parse(pd.__version__) >= version.parse(\"1.3.2\"):\n",
    "#                 x = pd.DataFrame([[txt, labels[l]]], columns=['review', 'sentiment'])\n",
    "#                 df = pd.concat([df, x], ignore_index=False)\n",
    "\n",
    "#             else:\n",
    "#                 df = df.append([[txt, labels[l]]], \n",
    "#                                ignore_index=True)\n",
    "#             pbar.update()\n",
    "            \n",
    "# df.columns = ['review', 'sentiment']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3faac3fc-6d1f-44f1-a1fa-1194801e8f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Shuffling the DataFrame:\n",
    "# import numpy as np\n",
    "\n",
    "\n",
    "# if version.parse(pd.__version__) >= version.parse(\"1.3.2\"):\n",
    "#     df = df.sample(frac=1, random_state=0).reset_index(drop=True)\n",
    "    \n",
    "# else:\n",
    "#     np.random.seed(0)\n",
    "#     df = df.reindex(np.random.permutation(df.index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "96744de2-d331-4ac9-8144-6fabf2d82d54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving the dataset\n",
    "# df.to_csv('movie_data.csv', index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "32a85d09-4a19-4786-90a4-ed9b315c642e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "movies_df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "\n",
    "# df = df.rename(columns={\"0\": \"review\", \"1\": \"sentiment\"})\n",
    "\n",
    "df = movies_df.copy()\n",
    "\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9a75c261-08a1-4022-ae70-1c5375d1a26c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 2)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe97a2f-9545-408e-b81f-4191eaf9283b",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Introducing the bag-of-words model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d663431-31f6-4b8d-832d-18f307978f87",
   "metadata": {},
   "source": [
    "**bag-of-words model, allows us to represent text as numerical feature vectors.**\n",
    "\n",
    "*The idea behind bag-of-words is quite simple:*\n",
    "\n",
    "1. We create a vocabulary of unique tokens—for example, words—from the entire set of documents. \n",
    "\n",
    "2. We construct a feature vector from each document that contains the counts of how often each word occurs in the particular document. \n",
    "\n",
    "*Since the unique words in each document represent only a small subset of all the words in the bag-of-words vocabulary, the feature vectors will mostly consist of zeros, which is why we call them* **sparse.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "172692b7-0f4b-41bc-9180-9e42f7b244fc",
   "metadata": {},
   "source": [
    "### Transforming documents into feature vectors\n",
    "\n",
    "Using ```CountVectorizer``` that takes an array of text data, which can be documents or sentences, and construct the bag-of-words model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "88d43348-2e1f-4cd0-ad02-b7c731fc957e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer()\n",
    "\n",
    "docs = np.array([\n",
    "        'The sun is shining',\n",
    "        'The weather is sweet',\n",
    "        'The sun is shining, the weather is sweet, and one and one is two'])\n",
    "\n",
    "bag = count.fit_transform(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1b981e4-b182-4d33-90a2-3ca57a82ae6f",
   "metadata": {},
   "source": [
    "Now let us print the contents of the vocabulary to get a better understanding of the underlying concepts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e9004a99-e370-4ac9-86f5-5a249b164477",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 6, 'sun': 4, 'is': 1, 'shining': 3, 'weather': 8, 'sweet': 5, 'and': 0, 'one': 2, 'two': 7}\n"
     ]
    }
   ],
   "source": [
    "print(count.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a375b938-adae-48e9-afe8-07db70b3ae48",
   "metadata": {},
   "source": [
    "As we see abovr, **the vocabulary is stored in a Python dictionary that maps the unique words to integer indices.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "debe3c76-84f7-4dea-9b45-66af70aecfe7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 0 1 1 0 1 0 0]\n",
      " [0 1 0 0 0 1 1 0 1]\n",
      " [2 3 2 1 1 1 2 1 1]]\n"
     ]
    }
   ],
   "source": [
    "# print the feature vectors\n",
    "\n",
    "print(bag.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca9d4f7a-bb7a-4d21-95c2-6979a2aaa2de",
   "metadata": {},
   "source": [
    "***Each index position in the feature vectors shown above corresponds to the integer values that are stored as dictionary items in the ```CountVectorizer``` vocabulary.*** \n",
    "\n",
    "For example, the first feature at index position ```0``` resembles the count of the word ```\"and\"```, which only occurs in the last document, and the word ```\"is\"``` at index position ```1``` (the 2nd feature in the document vectors) occurs in all three sentences. \n",
    "\n",
    "Those values in the feature vectors are also called the \n",
    "\n",
    "**raw term frequencies: tf (t,d)—the number of times a term t occurs in a document d.**\n",
    "\n",
    "\n",
    "In the bag-of-words model, the word or term order in a sentence or document does not matter. The order in which the term frequencies appear in the feature vector is derived from the vocabulary indices, which are usually assigned alphabetically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "638863b1-83b3-4b1d-89a2-0b4267255da3",
   "metadata": {},
   "source": [
    "### N-gram Models\n",
    "\n",
    "The sequence of items in the bag-of-words model that we just created is also called the **1-gram or unigram model**—each item or token in the vocabulary represents a single word. \n",
    "\n",
    "More generally, the contiguous sequences of items in NLP—words, letters, or symbols—are also called **n-grams**. \n",
    "\n",
    "*The choice of the number, n, in the n-gram model depends on the particular application;* for example, n-grams of size 3 and 4 yield good performances in the anti-spam filtering of email messages.\n",
    "\n",
    "***To summarize the concept of the n-gram representation, the 1-gram and 2-gram representations of our first document, “the sun is shining”, would be constructed as follows:***\n",
    "- 1-gram: “the”,  “sun”,  “is”,  “shining” \n",
    "- 2-gram: “the sun”,  “sun is”,  “is shining” \n",
    "- Similarly n-gram is n-unique combination of words in order in the document\n",
    "\n",
    "The ```CountVectorizer``` class allows us to use different n-gram models via its ```ngram_range``` parameter. While a 1-gram representation is used by default, for 2-gram representation, set, ```ngram_range=(2,2)```."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f93f72ff-a55d-48f0-98d3-8d1fffe903b1",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Assessing word relevancy via tf-idf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "379d76fa-a295-4657-933c-1ed186fc36b0",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### **term frequency-inverse document frequency(tf-idf)**\n",
    "\n",
    "- When we are analyzing text data, we often encounter words that occur across multiple documents from both classes. These frequently occurring words typically don’t contain useful or discriminatory information.\n",
    "\n",
    "- **tf-idf can be used to downweight these frequently occurring words in the feature vectors.** \n",
    "\n",
    "- **The tf-idf can be defined as the product of the term frequency and the inverse document frequency:** \n",
    "\n",
    "$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times \\text{idf}(t,d)$$\n",
    "\n",
    "- $\\text{tf}(t,d)$ - **term frequency**—***the number of times a term t occurs in a document d.***\n",
    "\n",
    "- $\\text{idf}(t,d)$ - **inverse document frequency**, calculated as\n",
    "\n",
    "$$\\text{idf}(t,d) = \\log\\frac{n_d}{1 + \\text{df}(d,t)}$$\n",
    "\n",
    "- $n_d$ = ***total # of documents***\n",
    "- $\\text{df}(d,t)$ - ***# of documents, $d$, that contains the term, $t$.***\n",
    "    \n",
    "\n",
    "Note that adding the constant 1 to the denominator is optional and serves the purpose of assigning a non-zero value to terms that occur in none of the training examples; the log is used to ensure that low document frequencies are not given too much weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "679f9a90-aa2f-449a-8d77-e46495dbcfc1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.   0.43 0.   0.56 0.56 0.   0.43 0.   0.  ]\n",
      " [0.   0.43 0.   0.   0.   0.56 0.43 0.   0.56]\n",
      " [0.5  0.45 0.5  0.19 0.19 0.19 0.3  0.25 0.19]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf = TfidfTransformer(use_idf=True,\n",
    "                         norm = 'l2',\n",
    "                         smooth_idf = True)\n",
    "\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "print(tfidf.fit_transform(count.fit_transform(docs)).toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a093f93-4f2a-401a-983b-d7fe968e342c",
   "metadata": {},
   "source": [
    "As we saw earlier, the word \"is\" had the largest term frequency in the 3rd document, being the most frequently occurring word. However, after transforming the same feature vector into tf-idfs, we see that the word \"is\" is now associated with a relatively small tf-idf (0.45) in document 3 since it is also contained in documents 1 and 2 and thus is unlikely to contain any useful, discriminatory information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ddc8838-e4d8-408e-9677-813543b5b916",
   "metadata": {},
   "source": [
    "#### $\\rightarrow$ Sklearn ```TfidfTransformer``` calculates tf-idf slightly differently:\n",
    "\n",
    "**In sklearn inverse document frequency(idf):**\n",
    "\n",
    "$$\\text{idf}(t,d) = \\log\\frac{1 + n_d}{1 + \\text{df}(d,t)}$$\n",
    "\n",
    "**Similarly tf-idf:**\n",
    "\n",
    "$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times (\\text{idf}(t,d) + 1)$$\n",
    "\n",
    "\n",
    "$\"+1\"$ in the previous eq. is due to setting ```smooth_idf=True```, which is helpful for assigning zero weight (that is, $idf(t, d) = log(1) = 0$) to terms that occur in all documents.\n",
    "\n",
    "\n",
    "Also, it is more typical to mormalize the raw term fequency $\\text{tf}(t,d)$ before calculating $\\text{tf-idf}(t,d)$. ```TfidfTransformer``` class by default uses L2-normalization(```norm='l2'```), returning a vector of length 1, as: $$v_{norm} = \\frac{v}{||v||_2}$$ \n",
    "\n",
    "-------------------------------------\n",
    "\n",
    "##### For example: Let's calculate the tf-idf of the word ```\"is\"``` in the 3rd document.\n",
    "\n",
    "3rd doc: ```'The sun is shining, the weather is sweet, and one and one is two'```\n",
    "\n",
    "$$\\text{term fequency of 'is'  in 3rd doc.} = \\text{tf}(\"is\",d_3) = 3$$\n",
    "\n",
    "$$\\text{df}(\"is\", d_3) = 3$$\n",
    "\n",
    "$$\\implies \\text{idf}(\"is\", d_3) = \\log \\frac{1+3}{1+3} = 0$$\n",
    "\n",
    "$$\\implies \\text{tf-idf}(\"is\",d_3) = 3 \\times (0+1) = 3$$\n",
    "\n",
    "\n",
    "Now, similarly repeating above for all terms in the 3rd doc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0f9d4b7c-c2ce-4909-a207-3b5e6eda0eaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf-idf of term \"is\" = 3.00\n"
     ]
    }
   ],
   "source": [
    "# for \"is\" in 3rd doc\n",
    "tf_is = 3\n",
    "n_docs = 3\n",
    "idf_is = np.log((n_docs+1) / (3+1))\n",
    "tfidf_is = tf_is * (idf_is + 1)\n",
    "print(f'tf-idf of term \"is\" = {tfidf_is:.2f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1372339e-9a93-47eb-a04d-319e521969e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.39, 3.  , 3.39, 1.29, 1.29, 1.29, 2.  , 1.69, 1.29])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = TfidfTransformer(use_idf=True, norm = None, smooth_idf = True)\n",
    "raw_tfidf = tfidf.fit_transform(count.fit_transform(docs)).toarray()[-1]\n",
    "raw_tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f90c95a-e929-4313-8722-59bb95fcf475",
   "metadata": {},
   "source": [
    "For all terms in the third document, tf-idf vectors is as: ```[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0, 1.69, 1.29]```. \n",
    "\n",
    "However, notice that the values in this feature vector are different from the values that we obtained from ```TfidfTransformer``` that we used previously. The final step in this tf-idf calculation is the L2-normalization, which can be applied as follows: \n",
    "\n",
    "$$\\text{tfi-df}(d_3)_{norm} = \\frac{[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]}{\\sqrt{[3.39^2, 3.0^2, 3.39^2, 1.29^2, 1.29^2, 1.29^2, 2.0^2 , 1.69^2, 1.29^2]}}$$\n",
    "\n",
    "$$=[0.5, 0.45, 0.5, 0.19, 0.19, 0.19, 0.3, 0.25, 0.19]$$\n",
    "\n",
    "$$\\Rightarrow \\text{tfi-df}_{norm}(\"is\", d3) = 0.45$$\n",
    "\n",
    "As you can see, the results now match the results returned by scikit-learn’s ```TfidfTransformer```. Thus, this is how sklearn implements tf-idf."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "72c13c0a-3f26-4999-8453-3bcbad065ea0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.5 , 0.45, 0.5 , 0.19, 0.19, 0.19, 0.3 , 0.25, 0.19])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = TfidfTransformer(use_idf=True, norm = 'l2', smooth_idf = True)\n",
    "raw_tfidf = tfidf.fit_transform(count.fit_transform(docs)).toarray()[-1]\n",
    "raw_tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1741d4d-0a0b-4688-8d25-67b95e6e0dcb",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Cleaning the text data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a1ffee-7778-458c-ae30-ff852a50f960",
   "metadata": {},
   "source": [
    "Before applying bag-of-words model, it is imp. to clean up the text data by stripping all the unwanted characters. \n",
    "\n",
    "Below we can see the text contains HTML markup as well as punctuation and other non-letter characters. HTML markup does not contain many useful semantics so we will remove those.\n",
    "\n",
    "Punctuation marks can represent useful, additional information in certain NLP contexts. However, for simplicity, remove all punctuation marks except for emoticon characters, such as :), since those are certainly useful for sentiment analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "603acab2-576f-438f-a415-0099b2ae6acc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven.<br /><br />Title (Brazil): Not Available'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'review'][-50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "9c883db0-96aa-481e-9bd6-1116b20d9f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def preprocessor(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    \n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',\n",
    "                           text)\n",
    "    \n",
    "    text = (re.sub('[\\W]+', ' ', text.lower()) +\n",
    "            ' '.join(emoticons).replace('-', ''))\n",
    "    \n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e7fde90-78e0-40b7-9caa-b28bfca28f1a",
   "metadata": {},
   "source": [
    "- Via the first regex, ```<[^>]*>```, in the code, we tried to remove all of the HTML markup from the movie reviews.\n",
    "\n",
    "- After we removed the HTML markup, we used a slightly more complex regex to find emoticons, which we temporarily stored as ```emoticons```.\n",
    "\n",
    "- Next, we removed all non-word characters from the text via the regex ```[\\W]+``` and converted the text into lowercase characters.\n",
    "\n",
    "- Eventually, we added the temporarily stored emoticons to the end of the processed document string. Additionally, we removed the nose character (- in :-)) from the emoticons for consistency."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecf9f156-d119-4fd2-8554-94796073a887",
   "metadata": {},
   "source": [
    "***Although the addition of the emoticon characters to the end of the cleaned document strings may not look like the most elegant approach, we must note that the order of the words doesn’t matter in our bag-of-words model if our vocabulary consists of only one-word tokens.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b3910464-1015-4c39-90c8-ef01ea53d797",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven.<br /><br />Title (Brazil): Not Available'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'review'][-50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b5cca333-e19e-4a66-9a2e-b4c831578b15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven title brazil not available'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(df.loc[0, 'review'][-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "a17b2660-c28f-4191-90eb-fbf2214a11a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is a test :) :( :)'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(\"</a>This :) is :( a test :-)!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "3924b2c0-fd4a-4d74-9629-b3ac2d24be3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying the preprocessor to our dataframe\n",
    "\n",
    "df['review'] = df['review'].apply(preprocessor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "10c1bcb2-c2a0-45c2-9f65-a9b128218cf8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>in 1974 the teenager martha moxley maggie grac...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ok so i really like kris kristofferson and his...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spoiler do not read this if you think about w...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>hi for all the people who have seen this wonde...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>i recently bought the dvd forgetting just how ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  in 1974 the teenager martha moxley maggie grac...          1\n",
       "1  ok so i really like kris kristofferson and his...          0\n",
       "2   spoiler do not read this if you think about w...          0\n",
       "3  hi for all the people who have seen this wonde...          1\n",
       "4  i recently bought the dvd forgetting just how ...          0"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcafc053-bc7c-42d0-9653-de9006dd13bc",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Processing documents into tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc3c97e6-627b-4ff9-9571-aa3c946215ec",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "**Tokenization is the process of tokenizing or splitting a string, text into a list of tokens. One can think of token as parts like a word is a token in a sentence, and a sentence is a token in a paragraph.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "043eea0e-8613-4d0c-bd12-c95e271923f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenizer(text):\n",
    "    return text.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "dbd0d345-ddbe-4d0e-8ced-b38c73697ca1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runners', 'like', 'running', 'and', 'thus', 'they', 'run']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e89b3a10-a675-4093-9227-d239ca13777e",
   "metadata": {},
   "source": [
    "### Stemming/ Porter stemmer algoritm\n",
    "In the context of tokenization, another useful technique is **word stemming**, ***which is the process of transforming a word into its root form. It allows us to map related words to the same stem.***\n",
    "\n",
    "\n",
    "So, like root word for:\n",
    "- running -> run\n",
    "- thus -> thu\n",
    "- and -> and\n",
    "- runners -> runner, etc.\n",
    "\n",
    "```nltk``` library implements potter stemming algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e168dbe9-6347-455a-aa77-667cb44f3410",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "porter = PorterStemmer()\n",
    "\n",
    "def tokenizer_porter(text):\n",
    "    return [porter.stem(word) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "a6274464-40bb-4c72-939e-d85f207c64d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'and', 'thu', 'they', 'run']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_porter('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5850ef50-5d49-492c-b117-5b9c13fb8d6d",
   "metadata": {},
   "source": [
    "**Using the ```PorterStemmer``` from the ```nltk``` package, we modified our tokenizer function to reduce words to their root form.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ad48747-b1a4-448d-adaa-47c32be3ee7c",
   "metadata": {},
   "source": [
    "### Lemmatization\n",
    "\n",
    "**While stemming can create non-real words**, such as ```'thu'``` (from ```'thus'```), as shown in the previous example, a technique called ***lemmatization aims to obtain the canonical (grammatically correct) forms of individual words—the so-called lemmas.*** \n",
    "\n",
    "***However, lemmatization is computationally more difficult and expensive compared to stemming and, in practice, it has been observed that stemming and lemmatization have little impact on the performance of text classification.***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1fd5ff6-a775-4cef-9c76-0fae5582d1be",
   "metadata": {},
   "source": [
    "### Diff. b/w Stemming & Lemmatization\n",
    "\n",
    "**Stemming identifies the common root form of a word by removing or replacing word suffixes (e.g. “flooding” is stemmed as “flood”), while lemmatization identifies the inflected forms of a word and returns its base form (e.g. “better” is lemmatized as “good”).**\n",
    "\n",
    "\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "    <td> <img src=\"Images/stem_lem_1.jpg\"  alt=\"1\"></td>\n",
    "    <td><img src=\"Images/stem_lem_2.png\" alt=\"2\"></td>   \n",
    "  </tr>\n",
    "</table>\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "    <td> <img src=\"Images/stem_lem_3.png\"  alt=\"1\"></td>\n",
    "    <td><img src=\"Images/stem_lem_4.png\" alt=\"2\"></td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ddd2074-d5af-4d94-bed0-35ac4765221f",
   "metadata": {},
   "source": [
    "### Stop Words Removal\n",
    "\n",
    "**Stop words are simply those words that are extremely common in all sorts of texts** and probably bear no (or only a little) useful information that can be used to distinguish between different classes of documents.*\n",
    "\n",
    "*Examples of stop words are *is, and, has, and like.*\n",
    "\n",
    "Removing stop words can be useful if we are working with raw or normalized term frequencies rather than tf-idfs, which already downweight the frequently occurring words.\n",
    "\n",
    "\n",
    "```nltk``` library has set of 127 English stop words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "15227a68-f409-49ad-9541-8cad3f546a7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import nltk\n",
    "\n",
    "# nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "4b647ce1-7d0d-47fc-9e86-98c83d0b5ece",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'run', 'lot']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop = stopwords.words(\"english\")\n",
    "\n",
    "[w for w in tokenizer_porter('a runner likes running and runs a lot')\n",
    " if w not in stop]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "471a1416-6917-4abb-af7b-3c4caee2d930",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Training a logistic regression model for document classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "d933173f-ec53-4f0b-9d2b-d6350ca18267",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df.loc[:25000, 'review'].values\n",
    "y_train = df.loc[:25000, 'sentiment'].values\n",
    "X_test = df.loc[25000:, 'review'].values\n",
    "y_test = df.loc[25000:, 'sentiment'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "19753312-22b0-4d34-90ef-27b6f2cc3069",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "tfidf = TfidfVectorizer(strip_accents=None,\n",
    "                        lowercase=False,\n",
    "                        preprocessor=None)\n",
    "\n",
    "\"\"\"\n",
    "param_grid = [{'vect__ngram_range': [(1, 1)],\n",
    "               'vect__stop_words': [stop, None],\n",
    "               'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "               'clf__penalty': ['l1', 'l2'],\n",
    "               'clf__C': [1.0, 10.0, 100.0]},\n",
    "              {'vect__ngram_range': [(1, 1)],\n",
    "               'vect__stop_words': [stop, None],\n",
    "               'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "               'vect__use_idf':[False],\n",
    "               'vect__norm':[None],\n",
    "               'clf__penalty': ['l1', 'l2'],\n",
    "               'clf__C': [1.0, 10.0, 100.0]},\n",
    "              ]\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "small_param_grid = [{'vect__ngram_range': [(1, 1)],\n",
    "                     'vect__stop_words': [None],\n",
    "                     'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "                     'clf__penalty': ['l2'],\n",
    "                     'clf__C': [1.0, 10.0]},\n",
    "                    {'vect__ngram_range': [(1, 1)],\n",
    "                     'vect__stop_words': [stop, None],\n",
    "                     'vect__tokenizer': [tokenizer],\n",
    "                     'vect__use_idf':[False],\n",
    "                     'vect__norm':[None],\n",
    "                     'clf__penalty': ['l1','l2'],\n",
    "                  'clf__C': [1.0, 10.0]},\n",
    "              ]\n",
    "\n",
    "\n",
    "lr_tfidf = Pipeline([('vect', tfidf),\n",
    "                     ('clf', LogisticRegression(solver='liblinear'))])\n",
    "\n",
    "gs_lr_tfidf = GridSearchCV(lr_tfidf, small_param_grid,\n",
    "                           scoring='accuracy',\n",
    "                           cv=5,\n",
    "                           verbose=2,\n",
    "                           n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "cee77096-878d-4bb6-9950-2f64cc706556",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 12 candidates, totalling 60 fits\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                                        TfidfVectorizer(lowercase=False)),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        LogisticRegression(solver=&#x27;liblinear&#x27;))]),\n",
       "             n_jobs=-1,\n",
       "             param_grid=[{&#x27;clf__C&#x27;: [1.0, 10.0], &#x27;clf__penalty&#x27;: [&#x27;l2&#x27;],\n",
       "                          &#x27;vect__ngram_range&#x27;: [(1, 1)],\n",
       "                          &#x27;vect__stop_words&#x27;: [None],\n",
       "                          &#x27;vect__tokenizer&#x27;: [&lt;function tokenizer at 0x00000233B25D03A0&gt;,\n",
       "                                              &lt;function tokenizer_porter at 0x00000233...\n",
       "                          &#x27;vect__stop_words&#x27;: [[&#x27;i&#x27;, &#x27;me&#x27;, &#x27;my&#x27;, &#x27;myself&#x27;, &#x27;we&#x27;,\n",
       "                                                &#x27;our&#x27;, &#x27;ours&#x27;, &#x27;ourselves&#x27;,\n",
       "                                                &#x27;you&#x27;, &quot;you&#x27;re&quot;, &quot;you&#x27;ve&quot;,\n",
       "                                                &quot;you&#x27;ll&quot;, &quot;you&#x27;d&quot;, &#x27;your&#x27;,\n",
       "                                                &#x27;yours&#x27;, &#x27;yourself&#x27;,\n",
       "                                                &#x27;yourselves&#x27;, &#x27;he&#x27;, &#x27;him&#x27;,\n",
       "                                                &#x27;his&#x27;, &#x27;himself&#x27;, &#x27;she&#x27;,\n",
       "                                                &quot;she&#x27;s&quot;, &#x27;her&#x27;, &#x27;hers&#x27;,\n",
       "                                                &#x27;herself&#x27;, &#x27;it&#x27;, &quot;it&#x27;s&quot;, &#x27;its&#x27;,\n",
       "                                                &#x27;itself&#x27;, ...],\n",
       "                                               None],\n",
       "                          &#x27;vect__tokenizer&#x27;: [&lt;function tokenizer at 0x00000233B25D03A0&gt;],\n",
       "                          &#x27;vect__use_idf&#x27;: [False]}],\n",
       "             scoring=&#x27;accuracy&#x27;, verbose=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[(&#x27;vect&#x27;,\n",
       "                                        TfidfVectorizer(lowercase=False)),\n",
       "                                       (&#x27;clf&#x27;,\n",
       "                                        LogisticRegression(solver=&#x27;liblinear&#x27;))]),\n",
       "             n_jobs=-1,\n",
       "             param_grid=[{&#x27;clf__C&#x27;: [1.0, 10.0], &#x27;clf__penalty&#x27;: [&#x27;l2&#x27;],\n",
       "                          &#x27;vect__ngram_range&#x27;: [(1, 1)],\n",
       "                          &#x27;vect__stop_words&#x27;: [None],\n",
       "                          &#x27;vect__tokenizer&#x27;: [&lt;function tokenizer at 0x00000233B25D03A0&gt;,\n",
       "                                              &lt;function tokenizer_porter at 0x00000233...\n",
       "                          &#x27;vect__stop_words&#x27;: [[&#x27;i&#x27;, &#x27;me&#x27;, &#x27;my&#x27;, &#x27;myself&#x27;, &#x27;we&#x27;,\n",
       "                                                &#x27;our&#x27;, &#x27;ours&#x27;, &#x27;ourselves&#x27;,\n",
       "                                                &#x27;you&#x27;, &quot;you&#x27;re&quot;, &quot;you&#x27;ve&quot;,\n",
       "                                                &quot;you&#x27;ll&quot;, &quot;you&#x27;d&quot;, &#x27;your&#x27;,\n",
       "                                                &#x27;yours&#x27;, &#x27;yourself&#x27;,\n",
       "                                                &#x27;yourselves&#x27;, &#x27;he&#x27;, &#x27;him&#x27;,\n",
       "                                                &#x27;his&#x27;, &#x27;himself&#x27;, &#x27;she&#x27;,\n",
       "                                                &quot;she&#x27;s&quot;, &#x27;her&#x27;, &#x27;hers&#x27;,\n",
       "                                                &#x27;herself&#x27;, &#x27;it&#x27;, &quot;it&#x27;s&quot;, &#x27;its&#x27;,\n",
       "                                                &#x27;itself&#x27;, ...],\n",
       "                                               None],\n",
       "                          &#x27;vect__tokenizer&#x27;: [&lt;function tokenizer at 0x00000233B25D03A0&gt;],\n",
       "                          &#x27;vect__use_idf&#x27;: [False]}],\n",
       "             scoring=&#x27;accuracy&#x27;, verbose=2)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;vect&#x27;, TfidfVectorizer(lowercase=False)),\n",
       "                (&#x27;clf&#x27;, LogisticRegression(solver=&#x27;liblinear&#x27;))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer(lowercase=False)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(solver=&#x27;liblinear&#x27;)</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('vect',\n",
       "                                        TfidfVectorizer(lowercase=False)),\n",
       "                                       ('clf',\n",
       "                                        LogisticRegression(solver='liblinear'))]),\n",
       "             n_jobs=-1,\n",
       "             param_grid=[{'clf__C': [1.0, 10.0], 'clf__penalty': ['l2'],\n",
       "                          'vect__ngram_range': [(1, 1)],\n",
       "                          'vect__stop_words': [None],\n",
       "                          'vect__tokenizer': [<function tokenizer at 0x00000233B25D03A0>,\n",
       "                                              <function tokenizer_porter at 0x00000233...\n",
       "                          'vect__stop_words': [['i', 'me', 'my', 'myself', 'we',\n",
       "                                                'our', 'ours', 'ourselves',\n",
       "                                                'you', \"you're\", \"you've\",\n",
       "                                                \"you'll\", \"you'd\", 'your',\n",
       "                                                'yours', 'yourself',\n",
       "                                                'yourselves', 'he', 'him',\n",
       "                                                'his', 'himself', 'she',\n",
       "                                                \"she's\", 'her', 'hers',\n",
       "                                                'herself', 'it', \"it's\", 'its',\n",
       "                                                'itself', ...],\n",
       "                                               None],\n",
       "                          'vect__tokenizer': [<function tokenizer at 0x00000233B25D03A0>],\n",
       "                          'vect__use_idf': [False]}],\n",
       "             scoring='accuracy', verbose=2)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_lr_tfidf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d17bba00-1246-465c-9d57-0428750125aa",
   "metadata": {},
   "source": [
    "- In the previous code example, we replaced ```CountVectorizer``` and ```TfidfTransformer``` from the previous subsection with ```TfidfVectorizer```, which combines ```CountVectorizer``` with the ```TfidfTransformer```. \n",
    "\n",
    "\n",
    "- Our ```param_grid``` consisted of two parameter dictionaries. \n",
    "\n",
    "    - In the first dictionary, we used ```TfidfVectorizer``` **with its default settings** (```use_idf=True```, ```smooth_idf=True```, and ```norm='l2'```) to calculate the tf-idfs; \n",
    "\n",
    "    - in the second dictionary, we set those parameters to ```use_idf=False```, ```smooth_idf=False```, and ```norm=None``` **in order to train a model based on raw term frequencies.** \n",
    "\n",
    "- Furthermore, for the logistic regression classifier itself, we trained models using L2 regularization via the penalty parameter and compared different regularization strengths by defining a range of values for the inverse-regularization parameter ```C```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "d37a00f0-9562-4e8c-9a59-982423d26be8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameter set: {'clf__C': 10.0, 'clf__penalty': 'l2', 'vect__ngram_range': (1, 1), 'vect__stop_words': None, 'vect__tokenizer': <function tokenizer at 0x00000233B25D03A0>}\n",
      "CV Accuracy: 0.897\n"
     ]
    }
   ],
   "source": [
    "print(f'Best parameter set: {gs_lr_tfidf.best_params_}')\n",
    "print(f'CV Accuracy: {gs_lr_tfidf.best_score_:.3f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "baa306a7-3a71-49d6-bdb2-f55d4e3498f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.899\n"
     ]
    }
   ],
   "source": [
    "clf = gs_lr_tfidf.best_estimator_\n",
    "print(f'Test Accuracy: {clf.score(X_test, y_test):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "920b50fc-815e-41d2-bb74-eea228784900",
   "metadata": {},
   "source": [
    "### The naïve Bayes classifier\n",
    "\n",
    "A still very popular classifier for text classification is the naïve Bayes classifier, which gained popularity in applications of email spam filtering. **Naïve Bayes classifiers** are easy to implement, computationally efficient, and tend to **perform particularly well on relatively small datasets** compared to other algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a311baf0-f167-40a1-9d7a-b18b7eda7771",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Working with bigger data - online algorithms and out-of-core learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe698db-b663-43f8-b0b0-12f750f734d1",
   "metadata": {},
   "source": [
    "**out-of-core learning**, allows us to work with such large datasets by fitting the classifier incrementally on smaller batches of a dataset.\n",
    "\n",
    "\n",
    "**Stochastic Gradient Descent**: ***it is an optimization algorithm that updates the model’s weights using one example at a time.*** \n",
    "\n",
    "We will make use of the ```partial_fit``` function of ```SGDClassifier``` in scikit-learn to stream the documents directly from our local drive and train a logistic regression model using small mini-batches of documents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "3ac9a496-3bd4-4652-a996-e7e00750f7f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a tokenizer func. that cleans the unprocessed text data \n",
    "\n",
    "import numpy as np\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop = stopwords.words(\"english\")\n",
    "\n",
    "def tokenizer(text):\n",
    "    # substituting/Removing any html tag elments alongs with it's contents\n",
    "    # in our text\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    \n",
    "    # getting all emotions signs\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)', \n",
    "                           text.lower())\n",
    "    \n",
    "    # removing all emoticons and appending at the end, also removing the\n",
    "    # nose '-' symbol in ':-)' from consistensy\n",
    "    text = re.sub('[\\W]+', ' ', text.lower()) +\\\n",
    "        ' '.join(emoticons).replace('-', '')\n",
    "    \n",
    "    # tokenization\n",
    "    tokenized = [w for w in text.split() if w not in stop]\n",
    "    \n",
    "    return tokenized\n",
    "\n",
    "\n",
    "# define a generator func., stream_docs, that reads in and \n",
    "# returns one document at a time:\n",
    "def stream_docs(path):\n",
    "    with open(path, 'r', encoding='utf-8') as csv:\n",
    "        next(csv)  # skip header\n",
    "        for line in csv:\n",
    "            text, label = line[:-3], int(line[-2])\n",
    "            yield text, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5bdb0905-09c0-4226-bcc2-bc1950895a64",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('\"In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70\\'s, they discover the criminal and a net of power and money to cover the murder.<br /><br />\"\"Murder in Greenwich\"\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.<br /><br />Title (Brazil): Not Available\"',\n",
       " 1)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(stream_docs(path='movie_data.csv'))\n",
    "\n",
    "# stream_docs works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "4e85dc20-b81e-43fe-9305-7f8f950a9016",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a func., get_minibatch, that will take a document stream\n",
    "# from the stream_docs func. and return a particular number of \n",
    "# documents specified by the size parameter:\n",
    "\n",
    "def get_minibatch(doc_stream, size):\n",
    "    docs, y = [], []\n",
    "    try:\n",
    "        for _ in range(size):\n",
    "            text, label = next(doc_stream)\n",
    "            docs.append(text)\n",
    "            y.append(label)\n",
    "    except StopIteration:\n",
    "        return None, None\n",
    "    return docs, y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0d99fd-35c2-4d83-8174-0cfeda493076",
   "metadata": {},
   "source": [
    "Unfortunately, we can’t use ```CountVectorizer``` for out-of-core learning since it requires holding the complete vocabulary in memory. Also, ```TfidfVectorizer``` needs to keep all the feature vectors of the training dataset in memory to calculate the inverse document frequencies. \n",
    "\n",
    "However, another useful vectorizer for text processing implemented in scikit-learn is ```HashingVectorizer```. ```HashingVectorizer``` is data-independent and makes use of the hashing trick via the 32-bit ```MurmurHash3``` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "d94b1500-546d-4966-aa57-6131e03e87d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "\n",
    "vect = HashingVectorizer(decode_error='ignore', \n",
    "                         n_features=2**21,\n",
    "                         preprocessor=None, \n",
    "                         tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "985ede1b-0fbc-43d0-8774-33992408f1fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from distutils.version import LooseVersion as Version\n",
    "from sklearn import __version__ as sklearn_version\n",
    "\n",
    "# loss='log_loss' for logistic regression\n",
    "clf = SGDClassifier(loss='log_loss', random_state=1)\n",
    "\n",
    "\n",
    "doc_stream = stream_docs(path='movie_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "2c2b9501-3d25-4f79-8737-253f9b0cc136",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:00:18\n"
     ]
    }
   ],
   "source": [
    "# start out-of-core learning \n",
    "\n",
    "import pyprind\n",
    "pbar = pyprind.ProgBar(45)\n",
    "\n",
    "classes = np.array([0,1])\n",
    "\n",
    "for _ in range(45):\n",
    "    X_train, y_train = get_minibatch(doc_stream, size=1000)\n",
    "    \n",
    "    if not X_train:\n",
    "        break\n",
    "        \n",
    "    X_train = vect.transform(X_train)\n",
    "    clf.partial_fit(X_train, y_train, classes=classes)\n",
    "    pbar.update()\n",
    "    \n",
    "    \n",
    "# we iterated over 45 mini-batches of documents where each mini-batch\n",
    "# consists of 1,000 documents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "77ce08fb-c668-41dc-be02-f0175a6f6fd7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.868\n"
     ]
    }
   ],
   "source": [
    "# Having completed the incremental learning process, we will use the last \n",
    "# 5,000 documents to evaluate the performance of our model:\n",
    "\n",
    "X_test, y_test = get_minibatch(doc_stream, size=5000)\n",
    "\n",
    "X_test = vect.transform(X_test)\n",
    "\n",
    "print(f'Accuracy: {clf.score(X_test, y_test):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba9eb46-4820-4b4a-9dbe-f5a6cdafc592",
   "metadata": {},
   "source": [
    "As you can see, the accuracy of the model is approximately 87 percent, slightly below the accuracy that we achieved in the previous section using the grid search for hyperparameter tuning. However, out-of-core learning is very memory efficient, and it took less than a minute to complete. \n",
    "\n",
    "\n",
    "Finally, we can use the last 5,000 documents to update our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "73869fe7-c34d-446b-ae4d-15836c3a5938",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = clf.partial_fit(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec8feff-1693-4606-bb19-575cddde5ca3",
   "metadata": {},
   "source": [
    "### The word2vec model \n",
    "\n",
    "**A more modern alternative to the bag-of-words model is word2vec**, an algorithm by Google.\n",
    "\n",
    "The word2vec algorithm is an unsupervised learning algorithm based on neural networks that attempts to automatically learn the relationship between words.\n",
    "\n",
    "The idea behind word2vec is to put words that have similar meanings into similar clusters, and via clever vector spacing, the model can reproduce certain words using simple vector math, for example, king – man   woman = queen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8676f165-33ca-45b0-b428-8a1c5886d1af",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Topic modelling with Latent Dirichlet Allocation(LDA)\n",
    "<a href=\"https://youtu.be/djxXHg17oTA\">Pronunciation</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bef49ae-ee76-4199-bb24-670809eacb14",
   "metadata": {},
   "source": [
    "- **Topic modeling :** ***describes the broad task of assigning topics to unlabeled text documents.*** \n",
    "\n",
    "For example, a typical application is the categorization of documents in a large text corpus of newspaper articles. In applications of topic modeling, we then aim to assign category labels to those articles, for example, sports, finance, world news, politics, and local news. \n",
    "\n",
    "Topic modeling can be considered as a clustering task, a subcategory of unsupervised learning. \n",
    "\n",
    "- A popular technique for topic modeling called **latent Dirichlet allocation (LDA)**. \n",
    "\n",
    "However, note that while latent Dirichlet allocation is often abbreviated as LDA, it is not to be confused with linear discriminant analysis, a supervised dimensionality reduction technique."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eacf460-59b1-4679-a197-2ba08a119b64",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Decomposing text documents with LDA\n",
    "\n",
    "*Since the mathematics behind LDA is quite involved and requires knowledge of Bayesian inference, we will approach this topic from a practitioner’s perspective and interpret LDA using layman’s terms.*\n",
    "\n",
    "\n",
    "- ***LDA is a generative probabilistic model that tries to find groups of words that appear frequently together across different documents. These frequently appearing words represent our topics, assuming that each document is a mixture of different words.***\n",
    "\n",
    "- ***The input to an LDA is the bag-of-words model.*** Given a bag-of-words matrix as input, LDA decomposes it into two new matrices:\n",
    "\n",
    "    - **A document-to-topic matrix** \n",
    "    - **A word-to-topic matrix** \n",
    "    \n",
    "\n",
    "- LDA decomposes the bag-of-words matrix in such a way that if we multiply those two matrices together, we will be able to reproduce the input, the bag-of-words matrix, with the lowest possible error. \n",
    "\n",
    "- *In practice, we are interested in those topics that LDA found in the bag-of-words matrix.* \n",
    "\n",
    "- ***The only downside may be that we must define the number of topics beforehand—the number of topics is a hyperparameter of LDA that has to be specified manually.***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f80f01f-c7e0-43c5-8c19-012e51f8cbab",
   "metadata": {},
   "source": [
    "### <a href=\"https://towardsdatascience.com/latent-dirichlet-allocation-lda-9d1cd064ffa2\">A Beginner’s Guide to Latent Dirichlet Allocation(LDA)</a>\n",
    "\n",
    "https://towardsdatascience.com/latent-dirichlet-allocation-lda-9d1cd064ffa2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "980728d7-5d96-4652-9554-9ae43e5d5612",
   "metadata": {},
   "source": [
    "### LDA with scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "e431851f-47ed-4079-a7a3-6ecd4b15fbf8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>hi for all the people who have seen this wonde...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I recently bought the DVD, forgetting just how...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0\n",
       "3  hi for all the people who have seen this wonde...          1\n",
       "4  I recently bought the DVD, forgetting just how...          0"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "7bfb9a16-a13a-4838-b97b-3d816c347cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using CountVectorizer to create the bag-of-words matrix \n",
    "# as input to the LDA.\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer(stop_words='english', \n",
    "                        max_df=.1, \n",
    "                        max_features=5000)\n",
    "\n",
    "X = count.fit_transform(df['review'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c26bc7c-3751-4de1-a26f-34006d974f21",
   "metadata": {},
   "source": [
    "we set the maximum document frequency of words to be considered to 10% (```max_df=.1```) to exclude words that occur too frequently across documents.\n",
    "\n",
    "The rationale behind the removal of frequently occurring words is that these might be common words appearing across all documents that are, therefore, less likely to be associated with a specific topic category of a given document.\n",
    "\n",
    "Also, we limited the number of words to be considered to the most frequently occurring 5,000 words (```max_features=5000```), to limit the dimensionality of this dataset to improve the inference performed by LDA. \n",
    "\n",
    "However, both ```max_df=.1``` and ```max_features=5000``` are hyperparameter values chosen arbitrarily, and can be tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "f75ee6ba-d7cd-4da4-a684-b30f017a48a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "# performing LDA with 10 topics\n",
    "lda = LatentDirichletAllocation(n_components=10,\n",
    "                                random_state=123,\n",
    "                                learning_method='batch')\n",
    "\n",
    "X_topics = lda.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba7b8cf4-37f8-4a4f-bc53-0ae9fbaef3a8",
   "metadata": {},
   "source": [
    "By setting ```learning_method='batch'```, we let the lda estimator do its estimation based on all available training data (the bag-of-words matrix) in one iteration, which is slower than the alternative ```'online'``` learning method, but can lead to more accurate results (setting ```learning_method='online'``` is analogous to online or mini-batch learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d6d08a4-f38f-4457-9548-27621e1d1267",
   "metadata": {},
   "source": [
    "#### Expectation-maximization \n",
    "The scikit-learn library’s implementation of LDA uses the expectation-maximization <a href=\"https://youtu.be/1jSonYih_sM\">(EM) algorithm</a> to update its parameter estimates iteratively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "211ebaa7-f817-4a30-b315-3a455ec22ad7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 5000)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.components_.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f809f5d-7a10-4b30-ae98-05ee7f999d25",
   "metadata": {},
   "source": [
    "To analyze the results, let’s print the five most important words for each of the 10 topics. Note that the word importance values are ranked in increasing order. Thus, to print the top five words, we need to sort the topic array in reverse order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "e0e0d13e-469d-42d3-acfc-0d1d78494572",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic 1:\n",
      "worst minutes awful script stupid\n",
      "Topic 2:\n",
      "family mother father children girl\n",
      "Topic 3:\n",
      "american war dvd music tv\n",
      "Topic 4:\n",
      "human audience cinema art sense\n",
      "Topic 5:\n",
      "police guy car dead murder\n",
      "Topic 6:\n",
      "horror house sex girl woman\n",
      "Topic 7:\n",
      "role performance comedy actor performances\n",
      "Topic 8:\n",
      "series episode war episodes tv\n",
      "Topic 9:\n",
      "book version original read novel\n",
      "Topic 10:\n",
      "action fight guy guys cool\n"
     ]
    }
   ],
   "source": [
    "n_top_words = 5\n",
    "feature_names = count.get_feature_names_out()\n",
    "\n",
    "for topic_idx, topic in enumerate(lda.components_):\n",
    "    print(f'Topic {(topic_idx + 1)}:')\n",
    "    print(' '.join([feature_names[i]\n",
    "                    for i in topic.argsort()[:-n_top_words - 1:-1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5b17417-54cb-4b42-9820-8183ae3119cc",
   "metadata": {},
   "source": [
    "Based on reading the five most important words for each topic, you may guess that the LDA identified the following topics: \n",
    "\n",
    "1. Generally bad movies (not really a topic category) \n",
    "2. Movies about families \n",
    "3. War movies \n",
    "4. Art movies \n",
    "5. Crime movies \n",
    "6. Horror movies \n",
    "7. Comedy movie reviews \n",
    "8. Movies somehow related to TV shows \n",
    "9. Movies based on books \n",
    "10. Action movies\n",
    "\n",
    "\n",
    "To confirm that the categories make sense based on the reviews, let’s plot three movies from the horror movie category (horror movies belong to category 6 at index position ```5```):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "b4d03e33-fa9a-4298-9236-3a949dbc0c92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Horror movie #1:\n",
      "House of Dracula works from the same basic premise as House of Frankenstein from the year before; namely that Universal's three most famous monsters; Dracula, Frankenstein's Monster and The Wolf Man are appearing in the movie together. Naturally, the film is rather messy therefore, but the fact that ...\n",
      "\n",
      "Horror movie #2:\n",
      "Okay, what the hell kind of TRASH have I been watching now? \"The Witches' Mountain\" has got to be one of the most incoherent and insane Spanish exploitation flicks ever and yet, at the same time, it's also strangely compelling. There's absolutely nothing that makes sense here and I even doubt there  ...\n",
      "\n",
      "Horror movie #3:\n",
      "<br /><br />Horror movie time, Japanese style. Uzumaki/Spiral was a total freakfest from start to finish. A fun freakfest at that, but at times it was a tad too reliant on kitsch rather than the horror. The story is difficult to summarize succinctly: a carefree, normal teenage girl starts coming fac ...\n"
     ]
    }
   ],
   "source": [
    "horror = X_topics[:, 5].argsort()[::-1]\n",
    "\n",
    "for iter_idx, movie_idx in enumerate(horror[:3]):\n",
    "    print(f'\\nHorror movie #{(iter_idx + 1)}:')\n",
    "    print(df['review'][movie_idx][:300], '...')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f41c8afb-b7ea-4a33-8d49-419b27226700",
   "metadata": {},
   "source": [
    "Using the preceding code example, we printed the first 300 characters from the top three horror movies. The reviews—even though we don’t know which exact movie they belong to—sound like reviews of horror movies (however, one might argue that ```Horror movie #2``` could also be a goot fit for topic category 1: Generally bad movies)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "098a81d2-8082-4e09-9f1d-e03fadc51220",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "- We learned how to use machine learning algorithms to classify text documents based on their polarity, which is a basic task in sentiment analysis in the field of NLP. \n",
    "\n",
    "- bag-of-words model - to encode a document as a feature vector\n",
    "\n",
    "- tf-idf - to weight the term frequency by relevance using. \n",
    "\n",
    "- Working with text data can be computationally quite expensive due to the large feature vectors that are created during this process; **out-of-core or incremental learning** is used to train a machine learning algorithm without loading the whole dataset into a computer’s memory. \n",
    "\n",
    "- Lastly, the concept of **topic modeling using LDA** to categorize the movie reviews into different categories in an unsupervised fashion. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1980948-94be-4638-8db6-de99555e7fdf",
   "metadata": {},
   "source": [
    "***Reference:***\n",
    "\n",
    "***Raschka, Sebastian; Liu, Yuxi (Hayden); Mirjalili, Vahid. Machine Learning with PyTorch and Scikit-Learn: Develop machine learning and deep learning models with Python. Packt Publishing.*** "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}