4-Topic-Modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another popular text analysis technique is called topic modeling. The ultimate goal of topic modeling is to find various topics that are present in your corpus. Each document in the corpus will be made up of at least one topic, if not multiple topics.\n",
    "\n",
    "In this notebook, we will be covering the steps on how to do **Latent Dirichlet Allocation (LDA)**, which is one of many topic modeling techniques. It was specifically designed for text data.\n",
    "\n",
    "To use a topic modeling technique, you need to provide (1) a document-term matrix and (2) the number of topics you would like the algorithm to pick up.\n",
    "\n",
    "Once the topic modeling technique is applied, your job as a human is to interpret the results and see if the mix of words in each topic make sense. If they don't make sense, you can try changing up the number of topics, the terms in the document-term matrix, model parameters, or even try a different model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling - Attempt #1 (All Text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aaaaah</th>\n",
       "      <th>aaaaahhhhhhh</th>\n",
       "      <th>aaaaauuugghhhhhh</th>\n",
       "      <th>aaaahhhhh</th>\n",
       "      <th>aaah</th>\n",
       "      <th>aah</th>\n",
       "      <th>abc</th>\n",
       "      <th>abcs</th>\n",
       "      <th>ability</th>\n",
       "      <th>abject</th>\n",
       "      <th>...</th>\n",
       "      <th>zee</th>\n",
       "      <th>zen</th>\n",
       "      <th>zeppelin</th>\n",
       "      <th>zero</th>\n",
       "      <th>zillion</th>\n",
       "      <th>zombie</th>\n",
       "      <th>zombies</th>\n",
       "      <th>zoning</th>\n",
       "      <th>zoo</th>\n",
       "      <th>éclair</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 7444 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         aaaaah  aaaaahhhhhhh  aaaaauuugghhhhhh  aaaahhhhh  aaah  aah  abc  \\\n",
       "ali           0             0                 0          0     0    0    1   \n",
       "anthony       0             0                 0          0     0    0    0   \n",
       "bill          1             0                 0          0     0    0    0   \n",
       "bo            0             1                 1          1     0    0    0   \n",
       "dave          0             0                 0          0     1    0    0   \n",
       "hasan         0             0                 0          0     0    0    0   \n",
       "jim           0             0                 0          0     0    0    0   \n",
       "joe           0             0                 0          0     0    0    0   \n",
       "john          0             0                 0          0     0    0    0   \n",
       "louis         0             0                 0          0     0    3    0   \n",
       "mike          0             0                 0          0     0    0    0   \n",
       "ricky         0             0                 0          0     0    0    0   \n",
       "\n",
       "         abcs  ability  abject  ...  zee  zen  zeppelin  zero  zillion  \\\n",
       "ali         0        0       0  ...    0    0         0     0        0   \n",
       "anthony     0        0       0  ...    0    0         0     0        0   \n",
       "bill        1        0       0  ...    0    0         0     1        1   \n",
       "bo          0        1       0  ...    0    0         0     1        0   \n",
       "dave        0        0       0  ...    0    0         0     0        0   \n",
       "hasan       0        0       0  ...    2    1         0     1        0   \n",
       "jim         0        0       0  ...    0    0         0     0        0   \n",
       "joe         0        0       0  ...    0    0         0     0        0   \n",
       "john        0        0       0  ...    0    0         0     0        0   \n",
       "louis       0        0       0  ...    0    0         0     2        0   \n",
       "mike        0        0       0  ...    0    0         2     1        0   \n",
       "ricky       0        1       1  ...    0    0         0     0        0   \n",
       "\n",
       "         zombie  zombies  zoning  zoo  éclair  \n",
       "ali           1        0       0    0       0  \n",
       "anthony       0        0       0    0       0  \n",
       "bill          1        1       1    0       0  \n",
       "bo            0        0       0    0       0  \n",
       "dave          0        0       0    0       0  \n",
       "hasan         0        0       0    0       0  \n",
       "jim           0        0       0    0       0  \n",
       "joe           0        0       0    0       0  \n",
       "john          0        0       0    0       1  \n",
       "louis         0        0       0    0       0  \n",
       "mike          0        0       0    0       0  \n",
       "ricky         0        0       0    1       0  \n",
       "\n",
       "[12 rows x 7444 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's read in our document-term matrix\n",
    "import pandas as pd\n",
    "import pickle\n",
    "\n",
    "data = pd.read_pickle('dtm_stop.pkl')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the necessary modules for LDA with gensim\n",
    "# Terminal / Anaconda Navigator: conda install -c conda-forge gensim\n",
    "from gensim import matutils, models\n",
    "import scipy.sparse\n",
    "\n",
    "# import logging\n",
    "# logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ali</th>\n",
       "      <th>anthony</th>\n",
       "      <th>bill</th>\n",
       "      <th>bo</th>\n",
       "      <th>dave</th>\n",
       "      <th>hasan</th>\n",
       "      <th>jim</th>\n",
       "      <th>joe</th>\n",
       "      <th>john</th>\n",
       "      <th>louis</th>\n",
       "      <th>mike</th>\n",
       "      <th>ricky</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aaaaah</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaaaahhhhhhh</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaaaauuugghhhhhh</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaaahhhhh</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aaah</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  ali  anthony  bill  bo  dave  hasan  jim  joe  john  louis  \\\n",
       "aaaaah              0        0     1   0     0      0    0    0     0      0   \n",
       "aaaaahhhhhhh        0        0     0   1     0      0    0    0     0      0   \n",
       "aaaaauuugghhhhhh    0        0     0   1     0      0    0    0     0      0   \n",
       "aaaahhhhh           0        0     0   1     0      0    0    0     0      0   \n",
       "aaah                0        0     0   0     1      0    0    0     0      0   \n",
       "\n",
       "                  mike  ricky  \n",
       "aaaaah               0      0  \n",
       "aaaaahhhhhhh         0      0  \n",
       "aaaaauuugghhhhhh     0      0  \n",
       "aaaahhhhh            0      0  \n",
       "aaah                 0      0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# One of the required inputs is a term-document matrix\n",
    "tdm = data.transpose()\n",
    "tdm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're going to put the term-document matrix into a new gensim format, from df --> sparse matrix --> gensim corpus\n",
    "sparse_counts = scipy.sparse.csr_matrix(tdm)\n",
    "corpus = matutils.Sparse2Corpus(sparse_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gensim also requires dictionary of the all terms and their respective location in the term-document matrix\n",
    "cv = pickle.load(open(\"cv_stop.pkl\", \"rb\"))\n",
    "id2word = dict((v, k) for k, v in cv.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the corpus (term-document matrix) and id2word (dictionary of location: term), we need to specify two other parameters - the number of topics and the number of passes. Let's start the number of topics at 2, see if the results make sense, and increase the number from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.006*\"fucking\" + 0.005*\"shit\" + 0.005*\"say\" + 0.005*\"went\" + 0.005*\"didnt\" + 0.005*\"thing\" + 0.005*\"cause\" + 0.005*\"day\" + 0.005*\"good\" + 0.005*\"hes\"'),\n",
       " (1,\n",
       "  '0.008*\"fucking\" + 0.006*\"fuck\" + 0.006*\"shit\" + 0.005*\"want\" + 0.005*\"say\" + 0.005*\"going\" + 0.005*\"theyre\" + 0.004*\"love\" + 0.004*\"oh\" + 0.004*\"dad\"')]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now that we have the corpus (term-document matrix) and id2word (dictionary of location: term),\n",
    "# we need to specify two other parameters as well - the number of topics and the number of passes\n",
    "lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=2, passes=10)\n",
    "lda.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.006*\"cause\" + 0.005*\"oh\" + 0.005*\"really\" + 0.005*\"says\" + 0.004*\"day\" + 0.004*\"say\" + 0.004*\"mean\" + 0.004*\"way\" + 0.003*\"goes\" + 0.003*\"hey\"'),\n",
       " (1,\n",
       "  '0.000*\"fucking\" + 0.000*\"shit\" + 0.000*\"going\" + 0.000*\"didnt\" + 0.000*\"fuck\" + 0.000*\"say\" + 0.000*\"hes\" + 0.000*\"day\" + 0.000*\"guy\" + 0.000*\"man\"'),\n",
       " (2,\n",
       "  '0.010*\"fucking\" + 0.007*\"fuck\" + 0.007*\"shit\" + 0.006*\"say\" + 0.006*\"going\" + 0.005*\"theyre\" + 0.005*\"hes\" + 0.005*\"want\" + 0.005*\"didnt\" + 0.005*\"thing\"')]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# LDA for num_topics = 3\n",
    "lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=3, passes=10)\n",
    "lda.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.012*\"fucking\" + 0.012*\"fuck\" + 0.009*\"theyre\" + 0.007*\"shit\" + 0.006*\"theres\" + 0.005*\"man\" + 0.005*\"house\" + 0.004*\"oh\" + 0.004*\"kids\" + 0.004*\"white\"'),\n",
       " (1,\n",
       "  '0.008*\"shit\" + 0.007*\"fucking\" + 0.006*\"didnt\" + 0.005*\"want\" + 0.005*\"going\" + 0.005*\"fuck\" + 0.005*\"did\" + 0.005*\"say\" + 0.005*\"dad\" + 0.005*\"hey\"'),\n",
       " (2,\n",
       "  '0.006*\"say\" + 0.006*\"id\" + 0.006*\"fucking\" + 0.005*\"want\" + 0.005*\"love\" + 0.004*\"didnt\" + 0.004*\"going\" + 0.004*\"says\" + 0.004*\"little\" + 0.004*\"oh\"'),\n",
       " (3,\n",
       "  '0.007*\"fucking\" + 0.007*\"day\" + 0.006*\"shit\" + 0.006*\"thing\" + 0.006*\"cause\" + 0.006*\"went\" + 0.005*\"fuck\" + 0.005*\"theyre\" + 0.005*\"good\" + 0.005*\"hes\"')]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# LDA for num_topics = 4\n",
    "lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=4, passes=10)\n",
    "lda.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These topics aren't looking too great. We've tried modifying our parameters. Let's try modifying our terms list as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling - Attempt #2 (Nouns Only)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One popular trick is to look only at terms that are from one part of speech (only nouns, only adjectives, etc.). Check out the UPenn tag set: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create a function to pull out nouns from a string of text\n",
    "from nltk import word_tokenize, pos_tag\n",
    "\n",
    "def nouns(text):\n",
    "    '''Given a string of text, tokenize the text and pull out only the nouns.'''\n",
    "    is_noun = lambda pos: pos[:2] == 'NN'\n",
    "    tokenized = word_tokenize(text)\n",
    "    all_nouns = [word for (word, pos) in pos_tag(tokenized) if is_noun(pos)] \n",
    "    return ' '.join(all_nouns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>ladies and gentlemen please welcome to the sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>thank you thank you thank you san francisco th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>all right thank you thank you very much thank...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>bo what old macdonald had a farm e i e i o and...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>this is dave he tells dirty jokes for a living...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>whats up davis whats up im home i had to bri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>ladies and gentlemen please welcome to the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>ladies and gentlemen welcome joe rogan  wha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>armed with boyish charm and a sharp wit the fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>introfade the music out lets roll hold there l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>wow hey thank you thanks thank you guys hey se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>hello hello how you doing great thank you wow ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                transcript\n",
       "ali      ladies and gentlemen please welcome to the sta...\n",
       "anthony  thank you thank you thank you san francisco th...\n",
       "bill      all right thank you thank you very much thank...\n",
       "bo       bo what old macdonald had a farm e i e i o and...\n",
       "dave     this is dave he tells dirty jokes for a living...\n",
       "hasan      whats up davis whats up im home i had to bri...\n",
       "jim         ladies and gentlemen please welcome to the ...\n",
       "joe         ladies and gentlemen welcome joe rogan  wha...\n",
       "john     armed with boyish charm and a sharp wit the fo...\n",
       "louis    introfade the music out lets roll hold there l...\n",
       "mike     wow hey thank you thanks thank you guys hey se...\n",
       "ricky    hello hello how you doing great thank you wow ..."
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in the cleaned data, before the CountVectorizer step\n",
    "data_clean = pd.read_pickle('data_clean.pkl')\n",
    "data_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>ladies gentlemen stage ali hi thank hello na s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>thank thank people i em i francisco city world...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>thank thank pleasure georgia area oasis i june...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>macdonald farm e i o farm pig e i i snort macd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>jokes living stare work profound train thought...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>whats davis whats home i netflix la york i son...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>ladies gentlemen stage mr jim jefferies thank ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>ladies gentlemen joe fuck thanks phone fuckfac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>charm wit snl writer john mulaney marriage bee...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>music lets lights lights thank i i place place...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>wow hey thanks look insane years everyone i id...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>hello thank fuck thank im gon youre weve money...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                transcript\n",
       "ali      ladies gentlemen stage ali hi thank hello na s...\n",
       "anthony  thank thank people i em i francisco city world...\n",
       "bill     thank thank pleasure georgia area oasis i june...\n",
       "bo       macdonald farm e i o farm pig e i i snort macd...\n",
       "dave     jokes living stare work profound train thought...\n",
       "hasan    whats davis whats home i netflix la york i son...\n",
       "jim      ladies gentlemen stage mr jim jefferies thank ...\n",
       "joe      ladies gentlemen joe fuck thanks phone fuckfac...\n",
       "john     charm wit snl writer john mulaney marriage bee...\n",
       "louis    music lets lights lights thank i i place place...\n",
       "mike     wow hey thanks look insane years everyone i id...\n",
       "ricky    hello thank fuck thank im gon youre weve money..."
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the nouns function to the transcripts to filter only on nouns\n",
    "data_nouns = pd.DataFrame(data_clean.transcript.apply(nouns))\n",
    "data_nouns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aaaaahhhhhhh</th>\n",
       "      <th>aaaaauuugghhhhhh</th>\n",
       "      <th>aaaahhhhh</th>\n",
       "      <th>aah</th>\n",
       "      <th>abc</th>\n",
       "      <th>abcs</th>\n",
       "      <th>ability</th>\n",
       "      <th>abortion</th>\n",
       "      <th>abortions</th>\n",
       "      <th>abuse</th>\n",
       "      <th>...</th>\n",
       "      <th>yummy</th>\n",
       "      <th>ze</th>\n",
       "      <th>zealand</th>\n",
       "      <th>zee</th>\n",
       "      <th>zeppelin</th>\n",
       "      <th>zillion</th>\n",
       "      <th>zombie</th>\n",
       "      <th>zombies</th>\n",
       "      <th>zoo</th>\n",
       "      <th>éclair</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 4619 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         aaaaahhhhhhh  aaaaauuugghhhhhh  aaaahhhhh  aah  abc  abcs  ability  \\\n",
       "ali                 0                 0          0    0    1     0        0   \n",
       "anthony             0                 0          0    0    0     0        0   \n",
       "bill                0                 0          0    0    0     1        0   \n",
       "bo                  1                 1          1    0    0     0        1   \n",
       "dave                0                 0          0    0    0     0        0   \n",
       "hasan               0                 0          0    0    0     0        0   \n",
       "jim                 0                 0          0    0    0     0        0   \n",
       "joe                 0                 0          0    0    0     0        0   \n",
       "john                0                 0          0    0    0     0        0   \n",
       "louis               0                 0          0    3    0     0        0   \n",
       "mike                0                 0          0    0    0     0        0   \n",
       "ricky               0                 0          0    0    0     0        1   \n",
       "\n",
       "         abortion  abortions  abuse  ...  yummy  ze  zealand  zee  zeppelin  \\\n",
       "ali             0          0      0  ...      0   0        0    0         0   \n",
       "anthony         2          0      0  ...      0   0       10    0         0   \n",
       "bill            0          0      0  ...      0   1        0    0         0   \n",
       "bo              0          0      0  ...      0   0        0    0         0   \n",
       "dave            0          1      0  ...      0   0        0    0         0   \n",
       "hasan           0          0      0  ...      0   0        0    1         0   \n",
       "jim             0          0      0  ...      0   0        0    0         0   \n",
       "joe             0          0      1  ...      0   0        0    0         0   \n",
       "john            0          0      0  ...      0   0        0    0         0   \n",
       "louis           0          0      0  ...      0   0        0    0         0   \n",
       "mike            0          0      0  ...      0   0        0    0         2   \n",
       "ricky           0          0      0  ...      1   0        0    0         0   \n",
       "\n",
       "         zillion  zombie  zombies  zoo  éclair  \n",
       "ali            0       1        0    0       0  \n",
       "anthony        0       0        0    0       0  \n",
       "bill           1       1        1    0       0  \n",
       "bo             0       0        0    0       0  \n",
       "dave           0       0        0    0       0  \n",
       "hasan          0       0        0    0       0  \n",
       "jim            0       0        0    0       0  \n",
       "joe            0       0        0    0       0  \n",
       "john           0       0        0    0       1  \n",
       "louis          0       0        0    0       0  \n",
       "mike           0       0        0    0       0  \n",
       "ricky          0       0        0    1       0  \n",
       "\n",
       "[12 rows x 4619 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new document-term matrix using only nouns\n",
    "from sklearn.feature_extraction import text\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Re-add the additional stop words since we are recreating the document-term matrix\n",
    "add_stop_words = ['like', 'im', 'know', 'just', 'dont', 'thats', 'right', 'people',\n",
    "                  'youre', 'got', 'gonna', 'time', 'think', 'yeah', 'said']\n",
    "stop_words = text.ENGLISH_STOP_WORDS.union(add_stop_words)\n",
    "\n",
    "# Recreate a document-term matrix with only nouns\n",
    "cvn = CountVectorizer(stop_words=stop_words)\n",
    "data_cvn = cvn.fit_transform(data_nouns.transcript)\n",
    "data_dtmn = pd.DataFrame(data_cvn.toarray(), columns=cvn.get_feature_names())\n",
    "data_dtmn.index = data_nouns.index\n",
    "data_dtmn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the gensim corpus\n",
    "corpusn = matutils.Sparse2Corpus(scipy.sparse.csr_matrix(data_dtmn.transpose()))\n",
    "\n",
    "# Create the vocabulary dictionary\n",
    "id2wordn = dict((v, k) for k, v in cvn.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.008*\"man\" + 0.007*\"day\" + 0.007*\"thing\" + 0.006*\"cause\" + 0.006*\"guy\" + 0.006*\"way\" + 0.006*\"fuck\" + 0.006*\"shit\" + 0.005*\"women\" + 0.005*\"hes\"'),\n",
       " (1,\n",
       "  '0.009*\"thing\" + 0.009*\"life\" + 0.009*\"day\" + 0.009*\"shit\" + 0.008*\"hes\" + 0.007*\"man\" + 0.007*\"way\" + 0.006*\"kids\" + 0.006*\"dad\" + 0.006*\"fuck\"')]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's start with 2 topics\n",
    "ldan = models.LdaModel(corpus=corpusn, num_topics=2, id2word=id2wordn, passes=10)\n",
    "ldan.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.011*\"day\" + 0.009*\"thing\" + 0.008*\"man\" + 0.007*\"cause\" + 0.007*\"fuck\" + 0.007*\"shit\" + 0.006*\"hes\" + 0.006*\"way\" + 0.006*\"house\" + 0.006*\"lot\"'),\n",
       " (1,\n",
       "  '0.010*\"shit\" + 0.009*\"thing\" + 0.008*\"life\" + 0.008*\"hes\" + 0.008*\"guy\" + 0.008*\"man\" + 0.007*\"way\" + 0.007*\"day\" + 0.007*\"cause\" + 0.007*\"gon\"'),\n",
       " (2,\n",
       "  '0.013*\"dad\" + 0.007*\"life\" + 0.007*\"shes\" + 0.006*\"mom\" + 0.006*\"parents\" + 0.006*\"school\" + 0.005*\"girl\" + 0.005*\"home\" + 0.005*\"hes\" + 0.004*\"house\"')]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's try topics = 3\n",
    "ldan = models.LdaModel(corpus=corpusn, num_topics=3, id2word=id2wordn, passes=10)\n",
    "ldan.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.008*\"stuff\" + 0.007*\"point\" + 0.007*\"way\" + 0.007*\"bo\" + 0.007*\"cause\" + 0.006*\"repeat\" + 0.005*\"kind\" + 0.005*\"night\" + 0.005*\"id\" + 0.005*\"eye\"'),\n",
       " (1,\n",
       "  '0.001*\"way\" + 0.001*\"thing\" + 0.000*\"things\" + 0.000*\"life\" + 0.000*\"hes\" + 0.000*\"day\" + 0.000*\"night\" + 0.000*\"man\" + 0.000*\"cause\" + 0.000*\"shit\"'),\n",
       " (2,\n",
       "  '0.013*\"thing\" + 0.011*\"day\" + 0.009*\"hes\" + 0.009*\"life\" + 0.008*\"cause\" + 0.008*\"guy\" + 0.007*\"shit\" + 0.007*\"man\" + 0.007*\"way\" + 0.006*\"gon\"'),\n",
       " (3,\n",
       "  '0.012*\"shit\" + 0.010*\"fuck\" + 0.009*\"man\" + 0.009*\"lot\" + 0.008*\"dad\" + 0.008*\"day\" + 0.007*\"life\" + 0.006*\"way\" + 0.006*\"hes\" + 0.006*\"thing\"')]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's try 4 topics\n",
    "ldan = models.LdaModel(corpus=corpusn, num_topics=4, id2word=id2wordn, passes=10)\n",
    "ldan.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling - Attempt #3 (Nouns and Adjectives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create a function to pull out nouns from a string of text\n",
    "def nouns_adj(text):\n",
    "    '''Given a string of text, tokenize the text and pull out only the nouns and adjectives.'''\n",
    "    is_noun_adj = lambda pos: pos[:2] == 'NN' or pos[:2] == 'JJ'\n",
    "    tokenized = word_tokenize(text)\n",
    "    nouns_adj = [word for (word, pos) in pos_tag(tokenized) if is_noun_adj(pos)] \n",
    "    return ' '.join(nouns_adj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>transcript</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>ladies gentlemen welcome stage ali wong hi wel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>thank san francisco thank good people surprise...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>right thank thank pleasure greater atlanta geo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>old macdonald farm e i i o farm pig e i i snor...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>dirty jokes living stare most hard work profou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>whats davis whats im home i netflix special la...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>ladies gentlemen welcome stage mr jim jefferie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>ladies gentlemen joe fuck san francisco thanks...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>boyish charm sharp wit former snl writer john ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>music lets lights lights thank much i i i nice...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>wow hey thanks hey seattle nice look crazy ins...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>hello great thank fuck thank lovely welcome im...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                transcript\n",
       "ali      ladies gentlemen welcome stage ali wong hi wel...\n",
       "anthony  thank san francisco thank good people surprise...\n",
       "bill     right thank thank pleasure greater atlanta geo...\n",
       "bo       old macdonald farm e i i o farm pig e i i snor...\n",
       "dave     dirty jokes living stare most hard work profou...\n",
       "hasan    whats davis whats im home i netflix special la...\n",
       "jim      ladies gentlemen welcome stage mr jim jefferie...\n",
       "joe      ladies gentlemen joe fuck san francisco thanks...\n",
       "john     boyish charm sharp wit former snl writer john ...\n",
       "louis    music lets lights lights thank much i i i nice...\n",
       "mike     wow hey thanks hey seattle nice look crazy ins...\n",
       "ricky    hello great thank fuck thank lovely welcome im..."
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the nouns function to the transcripts to filter only on nouns\n",
    "data_nouns_adj = pd.DataFrame(data_clean.transcript.apply(nouns_adj))\n",
    "data_nouns_adj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aaaaah</th>\n",
       "      <th>aaaaahhhhhhh</th>\n",
       "      <th>aaaaauuugghhhhhh</th>\n",
       "      <th>aaaahhhhh</th>\n",
       "      <th>aah</th>\n",
       "      <th>abc</th>\n",
       "      <th>abcs</th>\n",
       "      <th>ability</th>\n",
       "      <th>abject</th>\n",
       "      <th>able</th>\n",
       "      <th>...</th>\n",
       "      <th>ze</th>\n",
       "      <th>zealand</th>\n",
       "      <th>zee</th>\n",
       "      <th>zeppelin</th>\n",
       "      <th>zero</th>\n",
       "      <th>zillion</th>\n",
       "      <th>zombie</th>\n",
       "      <th>zombies</th>\n",
       "      <th>zoo</th>\n",
       "      <th>éclair</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ali</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>anthony</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bill</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bo</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dave</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hasan</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>jim</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>joe</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>john</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>louis</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mike</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ricky</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12 rows × 5567 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         aaaaah  aaaaahhhhhhh  aaaaauuugghhhhhh  aaaahhhhh  aah  abc  abcs  \\\n",
       "ali           0             0                 0          0    0    1     0   \n",
       "anthony       0             0                 0          0    0    0     0   \n",
       "bill          1             0                 0          0    0    0     1   \n",
       "bo            0             1                 1          1    0    0     0   \n",
       "dave          0             0                 0          0    0    0     0   \n",
       "hasan         0             0                 0          0    0    0     0   \n",
       "jim           0             0                 0          0    0    0     0   \n",
       "joe           0             0                 0          0    0    0     0   \n",
       "john          0             0                 0          0    0    0     0   \n",
       "louis         0             0                 0          0    3    0     0   \n",
       "mike          0             0                 0          0    0    0     0   \n",
       "ricky         0             0                 0          0    0    0     0   \n",
       "\n",
       "         ability  abject  able  ...  ze  zealand  zee  zeppelin  zero  \\\n",
       "ali            0       0     2  ...   0        0    0         0     0   \n",
       "anthony        0       0     0  ...   0       10    0         0     0   \n",
       "bill           0       0     1  ...   1        0    0         0     0   \n",
       "bo             1       0     0  ...   0        0    0         0     1   \n",
       "dave           0       0     0  ...   0        0    0         0     0   \n",
       "hasan          0       0     1  ...   0        0    2         0     0   \n",
       "jim            0       0     1  ...   0        0    0         0     0   \n",
       "joe            0       0     2  ...   0        0    0         0     0   \n",
       "john           0       0     3  ...   0        0    0         0     0   \n",
       "louis          0       0     1  ...   0        0    0         0     0   \n",
       "mike           0       0     0  ...   0        0    0         2     0   \n",
       "ricky          1       1     2  ...   0        0    0         0     0   \n",
       "\n",
       "         zillion  zombie  zombies  zoo  éclair  \n",
       "ali            0       1        0    0       0  \n",
       "anthony        0       0        0    0       0  \n",
       "bill           1       1        1    0       0  \n",
       "bo             0       0        0    0       0  \n",
       "dave           0       0        0    0       0  \n",
       "hasan          0       0        0    0       0  \n",
       "jim            0       0        0    0       0  \n",
       "joe            0       0        0    0       0  \n",
       "john           0       0        0    0       1  \n",
       "louis          0       0        0    0       0  \n",
       "mike           0       0        0    0       0  \n",
       "ricky          0       0        0    1       0  \n",
       "\n",
       "[12 rows x 5567 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new document-term matrix using only nouns and adjectives, also remove common words with max_df\n",
    "cvna = CountVectorizer(stop_words=stop_words, max_df=.8)\n",
    "data_cvna = cvna.fit_transform(data_nouns_adj.transcript)\n",
    "data_dtmna = pd.DataFrame(data_cvna.toarray(), columns=cvna.get_feature_names())\n",
    "data_dtmna.index = data_nouns_adj.index\n",
    "data_dtmna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the gensim corpus\n",
    "corpusna = matutils.Sparse2Corpus(scipy.sparse.csr_matrix(data_dtmna.transpose()))\n",
    "\n",
    "# Create the vocabulary dictionary\n",
    "id2wordna = dict((v, k) for k, v in cvna.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.004*\"gun\" + 0.004*\"bo\" + 0.003*\"hasan\" + 0.003*\"guns\" + 0.003*\"repeat\" + 0.003*\"comedy\" + 0.003*\"um\" + 0.003*\"class\" + 0.003*\"mom\" + 0.003*\"son\"'),\n",
       " (1,\n",
       "  '0.004*\"joke\" + 0.004*\"mom\" + 0.003*\"friend\" + 0.003*\"parents\" + 0.003*\"ahah\" + 0.002*\"jokes\" + 0.002*\"dead\" + 0.002*\"ok\" + 0.002*\"clinton\" + 0.002*\"door\"')]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's start with 2 topics\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=2, id2word=id2wordna, passes=10)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.007*\"joke\" + 0.004*\"anthony\" + 0.004*\"gun\" + 0.004*\"tit\" + 0.003*\"jesus\" + 0.003*\"mad\" + 0.003*\"grandma\" + 0.003*\"dead\" + 0.003*\"hell\" + 0.003*\"older\"'),\n",
       " (1,\n",
       "  '0.005*\"ahah\" + 0.004*\"hasan\" + 0.004*\"door\" + 0.004*\"son\" + 0.004*\"mom\" + 0.003*\"guns\" + 0.003*\"ass\" + 0.003*\"class\" + 0.003*\"parents\" + 0.003*\"american\"'),\n",
       " (2,\n",
       "  '0.004*\"joke\" + 0.003*\"mom\" + 0.003*\"bo\" + 0.003*\"clinton\" + 0.003*\"jenny\" + 0.003*\"parents\" + 0.003*\"repeat\" + 0.003*\"friend\" + 0.003*\"eye\" + 0.002*\"ok\"')]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's try 3 topics\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=3, id2word=id2wordna, passes=10)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.004*\"joke\" + 0.003*\"mom\" + 0.003*\"bo\" + 0.003*\"hasan\" + 0.003*\"parents\" + 0.003*\"door\" + 0.003*\"comedy\" + 0.003*\"repeat\" + 0.003*\"love\" + 0.003*\"eye\"'),\n",
       " (1,\n",
       "  '0.008*\"ahah\" + 0.006*\"guns\" + 0.006*\"son\" + 0.004*\"ass\" + 0.004*\"gay\" + 0.004*\"girlfriend\" + 0.004*\"american\" + 0.003*\"class\" + 0.003*\"gun\" + 0.003*\"business\"'),\n",
       " (2,\n",
       "  '0.005*\"mom\" + 0.004*\"parents\" + 0.004*\"joke\" + 0.004*\"ok\" + 0.004*\"clinton\" + 0.004*\"friend\" + 0.004*\"jenny\" + 0.003*\"anthony\" + 0.003*\"husband\" + 0.003*\"tit\"'),\n",
       " (3,\n",
       "  '0.000*\"joke\" + 0.000*\"friend\" + 0.000*\"mom\" + 0.000*\"son\" + 0.000*\"ahah\" + 0.000*\"parents\" + 0.000*\"jokes\" + 0.000*\"young\" + 0.000*\"ass\" + 0.000*\"boy\"')]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's try 4 topics\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=4, id2word=id2wordna, passes=10)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify Topics in Each Document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of the 9 topic models we looked at, the nouns and adjectives, 4 topic one made the most sense. So let's pull that down here and run it through some more iterations to get more fine-tuned topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.005*\"gun\" + 0.005*\"ass\" + 0.005*\"guns\" + 0.004*\"son\" + 0.003*\"husband\" + 0.003*\"class\" + 0.003*\"business\" + 0.003*\"girlfriend\" + 0.003*\"ok\" + 0.003*\"pregnant\"'),\n",
       " (1,\n",
       "  '0.007*\"jenny\" + 0.005*\"tit\" + 0.004*\"date\" + 0.004*\"accident\" + 0.004*\"parents\" + 0.003*\"worst\" + 0.003*\"morning\" + 0.003*\"ok\" + 0.003*\"marriage\" + 0.003*\"idea\"'),\n",
       " (2,\n",
       "  '0.008*\"joke\" + 0.005*\"mom\" + 0.004*\"jokes\" + 0.004*\"parents\" + 0.004*\"hasan\" + 0.003*\"door\" + 0.003*\"anthony\" + 0.003*\"jenner\" + 0.003*\"mad\" + 0.003*\"twitter\"'),\n",
       " (3,\n",
       "  '0.007*\"ahah\" + 0.006*\"bo\" + 0.005*\"clinton\" + 0.005*\"repeat\" + 0.005*\"wife\" + 0.004*\"gay\" + 0.004*\"mom\" + 0.004*\"eye\" + 0.004*\"friend\" + 0.004*\"contact\"')]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Our final LDA model (for now)\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=4, id2word=id2wordna, passes=80)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These four topics look pretty decent. Let's settle on these for now.\n",
    "* Topic 0: mom, parents\n",
    "* Topic 1: husband, wife\n",
    "* Topic 2: guns\n",
    "* Topic 3: profanity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 'ali'),\n",
       " (2, 'anthony'),\n",
       " (0, 'bill'),\n",
       " (3, 'bo'),\n",
       " (3, 'dave'),\n",
       " (2, 'hasan'),\n",
       " (0, 'jim'),\n",
       " (2, 'joe'),\n",
       " (3, 'john'),\n",
       " (1, 'louis'),\n",
       " (1, 'mike'),\n",
       " (2, 'ricky')]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at which topics each transcript contains\n",
    "corpus_transformed = ldana[corpusna]\n",
    "list(zip([a for [(a,b)] in corpus_transformed], data_dtmna.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For a first pass of LDA, these kind of make sense to me, so we'll call it a day for now.\n",
    "* Topic 0: mom, parents [Anthony, Hasan, Louis, Ricky]\n",
    "* Topic 1: husband, wife [Ali, John, Mike]\n",
    "* Topic 2: guns [Bill, Bo, Jim]\n",
    "* Topic 3: profanity [Dave, Joe]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Additional Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Try further modifying the parameters of the topic models above and see if you can get better topics.\n",
    "2. Create a new topic model that includes terms from a different [part of speech](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) and see if you can get better topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}