SMUMSE_ProgWorkshop_Lesson3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SMU Master of Science (Economics) Programming Workshop in Python\n",
    "\n",
    "\n",
    "## Introduction\n",
    "In today's class, we will be learning more about Data Cleaning and Manipulation with pandas and NumPy. Both pandas and NumPy are great libraries that contain many useful functions for data cleaning and manipulation. \n",
    "\n",
    "---\n",
    "\n",
    "pandas\n",
    "![](images/pandas_logo.png \"pandas\")\n",
    "pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.\n",
    "\n",
    "---\n",
    "\n",
    "NumPy\n",
    "![](images/numpy_logo.png \"NumPy\")\n",
    "NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays.\n",
    "\n",
    "You can find more information about pandas [here](https://pandas.pydata.org/), and more information about NumPy [here](https://www.numpy.org/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we go dive deep into data cleaning and manipulation, it is useful to take a step back and think about the differences between a function and a method:\n",
    "\n",
    "1. A function is a \"global\" function that can be used on different kinds of data structures. For example, the `len` function can be used on lists and strings.\n",
    "2. A method is a \"local\" function that can only be used on a specific kind of data structure. For example, the method `pivot_table` can only be used on the DataFrame data structure (we will see this in more detail)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Case Study Approach\n",
    "\n",
    "To learn how we can use pandas and NumPy, we will adopt a case study approach. We have attached 2 datasets (which you can find in the /data folder):\n",
    "\n",
    "1. The first dataset is from [Kaggle](https://www.kaggle.com/), a machine learning platform. More information about the data can be found [here](https://www.kaggle.com/c/titanic/data). This dataset contains information about the passengers that were involved in Titanic's sinking in 1912.\n",
    "\n",
    "2. The second dataset is a dataset on movie ratings that comes from [GroupLens](https://grouplens.org/). In this dataset, there are 1,000,000 entries on movie ratings by users. The metadata can be found [here](http://files.grouplens.org/datasets/movielens/ml-1m-README.txt). \n",
    "\n",
    "---\n",
    "\n",
    "We will begin our data analysis on the Titanic dataset first. We begin by importing key libraries, such as pandas and NumPy library into Jupyter, by calling the \"import\" function. In this case, we will only be dealing with the training data.\n",
    "\n",
    "---\n",
    "\n",
    "#### Data Manipulation with Titanic\n",
    "In the first part of data manipulation, we learn how to get summary statistics of the dataset, and the types of data we are dealing with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if data is the filename,\n",
    "\n",
    "1. data.shape returns the dimension of the dataset we are dealing with\n",
    "2. data.head() returns the first five rows of the dataset\n",
    "3 3. data.describe() returns the summary statistics of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>204</td>\n",
       "      <td>889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>891</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>681</td>\n",
       "      <td>NaN</td>\n",
       "      <td>147</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lundahl, Mr. Johan Svensson</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>NaN</td>\n",
       "      <td>B96 B98</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>577</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.204208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.693429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.910400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        PassengerId    Survived      Pclass                         Name  \\\n",
       "count    891.000000  891.000000  891.000000                          891   \n",
       "unique          NaN         NaN         NaN                          891   \n",
       "top             NaN         NaN         NaN  Lundahl, Mr. Johan Svensson   \n",
       "freq            NaN         NaN         NaN                            1   \n",
       "mean     446.000000    0.383838    2.308642                          NaN   \n",
       "std      257.353842    0.486592    0.836071                          NaN   \n",
       "min        1.000000    0.000000    1.000000                          NaN   \n",
       "25%      223.500000    0.000000    2.000000                          NaN   \n",
       "50%      446.000000    0.000000    3.000000                          NaN   \n",
       "75%      668.500000    1.000000    3.000000                          NaN   \n",
       "max      891.000000    1.000000    3.000000                          NaN   \n",
       "\n",
       "         Sex         Age       SibSp       Parch    Ticket        Fare  \\\n",
       "count    891  714.000000  891.000000  891.000000       891  891.000000   \n",
       "unique     2         NaN         NaN         NaN       681         NaN   \n",
       "top     male         NaN         NaN         NaN  CA. 2343         NaN   \n",
       "freq     577         NaN         NaN         NaN         7         NaN   \n",
       "mean     NaN   29.699118    0.523008    0.381594       NaN   32.204208   \n",
       "std      NaN   14.526497    1.102743    0.806057       NaN   49.693429   \n",
       "min      NaN    0.420000    0.000000    0.000000       NaN    0.000000   \n",
       "25%      NaN   20.125000    0.000000    0.000000       NaN    7.910400   \n",
       "50%      NaN   28.000000    0.000000    0.000000       NaN   14.454200   \n",
       "75%      NaN   38.000000    1.000000    0.000000       NaN   31.000000   \n",
       "max      NaN   80.000000    8.000000    6.000000       NaN  512.329200   \n",
       "\n",
       "          Cabin Embarked  \n",
       "count       204      889  \n",
       "unique      147        3  \n",
       "top     B96 B98        S  \n",
       "freq          4      644  \n",
       "mean        NaN      NaN  \n",
       "std         NaN      NaN  \n",
       "min         NaN      NaN  \n",
       "25%         NaN      NaN  \n",
       "50%         NaN      NaN  \n",
       "75%         NaN      NaN  \n",
       "max         NaN      NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We have access to all functions from pandas and numpy\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Since data is in the form of a csv file, we can use pd.read_csv\n",
    "\n",
    "train_data = pd.read_csv('data/titanic/train.csv') \n",
    "# print(len(train_data))\n",
    "# train_data.head()\n",
    "train_data.describe(include='all') # Describe gives a summary of the variables in the dataframe "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      int64\n",
       "Survived         int64\n",
       "Pclass           int64\n",
       "Name            object\n",
       "Sex             object\n",
       "Age            float64\n",
       "SibSp            int64\n",
       "Parch            int64\n",
       "Ticket          object\n",
       "Fare           float64\n",
       "Cabin           object\n",
       "Embarked        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.dtypes # an attribute of the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The describe( ) method gives us a quick summary of what we are dealing with. For example, we know that there are a total of 11 variables (excluding passenger ID), and there are a mixture of variables which are categorical ('Ticket', 'Cabin', 'Embarked') as well as numerical ('Pclass', 'Age', 'Fare').\n",
    "\n",
    "Let's take a look at the first 5 rows of the dataset. We can subset the data using square brackets, [ ], or use the DataFrame method, `head()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Moran, Mr. James</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>330877</td>\n",
       "      <td>8.4583</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>McCarthy, Mr. Timothy J</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17463</td>\n",
       "      <td>51.8625</td>\n",
       "      <td>E46</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Palsson, Master. Gosta Leonard</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>349909</td>\n",
       "      <td>21.0750</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>347742</td>\n",
       "      <td>11.1333</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Nasser, Mrs. Nicholas (Adele Achem)</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>237736</td>\n",
       "      <td>30.0708</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "5            6         0       3   \n",
       "6            7         0       1   \n",
       "7            8         0       3   \n",
       "8            9         1       3   \n",
       "9           10         1       2   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "5                                   Moran, Mr. James    male   NaN      0   \n",
       "6                            McCarthy, Mr. Timothy J    male  54.0      0   \n",
       "7                     Palsson, Master. Gosta Leonard    male   2.0      3   \n",
       "8  Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)  female  27.0      0   \n",
       "9                Nasser, Mrs. Nicholas (Adele Achem)  female  14.0      1   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  \n",
       "5      0            330877   8.4583   NaN        Q  \n",
       "6      0             17463  51.8625   E46        S  \n",
       "7      1            349909  21.0750   NaN        S  \n",
       "8      2            347742  11.1333   NaN        S  \n",
       "9      0            237736  30.0708   NaN        C  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[:10] # Returns the first 10 observations in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsetting Data\n",
    "\n",
    "Now that we have a better idea of what the data looks like, we can conduct some data analysis. Suppose we are interested in finding out who embarked at location, 'S'. One way to do so is by writing the following code, and it returns all passengers who embarked at the location, 'S'.\n",
    "\n",
    "In the following code block, we have provided a condition `[train_data['Embarked'] == 'S'` - that the column, \"Embarked\" should be equivalent to \"S\". The `[10]` refers to the first 10 rows that satisfies this condition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>527.777778</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36.248000</td>\n",
       "      <td>0.377778</td>\n",
       "      <td>0.311111</td>\n",
       "      <td>74.637320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>215.633410</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.936744</td>\n",
       "      <td>0.534657</td>\n",
       "      <td>0.668180</td>\n",
       "      <td>101.091948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>24.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>26.287500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>431.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>29.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>588.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>35.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>682.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>48.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>79.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>890.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId  Survived  Pclass        Age      SibSp      Parch  \\\n",
       "count    45.000000      45.0    45.0  40.000000  45.000000  45.000000   \n",
       "mean    527.777778       1.0     1.0  36.248000   0.377778   0.311111   \n",
       "std     215.633410       0.0     0.0  14.936744   0.534657   0.668180   \n",
       "min      24.000000       1.0     1.0   0.920000   0.000000   0.000000   \n",
       "25%     431.000000       1.0     1.0  27.000000   0.000000   0.000000   \n",
       "50%     588.000000       1.0     1.0  36.000000   0.000000   0.000000   \n",
       "75%     682.000000       1.0     1.0  48.000000   1.000000   0.000000   \n",
       "max     890.000000       1.0     1.0  80.000000   2.000000   2.000000   \n",
       "\n",
       "             Fare  \n",
       "count   45.000000  \n",
       "mean    74.637320  \n",
       "std    101.091948  \n",
       "min     26.287500  \n",
       "25%     29.700000  \n",
       "50%     35.500000  \n",
       "75%     79.200000  \n",
       "max    512.329200  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can pass in conditions to subset the data, 0 4 886 889 890\n",
    "variables_of_interest = [\"Survived\", \"Fare\", \"Pclass\"]\n",
    "train_data[variables_of_interest]\n",
    "\n",
    "# Use the ampersand logo to string conditions\n",
    "subset1 = train_data[(train_data['Survived'] == 1) & \n",
    "                     (train_data[\"Pclass\"] == 1) & \n",
    "                     (train_data[\"Sex\"] == \"male\")]\n",
    "subset1.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other than subsetting data on only 1 conditions, we can also subset data on 2 or more conditions, using the ampersand (`&`) logo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>438</th>\n",
       "      <td>439</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Fortune, Mr. Mark</td>\n",
       "      <td>male</td>\n",
       "      <td>64.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>19950</td>\n",
       "      <td>263.0</td>\n",
       "      <td>C23 C25 C27</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass               Name   Sex   Age  SibSp  \\\n",
       "438          439         0       1  Fortune, Mr. Mark  male  64.0      1   \n",
       "\n",
       "     Parch Ticket   Fare        Cabin Embarked  \n",
       "438      4  19950  263.0  C23 C25 C27        S  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[(train_data['Parch'] > 2) & (train_data['Pclass'] == 1) & (train_data['Age'] > 30)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-class assignment\n",
    "\n",
    "1. First class passengers should have a survival rate. Verify this.\n",
    "2. People below the age of 20 are more likely to survive. Verify this.\n",
    "3. First class passengers should have the highest average fare, followed by second class, then third class. Verify this.\n",
    "\n",
    "Helper functions/method:\n",
    "\n",
    "1. data.describe() : returns the mean, min, max etc. of the variables in the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.0</td>\n",
       "      <td>186.000000</td>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>461.597222</td>\n",
       "      <td>0.629630</td>\n",
       "      <td>1.0</td>\n",
       "      <td>38.233441</td>\n",
       "      <td>0.416667</td>\n",
       "      <td>0.356481</td>\n",
       "      <td>84.154687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>246.737616</td>\n",
       "      <td>0.484026</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.802856</td>\n",
       "      <td>0.611898</td>\n",
       "      <td>0.693997</td>\n",
       "      <td>78.380373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>270.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>30.923950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>472.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>60.287500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>670.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>49.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>93.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>890.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived  Pclass         Age       SibSp       Parch  \\\n",
       "count   216.000000  216.000000   216.0  186.000000  216.000000  216.000000   \n",
       "mean    461.597222    0.629630     1.0   38.233441    0.416667    0.356481   \n",
       "std     246.737616    0.484026     0.0   14.802856    0.611898    0.693997   \n",
       "min       2.000000    0.000000     1.0    0.920000    0.000000    0.000000   \n",
       "25%     270.750000    0.000000     1.0   27.000000    0.000000    0.000000   \n",
       "50%     472.000000    1.000000     1.0   37.000000    0.000000    0.000000   \n",
       "75%     670.500000    1.000000     1.0   49.000000    1.000000    0.000000   \n",
       "max     890.000000    1.000000     1.0   80.000000    3.000000    4.000000   \n",
       "\n",
       "             Fare  \n",
       "count  216.000000  \n",
       "mean    84.154687  \n",
       "std     78.380373  \n",
       "min      0.000000  \n",
       "25%     30.923950  \n",
       "50%     60.287500  \n",
       "75%     93.500000  \n",
       "max    512.329200  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Question 1\n",
    "subset1 = train_data[train_data['Pclass'] == 1]\n",
    "subset2 = train_data[train_data['Pclass'] != 1]\n",
    "\n",
    "subset1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>675.000000</td>\n",
       "      <td>675.000000</td>\n",
       "      <td>675.000000</td>\n",
       "      <td>528.000000</td>\n",
       "      <td>675.000000</td>\n",
       "      <td>675.000000</td>\n",
       "      <td>675.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>441.008889</td>\n",
       "      <td>0.305185</td>\n",
       "      <td>2.727407</td>\n",
       "      <td>26.692708</td>\n",
       "      <td>0.557037</td>\n",
       "      <td>0.389630</td>\n",
       "      <td>15.580055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>260.640463</td>\n",
       "      <td>0.460827</td>\n",
       "      <td>0.445623</td>\n",
       "      <td>13.183785</td>\n",
       "      <td>1.217183</td>\n",
       "      <td>0.839075</td>\n",
       "      <td>12.626406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>208.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.854200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>433.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>667.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>34.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>19.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>73.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   675.000000  675.000000  675.000000  528.000000  675.000000   \n",
       "mean    441.008889    0.305185    2.727407   26.692708    0.557037   \n",
       "std     260.640463    0.460827    0.445623   13.183785    1.217183   \n",
       "min       1.000000    0.000000    2.000000    0.420000    0.000000   \n",
       "25%     208.500000    0.000000    2.000000   19.000000    0.000000   \n",
       "50%     433.000000    0.000000    3.000000   26.000000    0.000000   \n",
       "75%     667.500000    1.000000    3.000000   34.000000    1.000000   \n",
       "max     891.000000    1.000000    3.000000   74.000000    8.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  675.000000  675.000000  \n",
       "mean     0.389630   15.580055  \n",
       "std      0.839075   12.626406  \n",
       "min      0.000000    0.000000  \n",
       "25%      0.000000    7.854200  \n",
       "50%      0.000000   10.500000  \n",
       "75%      0.000000   19.500000  \n",
       "max      6.000000   73.500000  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset2.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>550.000000</td>\n",
       "      <td>550.000000</td>\n",
       "      <td>550.000000</td>\n",
       "      <td>550.000000</td>\n",
       "      <td>550.000000</td>\n",
       "      <td>550.000000</td>\n",
       "      <td>550.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>452.052727</td>\n",
       "      <td>0.383636</td>\n",
       "      <td>2.149091</td>\n",
       "      <td>34.982727</td>\n",
       "      <td>0.332727</td>\n",
       "      <td>0.314545</td>\n",
       "      <td>35.559173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>253.910629</td>\n",
       "      <td>0.486714</td>\n",
       "      <td>0.853339</td>\n",
       "      <td>11.795702</td>\n",
       "      <td>0.569586</td>\n",
       "      <td>0.826016</td>\n",
       "      <td>55.624437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>237.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.050000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>453.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.454200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>666.750000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>34.375000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   550.000000  550.000000  550.000000  550.000000  550.000000   \n",
       "mean    452.052727    0.383636    2.149091   34.982727    0.332727   \n",
       "std     253.910629    0.486714    0.853339   11.795702    0.569586   \n",
       "min       1.000000    0.000000    1.000000   20.000000    0.000000   \n",
       "25%     237.750000    0.000000    1.000000   26.000000    0.000000   \n",
       "50%     453.500000    0.000000    2.000000   32.000000    0.000000   \n",
       "75%     666.750000    1.000000    3.000000   42.000000    1.000000   \n",
       "max     891.000000    1.000000    3.000000   80.000000    3.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  550.000000  550.000000  \n",
       "mean     0.314545   35.559173  \n",
       "std      0.826016   55.624437  \n",
       "min      0.000000    0.000000  \n",
       "25%      0.000000    8.050000  \n",
       "50%      0.000000   14.454200  \n",
       "75%      0.000000   34.375000  \n",
       "max      6.000000  512.329200  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Question 1\n",
    "subset1 = train_data[train_data['Age'] >= 20]\n",
    "subset2 = train_data[train_data['Age'] < 20]\n",
    "\n",
    "subset1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>164.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>164.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>436.945122</td>\n",
       "      <td>0.481707</td>\n",
       "      <td>2.530488</td>\n",
       "      <td>11.979695</td>\n",
       "      <td>1.115854</td>\n",
       "      <td>0.823171</td>\n",
       "      <td>31.794741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>276.366771</td>\n",
       "      <td>0.501196</td>\n",
       "      <td>0.712929</td>\n",
       "      <td>6.656828</td>\n",
       "      <td>1.487780</td>\n",
       "      <td>0.828451</td>\n",
       "      <td>42.632362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>183.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>9.316675</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>422.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>19.258300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>694.250000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>31.387500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>888.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>263.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   164.000000  164.000000  164.000000  164.000000  164.000000   \n",
       "mean    436.945122    0.481707    2.530488   11.979695    1.115854   \n",
       "std     276.366771    0.501196    0.712929    6.656828    1.487780   \n",
       "min       8.000000    0.000000    1.000000    0.420000    0.000000   \n",
       "25%     183.750000    0.000000    2.000000    5.000000    0.000000   \n",
       "50%     422.500000    0.000000    3.000000   15.000000    1.000000   \n",
       "75%     694.250000    1.000000    3.000000   18.000000    1.000000   \n",
       "max     888.000000    1.000000    3.000000   19.000000    5.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  164.000000  164.000000  \n",
       "mean     0.823171   31.794741  \n",
       "std      0.828451   42.632362  \n",
       "min      0.000000    0.000000  \n",
       "25%      0.000000    9.316675  \n",
       "50%      1.000000   19.258300  \n",
       "75%      2.000000   31.387500  \n",
       "max      3.000000  263.000000  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset2.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.0</td>\n",
       "      <td>186.000000</td>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.000000</td>\n",
       "      <td>216.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>461.597222</td>\n",
       "      <td>0.629630</td>\n",
       "      <td>1.0</td>\n",
       "      <td>38.233441</td>\n",
       "      <td>0.416667</td>\n",
       "      <td>0.356481</td>\n",
       "      <td>84.154687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>246.737616</td>\n",
       "      <td>0.484026</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.802856</td>\n",
       "      <td>0.611898</td>\n",
       "      <td>0.693997</td>\n",
       "      <td>78.380373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.920000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>270.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>30.923950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>472.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>60.287500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>670.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>49.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>93.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>890.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived  Pclass         Age       SibSp       Parch  \\\n",
       "count   216.000000  216.000000   216.0  186.000000  216.000000  216.000000   \n",
       "mean    461.597222    0.629630     1.0   38.233441    0.416667    0.356481   \n",
       "std     246.737616    0.484026     0.0   14.802856    0.611898    0.693997   \n",
       "min       2.000000    0.000000     1.0    0.920000    0.000000    0.000000   \n",
       "25%     270.750000    0.000000     1.0   27.000000    0.000000    0.000000   \n",
       "50%     472.000000    1.000000     1.0   37.000000    0.000000    0.000000   \n",
       "75%     670.500000    1.000000     1.0   49.000000    1.000000    0.000000   \n",
       "max     890.000000    1.000000     1.0   80.000000    3.000000    4.000000   \n",
       "\n",
       "             Fare  \n",
       "count  216.000000  \n",
       "mean    84.154687  \n",
       "std     78.380373  \n",
       "min      0.000000  \n",
       "25%     30.923950  \n",
       "50%     60.287500  \n",
       "75%     93.500000  \n",
       "max    512.329200  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset1 = train_data[train_data['Pclass'] == 1]\n",
    "subset2 = train_data[train_data['Pclass'] == 2]\n",
    "subset3 = train_data[train_data['Pclass'] == 3]\n",
    "\n",
    "subset1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>184.000000</td>\n",
       "      <td>184.000000</td>\n",
       "      <td>184.0</td>\n",
       "      <td>173.000000</td>\n",
       "      <td>184.000000</td>\n",
       "      <td>184.000000</td>\n",
       "      <td>184.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>445.956522</td>\n",
       "      <td>0.472826</td>\n",
       "      <td>2.0</td>\n",
       "      <td>29.877630</td>\n",
       "      <td>0.402174</td>\n",
       "      <td>0.380435</td>\n",
       "      <td>20.662183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>250.852161</td>\n",
       "      <td>0.500623</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.001077</td>\n",
       "      <td>0.601633</td>\n",
       "      <td>0.690963</td>\n",
       "      <td>13.417399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.670000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>234.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>13.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>435.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>36.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>26.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>887.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>73.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived  Pclass         Age       SibSp       Parch  \\\n",
       "count   184.000000  184.000000   184.0  173.000000  184.000000  184.000000   \n",
       "mean    445.956522    0.472826     2.0   29.877630    0.402174    0.380435   \n",
       "std     250.852161    0.500623     0.0   14.001077    0.601633    0.690963   \n",
       "min      10.000000    0.000000     2.0    0.670000    0.000000    0.000000   \n",
       "25%     234.500000    0.000000     2.0   23.000000    0.000000    0.000000   \n",
       "50%     435.500000    0.000000     2.0   29.000000    0.000000    0.000000   \n",
       "75%     668.000000    1.000000     2.0   36.000000    1.000000    1.000000   \n",
       "max     887.000000    1.000000     2.0   70.000000    3.000000    3.000000   \n",
       "\n",
       "             Fare  \n",
       "count  184.000000  \n",
       "mean    20.662183  \n",
       "std     13.417399  \n",
       "min      0.000000  \n",
       "25%     13.000000  \n",
       "50%     14.250000  \n",
       "75%     26.000000  \n",
       "max     73.500000  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset2.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>491.000000</td>\n",
       "      <td>491.000000</td>\n",
       "      <td>491.0</td>\n",
       "      <td>355.000000</td>\n",
       "      <td>491.000000</td>\n",
       "      <td>491.000000</td>\n",
       "      <td>491.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>439.154786</td>\n",
       "      <td>0.242363</td>\n",
       "      <td>3.0</td>\n",
       "      <td>25.140620</td>\n",
       "      <td>0.615071</td>\n",
       "      <td>0.393075</td>\n",
       "      <td>13.675550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>264.441453</td>\n",
       "      <td>0.428949</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.495398</td>\n",
       "      <td>1.374883</td>\n",
       "      <td>0.888861</td>\n",
       "      <td>11.778142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>200.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>432.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.050000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>666.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>15.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.0</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>69.550000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived  Pclass         Age       SibSp       Parch  \\\n",
       "count   491.000000  491.000000   491.0  355.000000  491.000000  491.000000   \n",
       "mean    439.154786    0.242363     3.0   25.140620    0.615071    0.393075   \n",
       "std     264.441453    0.428949     0.0   12.495398    1.374883    0.888861   \n",
       "min       1.000000    0.000000     3.0    0.420000    0.000000    0.000000   \n",
       "25%     200.000000    0.000000     3.0   18.000000    0.000000    0.000000   \n",
       "50%     432.000000    0.000000     3.0   24.000000    0.000000    0.000000   \n",
       "75%     666.500000    0.000000     3.0   32.000000    1.000000    0.000000   \n",
       "max     891.000000    1.000000     3.0   74.000000    8.000000    6.000000   \n",
       "\n",
       "             Fare  \n",
       "count  491.000000  \n",
       "mean    13.675550  \n",
       "std     11.778142  \n",
       "min      0.000000  \n",
       "25%      7.750000  \n",
       "50%      8.050000  \n",
       "75%     15.500000  \n",
       "max     69.550000  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset3.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding Survival Rates of Males and Females"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7420382165605095"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[train_data[\"Sex\"] == 'female'][\"Survived\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18890814558058924"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[train_data[\"Sex\"] == 'male'][\"Survived\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary Statistics\n",
    "\n",
    "After subsetting data, we may be interested in finding the mean or variance of the observations that satisfy a particular condition. One can call the `.mean()` method to find the mean of a particular variable of interest, when the `np.var()` function can be used to calculate the variance of the variable of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np.mean(x) returns the mean of x, x can be a column or a list\n",
    "# np.var(x) returns the variance of x\n",
    "# np.std(x) returns the standard deviation of x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>32.204208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>49.693429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.910400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>14.454200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   891.000000  891.000000  891.000000  714.000000  891.000000   \n",
       "mean    446.000000    0.383838    2.308642   29.699118    0.523008   \n",
       "std     257.353842    0.486592    0.836071   14.526497    1.102743   \n",
       "min       1.000000    0.000000    1.000000    0.420000    0.000000   \n",
       "25%     223.500000    0.000000    2.000000   20.125000    0.000000   \n",
       "50%     446.000000    0.000000    3.000000   28.000000    0.000000   \n",
       "75%     668.500000    1.000000    3.000000   38.000000    1.000000   \n",
       "max     891.000000    1.000000    3.000000   80.000000    8.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  891.000000  891.000000  \n",
       "mean     0.381594   32.204208  \n",
       "std      0.806057   49.693429  \n",
       "min      0.000000    0.000000  \n",
       "25%      0.000000    7.910400  \n",
       "50%      0.000000   14.454200  \n",
       "75%      0.000000   31.000000  \n",
       "max      6.000000  512.329200  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>376.000000</td>\n",
       "      <td>376.000000</td>\n",
       "      <td>376.000000</td>\n",
       "      <td>323.000000</td>\n",
       "      <td>376.000000</td>\n",
       "      <td>376.000000</td>\n",
       "      <td>376.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.970745</td>\n",
       "      <td>0.529255</td>\n",
       "      <td>1.670213</td>\n",
       "      <td>31.740712</td>\n",
       "      <td>0.986702</td>\n",
       "      <td>0.736702</td>\n",
       "      <td>62.990636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>254.335090</td>\n",
       "      <td>0.499808</td>\n",
       "      <td>0.824793</td>\n",
       "      <td>16.674316</td>\n",
       "      <td>1.483180</td>\n",
       "      <td>1.044265</td>\n",
       "      <td>64.795404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.830000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.212500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>236.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>26.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>455.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>39.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>661.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>76.729200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>890.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>512.329200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       PassengerId    Survived      Pclass         Age       SibSp  \\\n",
       "count   376.000000  376.000000  376.000000  323.000000  376.000000   \n",
       "mean    446.970745    0.529255    1.670213   31.740712    0.986702   \n",
       "std     254.335090    0.499808    0.824793   16.674316    1.483180   \n",
       "min       2.000000    0.000000    1.000000    0.830000    0.000000   \n",
       "25%     236.250000    0.000000    1.000000   21.000000    0.000000   \n",
       "50%     455.500000    1.000000    1.000000   32.000000    1.000000   \n",
       "75%     661.500000    1.000000    2.000000   44.000000    1.000000   \n",
       "max     890.000000    1.000000    3.000000   80.000000    8.000000   \n",
       "\n",
       "            Parch        Fare  \n",
       "count  376.000000  376.000000  \n",
       "mean     0.736702   62.990636  \n",
       "std      1.044265   64.795404  \n",
       "min      0.000000   20.212500  \n",
       "25%      0.000000   26.550000  \n",
       "50%      0.000000   39.000000  \n",
       "75%      1.000000   76.729200  \n",
       "max      6.000000  512.329200  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[train_data['Fare'] > 20].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    714.000000\n",
       "mean      29.699118\n",
       "std       14.526497\n",
       "min        0.420000\n",
       "25%       20.125000\n",
       "50%       28.000000\n",
       "75%       38.000000\n",
       "max       80.000000\n",
       "Name: Age, dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.describe()['Age']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to find the variance of a specific column, we can use the function `np.var`. Similarly, we can find the average fare using the function, `np.mean(train_data['Fare'])`. There exists other functions too, such as `np.median`.\n",
    "\n",
    "1. np.mean()\n",
    "2. np.max()\n",
    "3. np.min()\n",
    "4. np.var()\n",
    "5. np.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70.36486220472443"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# .mean() method only applies for dataframes\n",
    "subset = train_data[(train_data['Embarked'] == 'S') & (train_data['Pclass'] == 1)]\n",
    "subset_fare = subset['Fare']\n",
    "subset_fare_mean = np.mean(subset_fare)\n",
    "\n",
    "subset_fare_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70.36486220472443"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(train_data[(train_data['Embarked'] == 'S') & (train_data['Pclass'] == 1)][\"Fare\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3431.532010935895"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# np.mean() is a function, applicable to dataframes, series, vectors, lists etc.\n",
    "# axis = 0 implies summing over rows, axis = 1 implies summing over columns\n",
    "subset2 = train_data[(train_data['Embarked'] == 'S') & (train_data['Pclass'] == 1)]\n",
    "subset_fare = subset2['Fare']\n",
    "subset_fare_var = np.var(subset_fare)\n",
    "\n",
    "subset_fare_var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3431.532010935895"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# np.mean() is a function, applicable to dataframes, series, vectors, lists etc.\n",
    "np.var(train_data[(train_data['Embarked'] == 'S') & (train_data['Pclass'] == 1)][\"Fare\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average fare is 32.2042079685746\n",
      "Fare variance is 2466.6653116850434\n"
     ]
    }
   ],
   "source": [
    "print('Average fare is', np.mean(train_data['Fare']))\n",
    "print('Fare variance is', np.var(train_data['Fare']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-class assignment:\n",
    "\n",
    "1. Are males or females more likely to survive?\n",
    "2. Are passengers with 1 or more parents/children (Parch) more likely to survive?\n",
    "3. Are passengers with 1 or more sibling/spouse (SibSp) more likely to survive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7420382165605095\n",
      "0.18890814558058924\n"
     ]
    }
   ],
   "source": [
    "subsetF = train_data[train_data['Sex'] == 'female']\n",
    "subsetM = train_data[train_data['Sex'] == 'male']\n",
    "\n",
    "print(np.mean(subsetF['Survived']))\n",
    "print(np.mean(subsetM['Survived']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4664310954063604\n",
      "0.34539473684210525\n"
     ]
    }
   ],
   "source": [
    "subsetMore = train_data[train_data['SibSp'] >= 1]\n",
    "subsetLess = train_data[train_data['SibSp'] < 1]\n",
    "\n",
    "print(subsetMore['Survived'].mean())\n",
    "print(subsetLess['Survived'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5117370892018779\n",
      "0.34365781710914456\n"
     ]
    }
   ],
   "source": [
    "subsetMore = train_data[train_data['Parch'] >= 1]\n",
    "subsetLess = train_data[train_data['Parch'] < 1]\n",
    "\n",
    "print(np.mean(subsetMore['Survived']))      \n",
    "print(subsetLess['Survived'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Data Manipulation\n",
    "Now that we have a basic understanding of data manipulation in Python, let's move on to more interesting stuff:\n",
    "\n",
    "1. Locating and imputing missing values in the data\n",
    "2. Pivot Tables\n",
    "3. Preliminary Text Cleaning\n",
    "4. Groupby\n",
    "\n",
    "---\n",
    "\n",
    "### 1. Locating and Imputing Missing Values\n",
    "To identify all missing values in the dataset, we can use the function `pd.isnull()`, and pass in our training data as the argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass   Name    Sex    Age  SibSp  Parch  Ticket  \\\n",
       "0        False     False   False  False  False  False  False  False   False   \n",
       "1        False     False   False  False  False  False  False  False   False   \n",
       "2        False     False   False  False  False  False  False  False   False   \n",
       "3        False     False   False  False  False  False  False  False   False   \n",
       "4        False     False   False  False  False  False  False  False   False   \n",
       "\n",
       "    Fare  Cabin  Embarked  \n",
       "0  False   True     False  \n",
       "1  False  False     False  \n",
       "2  False   True     False  \n",
       "3  False  False     False  \n",
       "4  False   True     False  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pd.isnull() is a function from the pandas library that returns True when the data is missing\n",
    "isnull_data = pd.isnull(train_data)\n",
    "isnull_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age            177\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin          687\n",
       "Embarked         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# False = 0, True = 1\n",
    "np.sum(isnull_data, axis=0) # By default sum across rows, to sum across columns, set axis=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To obtain the total number of missing values for each column, we can use the function, `np.sum()`. Note that `np.sum()` sums across each column. To sum across each row, we can pass in an additional argument, `axis=1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['C', 'Q', 'S'], dtype=object)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#otherwise, you can use the command, ?np.sum(), to check the arguments used in the function\n",
    "# pd.notnull() is the opposite of pd.isnull()\n",
    "np.unique(train_data[pd.notnull(train_data[\"Embarked\"])][\"Embarked\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age            177\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin          687\n",
       "Embarked         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(pd.isnull(train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>204</td>\n",
       "      <td>889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>891</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>681</td>\n",
       "      <td>NaN</td>\n",
       "      <td>147</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lundahl, Mr. Johan Svensson</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>NaN</td>\n",
       "      <td>B96 B98</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>577</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.204208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.693429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.910400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        PassengerId    Survived      Pclass                         Name  \\\n",
       "count    891.000000  891.000000  891.000000                          891   \n",
       "unique          NaN         NaN         NaN                          891   \n",
       "top             NaN         NaN         NaN  Lundahl, Mr. Johan Svensson   \n",
       "freq            NaN         NaN         NaN                            1   \n",
       "mean     446.000000    0.383838    2.308642                          NaN   \n",
       "std      257.353842    0.486592    0.836071                          NaN   \n",
       "min        1.000000    0.000000    1.000000                          NaN   \n",
       "25%      223.500000    0.000000    2.000000                          NaN   \n",
       "50%      446.000000    0.000000    3.000000                          NaN   \n",
       "75%      668.500000    1.000000    3.000000                          NaN   \n",
       "max      891.000000    1.000000    3.000000                          NaN   \n",
       "\n",
       "         Sex         Age       SibSp       Parch    Ticket        Fare  \\\n",
       "count    891  714.000000  891.000000  891.000000       891  891.000000   \n",
       "unique     2         NaN         NaN         NaN       681         NaN   \n",
       "top     male         NaN         NaN         NaN  CA. 2343         NaN   \n",
       "freq     577         NaN         NaN         NaN         7         NaN   \n",
       "mean     NaN   29.699118    0.523008    0.381594       NaN   32.204208   \n",
       "std      NaN   14.526497    1.102743    0.806057       NaN   49.693429   \n",
       "min      NaN    0.420000    0.000000    0.000000       NaN    0.000000   \n",
       "25%      NaN   20.125000    0.000000    0.000000       NaN    7.910400   \n",
       "50%      NaN   28.000000    0.000000    0.000000       NaN   14.454200   \n",
       "75%      NaN   38.000000    1.000000    0.000000       NaN   31.000000   \n",
       "max      NaN   80.000000    8.000000    6.000000       NaN  512.329200   \n",
       "\n",
       "          Cabin Embarked  \n",
       "count       204      889  \n",
       "unique      147        3  \n",
       "top     B96 B98        S  \n",
       "freq          4      644  \n",
       "mean        NaN      NaN  \n",
       "std         NaN      NaN  \n",
       "min         NaN      NaN  \n",
       "25%         NaN      NaN  \n",
       "50%         NaN      NaN  \n",
       "75%         NaN      NaN  \n",
       "max         NaN      NaN  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To talk about the data types when importing csv files in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data Cleaning:\n",
    "\n",
    "It appears that many of the Age and Cabin data are missing. In this case, we can do the following, depending on the column category:\n",
    "\n",
    "1. Fill the mean value (with `df.fillna()`)\n",
    "2. Create a new category\n",
    "3. Drop the columns\n",
    "\n",
    "---\n",
    "\n",
    "The DataFrame method `fillna()`, fills all missing values with an integer, float, or string provided by the user. Setting the argument, `inplace=True` results in the dataframe being altered. Otherwise, you can leave out the argument `inplace=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      NoCabin\n",
       "1          C85\n",
       "2      NoCabin\n",
       "3         C123\n",
       "4      NoCabin\n",
       "        ...   \n",
       "886    NoCabin\n",
       "887        B42\n",
       "888    NoCabin\n",
       "889       C148\n",
       "890    NoCabin\n",
       "Name: Cabin, Length: 891, dtype: object"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Start with Cabin variable\n",
    "\n",
    "# replaces NA values with NoCabin for the train_data Cabin column\n",
    "train_data['Cabin'].fillna(\"NoCabin\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       NaN\n",
       "1       C85\n",
       "2       NaN\n",
       "3      C123\n",
       "4       NaN\n",
       "       ... \n",
       "886     NaN\n",
       "887     B42\n",
       "888     NaN\n",
       "889    C148\n",
       "890     NaN\n",
       "Name: Cabin, Length: 891, dtype: object"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data['Cabin']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>887</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>888</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>889</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>890</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>891</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>Q</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>891 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass  \\\n",
       "0              1         0       3   \n",
       "1              2         1       1   \n",
       "2              3         1       3   \n",
       "3              4         1       1   \n",
       "4              5         0       3   \n",
       "..           ...       ...     ...   \n",
       "886          887         0       2   \n",
       "887          888         1       1   \n",
       "888          889         0       3   \n",
       "889          890         1       1   \n",
       "890          891         0       3   \n",
       "\n",
       "                                                  Name     Sex   Age  SibSp  \\\n",
       "0                              Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1    Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                               Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3         Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                             Allen, Mr. William Henry    male  35.0      0   \n",
       "..                                                 ...     ...   ...    ...   \n",
       "886                              Montvila, Rev. Juozas    male  27.0      0   \n",
       "887                       Graham, Miss. Margaret Edith  female  19.0      0   \n",
       "888           Johnston, Miss. Catherine Helen \"Carrie\"  female   NaN      1   \n",
       "889                              Behr, Mr. Karl Howell    male  26.0      0   \n",
       "890                                Dooley, Mr. Patrick    male  32.0      0   \n",
       "\n",
       "     Parch            Ticket     Fare    Cabin Embarked  \n",
       "0        0         A/5 21171   7.2500  NoCabin        S  \n",
       "1        0          PC 17599  71.2833      C85        C  \n",
       "2        0  STON/O2. 3101282   7.9250  NoCabin        S  \n",
       "3        0            113803  53.1000     C123        S  \n",
       "4        0            373450   8.0500  NoCabin        S  \n",
       "..     ...               ...      ...      ...      ...  \n",
       "886      0            211536  13.0000  NoCabin        S  \n",
       "887      0            112053  30.0000      B42        S  \n",
       "888      2        W./C. 6607  23.4500  NoCabin        S  \n",
       "889      0            111369  30.0000     C148        C  \n",
       "890      0            370376   7.7500  NoCabin        Q  \n",
       "\n",
       "[891 rows x 12 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# .copy() makes a new copy of the data\n",
    "data_copy = train_data.copy()\n",
    "\n",
    "# replaces Cabin column in copied data with the no-NA version\n",
    "data_copy[\"Cabin\"] = train_data['Cabin'].fillna(\"NoCabin\")\n",
    "\n",
    "data_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>891</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>681</td>\n",
       "      <td>NaN</td>\n",
       "      <td>148</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Wells, Miss. Joan</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>347082</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>577</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>687</td>\n",
       "      <td>644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.204208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.693429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.910400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        PassengerId    Survived      Pclass               Name   Sex  \\\n",
       "count    891.000000  891.000000  891.000000                891   891   \n",
       "unique          NaN         NaN         NaN                891     2   \n",
       "top             NaN         NaN         NaN  Wells, Miss. Joan  male   \n",
       "freq            NaN         NaN         NaN                  1   577   \n",
       "mean     446.000000    0.383838    2.308642                NaN   NaN   \n",
       "std      257.353842    0.486592    0.836071                NaN   NaN   \n",
       "min        1.000000    0.000000    1.000000                NaN   NaN   \n",
       "25%      223.500000    0.000000    2.000000                NaN   NaN   \n",
       "50%      446.000000    0.000000    3.000000                NaN   NaN   \n",
       "75%      668.500000    1.000000    3.000000                NaN   NaN   \n",
       "max      891.000000    1.000000    3.000000                NaN   NaN   \n",
       "\n",
       "               Age       SibSp       Parch  Ticket        Fare    Cabin  \\\n",
       "count   714.000000  891.000000  891.000000     891  891.000000      891   \n",
       "unique         NaN         NaN         NaN     681         NaN      148   \n",
       "top            NaN         NaN         NaN  347082         NaN  NoCabin   \n",
       "freq           NaN         NaN         NaN       7         NaN      687   \n",
       "mean     29.699118    0.523008    0.381594     NaN   32.204208      NaN   \n",
       "std      14.526497    1.102743    0.806057     NaN   49.693429      NaN   \n",
       "min       0.420000    0.000000    0.000000     NaN    0.000000      NaN   \n",
       "25%      20.125000    0.000000    0.000000     NaN    7.910400      NaN   \n",
       "50%      28.000000    0.000000    0.000000     NaN   14.454200      NaN   \n",
       "75%      38.000000    1.000000    0.000000     NaN   31.000000      NaN   \n",
       "max      80.000000    8.000000    6.000000     NaN  512.329200      NaN   \n",
       "\n",
       "       Embarked  \n",
       "count       889  \n",
       "unique        3  \n",
       "top           S  \n",
       "freq        644  \n",
       "mean        NaN  \n",
       "std         NaN  \n",
       "min         NaN  \n",
       "25%         NaN  \n",
       "50%         NaN  \n",
       "75%         NaN  \n",
       "max         NaN  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_copy.describe(include='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age            177\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin            0\n",
       "Embarked         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# S is the most frequent embarkation point, propose to replace missing embarkation points with \"S\"\n",
    "data_copy[\"Embarked\"] = train_data['Embarked'].fillna(\"S\")\n",
    "\n",
    "np.sum(pd.isnull(data_copy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>891</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>681</td>\n",
       "      <td>NaN</td>\n",
       "      <td>148</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lundahl, Mr. Johan Svensson</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>577</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>687</td>\n",
       "      <td>646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.204208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.693429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.910400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        PassengerId    Survived      Pclass                         Name  \\\n",
       "count    891.000000  891.000000  891.000000                          891   \n",
       "unique          NaN         NaN         NaN                          891   \n",
       "top             NaN         NaN         NaN  Lundahl, Mr. Johan Svensson   \n",
       "freq            NaN         NaN         NaN                            1   \n",
       "mean     446.000000    0.383838    2.308642                          NaN   \n",
       "std      257.353842    0.486592    0.836071                          NaN   \n",
       "min        1.000000    0.000000    1.000000                          NaN   \n",
       "25%      223.500000    0.000000    2.000000                          NaN   \n",
       "50%      446.000000    0.000000    3.000000                          NaN   \n",
       "75%      668.500000    1.000000    3.000000                          NaN   \n",
       "max      891.000000    1.000000    3.000000                          NaN   \n",
       "\n",
       "         Sex         Age       SibSp       Parch    Ticket        Fare  \\\n",
       "count    891  714.000000  891.000000  891.000000       891  891.000000   \n",
       "unique     2         NaN         NaN         NaN       681         NaN   \n",
       "top     male         NaN         NaN         NaN  CA. 2343         NaN   \n",
       "freq     577         NaN         NaN         NaN         7         NaN   \n",
       "mean     NaN   29.699118    0.523008    0.381594       NaN   32.204208   \n",
       "std      NaN   14.526497    1.102743    0.806057       NaN   49.693429   \n",
       "min      NaN    0.420000    0.000000    0.000000       NaN    0.000000   \n",
       "25%      NaN   20.125000    0.000000    0.000000       NaN    7.910400   \n",
       "50%      NaN   28.000000    0.000000    0.000000       NaN   14.454200   \n",
       "75%      NaN   38.000000    1.000000    0.000000       NaN   31.000000   \n",
       "max      NaN   80.000000    8.000000    6.000000       NaN  512.329200   \n",
       "\n",
       "          Cabin Embarked  \n",
       "count       891      891  \n",
       "unique      148        3  \n",
       "top     NoCabin        S  \n",
       "freq        687      646  \n",
       "mean        NaN      NaN  \n",
       "std         NaN      NaN  \n",
       "min         NaN      NaN  \n",
       "25%         NaN      NaN  \n",
       "50%         NaN      NaN  \n",
       "75%         NaN      NaN  \n",
       "max         NaN      NaN  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_copy.describe(include='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.69911764705882"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(train_data['Age'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId    0\n",
       "Survived       0\n",
       "Pclass         0\n",
       "Name           0\n",
       "Sex            0\n",
       "Age            0\n",
       "SibSp          0\n",
       "Parch          0\n",
       "Ticket         0\n",
       "Fare           0\n",
       "Cabin          0\n",
       "Embarked       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Replace missing \"ages\" with the average age of the people in the Titanic\n",
    "mean_age = np.mean(train_data['Age'])\n",
    "data_copy[\"Age\"] = train_data['Age'].fillna(mean_age)\n",
    "\n",
    "np.sum(pd.isnull(data_copy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_copy.to_csv('data/titanic/cleaned_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Lambda programming \n",
    "# This is another way of saying y is equivalent to the t_or_f function\n",
    "# y = lambda x: t_or_f(x)\n",
    "# print(x(5))\n",
    "\n",
    "def t_or_f(val):\n",
    "    '''\n",
    "    Function that accepts as input a string variable and verifies if it's equivalent to \"NoCabin\"\n",
    "    If it's \"NoCabin\", return False. Else, return True.\n",
    "    '''\n",
    "    if val == \"NoCabin\":\n",
    "        return False\n",
    "    else:\n",
    "        return True\n",
    "\n",
    "# Traditional method\n",
    "''' \n",
    "cabin_to_stay = []\n",
    "for c in data_copy[\"Cabin\"]:\n",
    "    boo = t_or_f(c)\n",
    "    cabin_to_stay.append(boo)\n",
    "    \n",
    "cabin_to_stay\n",
    "'''\n",
    "\n",
    "# Functional programming & .apply() method\n",
    "# Applying the t_or_f function to the data_copy column, \"Cabin\". Here, \"x\" is a placeholder/pointer\n",
    "data_copy[\"CabinToStay\"] = data_copy[\"Cabin\"].apply(lambda x: t_or_f(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "def have_cabin(cabin):\n",
    "    if cabin == 'NoCabin': \n",
    "        return False\n",
    "    else: \n",
    "        return True\n",
    "    \n",
    "# Here i am applying the have_cabin function to the cabin column of the data_copy dataframe\n",
    "# The variable after the keyword lambda is similar to a pointer in a for loop\n",
    "data_copy['CabinToStay2'] = data_copy['Cabin'].apply(lambda cabin: have_cabin(cabin))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>CabinToStay</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>887</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>888</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>889</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>890</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>891</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>Q</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>891 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass  \\\n",
       "0              1         0       3   \n",
       "1              2         1       1   \n",
       "2              3         1       3   \n",
       "3              4         1       1   \n",
       "4              5         0       3   \n",
       "..           ...       ...     ...   \n",
       "886          887         0       2   \n",
       "887          888         1       1   \n",
       "888          889         0       3   \n",
       "889          890         1       1   \n",
       "890          891         0       3   \n",
       "\n",
       "                                                  Name     Sex        Age  \\\n",
       "0                              Braund, Mr. Owen Harris    male  22.000000   \n",
       "1    Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.000000   \n",
       "2                               Heikkinen, Miss. Laina  female  26.000000   \n",
       "3         Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.000000   \n",
       "4                             Allen, Mr. William Henry    male  35.000000   \n",
       "..                                                 ...     ...        ...   \n",
       "886                              Montvila, Rev. Juozas    male  27.000000   \n",
       "887                       Graham, Miss. Margaret Edith  female  19.000000   \n",
       "888           Johnston, Miss. Catherine Helen \"Carrie\"  female  29.699118   \n",
       "889                              Behr, Mr. Karl Howell    male  26.000000   \n",
       "890                                Dooley, Mr. Patrick    male  32.000000   \n",
       "\n",
       "     SibSp  Parch            Ticket     Fare    Cabin Embarked  CabinToStay  \n",
       "0        1      0         A/5 21171   7.2500  NoCabin        S        False  \n",
       "1        1      0          PC 17599  71.2833      C85        C         True  \n",
       "2        0      0  STON/O2. 3101282   7.9250  NoCabin        S        False  \n",
       "3        1      0            113803  53.1000     C123        S         True  \n",
       "4        0      0            373450   8.0500  NoCabin        S        False  \n",
       "..     ...    ...               ...      ...      ...      ...          ...  \n",
       "886      0      0            211536  13.0000  NoCabin        S        False  \n",
       "887      0      0            112053  30.0000      B42        S         True  \n",
       "888      1      2        W./C. 6607  23.4500  NoCabin        S        False  \n",
       "889      0      0            111369  30.0000     C148        C         True  \n",
       "890      0      0            370376   7.7500  NoCabin        Q        False  \n",
       "\n",
       "[891 rows x 13 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cabin_to_stay_list = [] \n",
    "\n",
    "# Iterate through the cabin column, if the element is \"NoCabin\", appends False to the list\n",
    "# if the element is anything else, appends True to the list\n",
    "for cabin in data_copy['Cabin']:\n",
    "    if cabin == 'NoCabin':\n",
    "        cabin_to_stay_list.append(False)\n",
    "    else:\n",
    "        cabin_to_stay_list.append(True)\n",
    "\n",
    "# Creating a new column called 'CabinToStay' in the duplicate data\n",
    "data_copy['CabinToStay'] = cabin_to_stay_list\n",
    "data_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>CabinToStay</th>\n",
       "      <th>CabinToStay2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>887</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>888</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>889</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>S</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>890</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>891</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NoCabin</td>\n",
       "      <td>Q</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>891 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass  \\\n",
       "0              1         0       3   \n",
       "1              2         1       1   \n",
       "2              3         1       3   \n",
       "3              4         1       1   \n",
       "4              5         0       3   \n",
       "..           ...       ...     ...   \n",
       "886          887         0       2   \n",
       "887          888         1       1   \n",
       "888          889         0       3   \n",
       "889          890         1       1   \n",
       "890          891         0       3   \n",
       "\n",
       "                                                  Name     Sex        Age  \\\n",
       "0                              Braund, Mr. Owen Harris    male  22.000000   \n",
       "1    Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.000000   \n",
       "2                               Heikkinen, Miss. Laina  female  26.000000   \n",
       "3         Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.000000   \n",
       "4                             Allen, Mr. William Henry    male  35.000000   \n",
       "..                                                 ...     ...        ...   \n",
       "886                              Montvila, Rev. Juozas    male  27.000000   \n",
       "887                       Graham, Miss. Margaret Edith  female  19.000000   \n",
       "888           Johnston, Miss. Catherine Helen \"Carrie\"  female  29.699118   \n",
       "889                              Behr, Mr. Karl Howell    male  26.000000   \n",
       "890                                Dooley, Mr. Patrick    male  32.000000   \n",
       "\n",
       "     SibSp  Parch            Ticket     Fare    Cabin Embarked  CabinToStay  \\\n",
       "0        1      0         A/5 21171   7.2500  NoCabin        S        False   \n",
       "1        1      0          PC 17599  71.2833      C85        C         True   \n",
       "2        0      0  STON/O2. 3101282   7.9250  NoCabin        S        False   \n",
       "3        1      0            113803  53.1000     C123        S         True   \n",
       "4        0      0            373450   8.0500  NoCabin        S        False   \n",
       "..     ...    ...               ...      ...      ...      ...          ...   \n",
       "886      0      0            211536  13.0000  NoCabin        S        False   \n",
       "887      0      0            112053  30.0000      B42        S         True   \n",
       "888      1      2        W./C. 6607  23.4500  NoCabin        S        False   \n",
       "889      0      0            111369  30.0000     C148        C         True   \n",
       "890      0      0            370376   7.7500  NoCabin        Q        False   \n",
       "\n",
       "     CabinToStay2  \n",
       "0           False  \n",
       "1            True  \n",
       "2           False  \n",
       "3            True  \n",
       "4           False  \n",
       "..            ...  \n",
       "886         False  \n",
       "887          True  \n",
       "888         False  \n",
       "889          True  \n",
       "890         False  \n",
       "\n",
       "[891 rows x 14 columns]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we note that there were many missing values for the variable, \"Cabin\". One may posit that these individuals may not have been assigned to any cabins. We can create a new category, by filling the missing values with a particular string, \"No Cabin\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Pivot Tables\n",
    "\n",
    "We can use the method, `pivot_table` to generate conditional summary statistics. For example, one might be interested in the number of male or female passengers who survived, and embarked at different locations. Usually, the indexes and the columns associated with any pivot tables are categorical/discrete i.e. not continuous in nature.\n",
    "\n",
    "1. Gender of passengers\n",
    "2. Embarkation point\n",
    "3. Survival Rate\n",
    "\n",
    "In that case, one can use the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Sex</th>\n",
       "      <th>female</th>\n",
       "      <th>male</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>106.125798</td>\n",
       "      <td>67.226127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21.970121</td>\n",
       "      <td>19.741782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.118810</td>\n",
       "      <td>12.661633</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Sex         female       male\n",
       "Pclass                       \n",
       "1       106.125798  67.226127\n",
       "2        21.970121  19.741782\n",
       "3        16.118810  12.661633"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# aggfunc set to mean, because I'm interested in average survival rate\n",
    "# this is conditional on gender and embarkation point\n",
    "data_copy.pivot_table(values='Fare', \n",
    "                      index='Pclass', \n",
    "                      columns='Sex',\n",
    "                      aggfunc='mean')\n",
    "\n",
    "# For pivot tables, we want to condition on categorical columns (at most a few categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-class assignment\n",
    "\n",
    "Compare the mean survival rates of males and females embarking at different points.\n",
    "\n",
    "Helper functions:\n",
    "1. .pivot_table()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-class assignment\n",
    "\n",
    "Compare the mean survival rates of different passenger classes and whether they have a cabin to stay in.\n",
    "\n",
    "Please use the \"CabinToStay\" column on the new dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'C' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-55-11f253692898>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mC\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mM\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mQ\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mM\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mS\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mM\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'C' is not defined"
     ]
    }
   ],
   "source": [
    "1, T: 0.66\n",
    "2, T: 0.81\n",
    "3, T: 0.5\n",
    "\n",
    "1, F: \n",
    "2, F: \n",
    "3, F: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Embarked</th>\n",
       "      <th>C</th>\n",
       "      <th>Q</th>\n",
       "      <th>S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>73</td>\n",
       "      <td>36</td>\n",
       "      <td>205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>95</td>\n",
       "      <td>41</td>\n",
       "      <td>441</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Embarked   C   Q    S\n",
       "Sex                  \n",
       "female    73  36  205\n",
       "male      95  41  441"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To reset index, use data_copy.reset_index(inplace=True); irreversible\n",
    "\n",
    "data_copy.pivot_table(values = \"Age\",\n",
    "                      index = \"Sex\",\n",
    "                      columns = \"Embarked\",\n",
    "                      aggfunc = \"count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we created a pivot table, with the arguments `values='Survived`, `index='Sex'`, `columns='Embarked'` and `aggfunc='mean'`. This means that we want to find the mean survival rates, by gender and embarkation point. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In-class assignment:\n",
    "\n",
    "1. Using the DataFrame method, `pivot_table`, find the average fare paid by female passengers who survived. - \"Fare\", \"Sex\", \"Survived\"\n",
    "\n",
    "\n",
    "2. Using the DataFrame method, `pivot_table`, find the average age of male passengers who were of passenger class 1. - \"Age\", \"Sex\", \"Pclass\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Survived</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>23.024385</td>\n",
       "      <td>51.938573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>21.960993</td>\n",
       "      <td>40.821484</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Survived          0          1\n",
       "Sex                           \n",
       "female    23.024385  51.938573\n",
       "male      21.960993  40.821484"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Your code here\n",
    "data_copy.pivot_table(values='Fare', \n",
    "                      index='Sex', \n",
    "                      columns='Survived',\n",
    "                      aggfunc='mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Pclass</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>34.611765</td>\n",
       "      <td>28.722973</td>\n",
       "      <td>21.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>41.281386</td>\n",
       "      <td>30.740707</td>\n",
       "      <td>26.507589</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Pclass          1          2          3\n",
       "Sex                                    \n",
       "female  34.611765  28.722973  21.750000\n",
       "male    41.281386  30.740707  26.507589"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Your code here\n",
    "train_data.pivot_table(values='Age', \n",
    "                       index='Sex', \n",
    "                       columns='Pclass',\n",
    "                       aggfunc='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interested in pivoting around more than 1 column, we can pass in a list of variables\n",
    "\n",
    "E.g. Average Fare of M/F, whether they Survived, where they Embarked - \"Fare\", \"Sex\", \"Survived\", \"Embarked\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <th colspan=\"9\" halign=\"left\">0</th>\n",
       "      <th colspan=\"9\" halign=\"left\">1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Embarked</th>\n",
       "      <th colspan=\"3\" halign=\"left\">C</th>\n",
       "      <th colspan=\"3\" halign=\"left\">Q</th>\n",
       "      <th colspan=\"3\" halign=\"left\">S</th>\n",
       "      <th colspan=\"3\" halign=\"left\">C</th>\n",
       "      <th colspan=\"3\" halign=\"left\">Q</th>\n",
       "      <th colspan=\"3\" halign=\"left\">S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>28.71250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.653125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.904633</td>\n",
       "      <td>151.550000</td>\n",
       "      <td>18.250000</td>\n",
       "      <td>21.969018</td>\n",
       "      <td>117.710019</td>\n",
       "      <td>25.268457</td>\n",
       "      <td>14.71722</td>\n",
       "      <td>90.0</td>\n",
       "      <td>12.35</td>\n",
       "      <td>10.084033</td>\n",
       "      <td>96.743298</td>\n",
       "      <td>22.272951</td>\n",
       "      <td>13.171842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>80.84716</td>\n",
       "      <td>25.418225</td>\n",
       "      <td>8.720842</td>\n",
       "      <td>90.0</td>\n",
       "      <td>12.35</td>\n",
       "      <td>11.841550</td>\n",
       "      <td>53.563316</td>\n",
       "      <td>18.997561</td>\n",
       "      <td>12.758689</td>\n",
       "      <td>112.197806</td>\n",
       "      <td>25.433350</td>\n",
       "      <td>11.43584</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12.916667</td>\n",
       "      <td>51.832739</td>\n",
       "      <td>20.516667</td>\n",
       "      <td>17.033450</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Survived         0                                                            \\\n",
       "Embarked         C                           Q                             S   \n",
       "Pclass           1          2          3     1      2          3           1   \n",
       "Sex                                                                            \n",
       "female    28.71250        NaN  14.653125   NaN    NaN  10.904633  151.550000   \n",
       "male      80.84716  25.418225   8.720842  90.0  12.35  11.841550   53.563316   \n",
       "\n",
       "Survived                                 1                                    \\\n",
       "Embarked                                 C                          Q          \n",
       "Pclass            2          3           1          2         3     1      2   \n",
       "Sex                                                                            \n",
       "female    18.250000  21.969018  117.710019  25.268457  14.71722  90.0  12.35   \n",
       "male      18.997561  12.758689  112.197806  25.433350  11.43584   NaN    NaN   \n",
       "\n",
       "Survived                                              \n",
       "Embarked                     S                        \n",
       "Pclass            3          1          2          3  \n",
       "Sex                                                   \n",
       "female    10.084033  96.743298  22.272951  13.171842  \n",
       "male      12.916667  51.832739  20.516667  17.033450  "
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.pivot_table(values='Fare', \n",
    "                       index='Sex', \n",
    "                       columns=['Survived', 'Embarked', 'Pclass'],\n",
    "                       aggfunc='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Preliminary Text Analysis\n",
    "\n",
    "When provided with textual data, we can conduct text analysis on the data. Sometimes, we want to extract certain information from a text, for example, we may be interested in extracting the email addresses in an email.\n",
    "\n",
    "As a start, let's take a closer look at the \"Ticket\" column of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Futrelle\n",
      "Mrs. Jacques Heath (Lily May Peel)\n",
      "Mrs\n",
      "Jacques Heath (Lily May Peel)\n"
     ]
    }
   ],
   "source": [
    "x = \"Futrelle, Mrs. Jacques Heath (Lily May Peel)\"\n",
    "\n",
    "# Split is a method that works only for strings, and split on all , (comma followed by a space) within the string\n",
    "last_name = x.split(\", \")[0]\n",
    "hon_first_name = x.split(\", \")[1]\n",
    "\n",
    "print(last_name)\n",
    "print(hon_first_name)\n",
    "\n",
    "honorific = hon_first_name.split(\". \")[0]\n",
    "first_name = hon_first_name.split(\". \")[1]\n",
    "\n",
    "print(honorific)\n",
    "print(first_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_honorific_lastname(name):\n",
    "    last_name = name.split(\", \")[0]\n",
    "    hon_first_name = name.split(\", \")[1]\n",
    "    \n",
    "    honorific = hon_first_name.split(\". \")[0]\n",
    "    first_name = hon_first_name.split(\". \")[1]\n",
    "    \n",
    "    return last_name, honorific, first_name\n",
    "\n",
    "# how to assign and obtain elements from the function that returns 3 output\n",
    "'''\n",
    "last_name_x, honorific_x, first_name_x = get_honorific_lastname(x)\n",
    "get_honorific_lastname(x)[2]\n",
    "'''\n",
    "# Last Name\n",
    "data_copy[\"LastName\"] = data_copy[\"Name\"].apply(lambda x: get_honorific_lastname(x)[0])\n",
    "\n",
    "# Honorific\n",
    "data_copy[\"Honorific\"] = data_copy[\"Name\"].apply(lambda x: get_honorific_lastname(x)[1])\n",
    "\n",
    "# First Name\n",
    "data_copy[\"FirstName\"] = data_copy[\"Name\"].apply(lambda x: get_honorific_lastname(x)[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-class assignment:\n",
    "\n",
    "Use a pivot table to find the average survival rates of different honorifics and different genders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Sex</th>\n",
       "      <th>female</th>\n",
       "      <th>male</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Honorific</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Capt</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Col</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Don</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dr</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jonkheer</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lady</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Major</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Master</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.575000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Miss</th>\n",
       "      <td>0.697802</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mlle</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mme</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mr</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.156673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mrs</th>\n",
       "      <td>0.792000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ms</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rev</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sir</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the Countess</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Sex             female      male\n",
       "Honorific                       \n",
       "Capt               NaN  0.000000\n",
       "Col                NaN  0.500000\n",
       "Don                NaN  0.000000\n",
       "Dr            1.000000  0.333333\n",
       "Jonkheer           NaN  0.000000\n",
       "Lady          1.000000       NaN\n",
       "Major              NaN  0.500000\n",
       "Master             NaN  0.575000\n",
       "Miss          0.697802       NaN\n",
       "Mlle          1.000000       NaN\n",
       "Mme           1.000000       NaN\n",
       "Mr                 NaN  0.156673\n",
       "Mrs           0.792000       NaN\n",
       "Ms            1.000000       NaN\n",
       "Rev                NaN  0.000000\n",
       "Sir                NaN  1.000000\n",
       "the Countess  1.000000       NaN"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_copy.pivot_table(values = \"Survived\",\n",
    "                      index = \"Honorific\",\n",
    "                      columns = \"Sex\",\n",
    "                      aggfunc = \"mean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One may posit that the numbers on the ticket numbers may be informative of the ticket type of the passengers. For example, one can test the hypothesis that a larger ticket number may be more expensive than a cheaper ticket. Before we can test this hypothesis, we need to extract the \"numeric\" part of the ticket.\n",
    "\n",
    "To do so, we first note that the \"character\" part of the ticket and the \"numeric\" part of the ticket are split by a space. Thus, we can split each ticket into 2 parts by applying the \"split\" function on a string to obtain a list of length 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_copy contains everything we have done up till this point\n",
    "data_copy.to_csv(\"/Users/ngbolin/Downloads/PythonProgrammingWorkshop-master/data/titanic/cleaned_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Functional/Lambda Programming\n",
    "\n",
    "Note that the ticket column contains both integers and characters. We can define a simple function that separates on the space. This is a good time to introduce lambda programming. A lambda function is an anonymous one - it does not have a \"name\". Typically, programmers use lambda functions when they are intending to use the function once.\n",
    "\n",
    "\n",
    "Below, we show an example of a lambda function. Here, the argument for the lambda function is `x`, and the lambda function basically splits the argument, `x` by a space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split on \", \" for each name in the column \"Name\"\n",
    "# First i obtain a list, then i extract last element (a string)\n",
    "# Then i split on the string to obtain another list\n",
    "# Obtain first element of this list and name column, \"Title\"\n",
    "data_copy['Title'] = data_copy['Name'].apply(lambda x: (x.split(', ')[-1]).split('. ')[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we are only interested in the integer component of the ticket code. How can we do so? It turns out that to extract the last term in a list, we can subset the list using `[-1]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'c'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "['a', 'b', 'c'][-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if, instead, we are interested in the string component of the ticket code? How can we get it? One way of deriving the string component of the code is through the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A/5', 'PC', 'STON/O2.', 'Null', 'Null']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_string_component_from_ticket(data, column='Ticket'):\n",
    "    string_lst = []\n",
    "    for row in new_train_data['Ticket'].apply(lambda x: x.split(' ')):\n",
    "        if len(row) > 1: string_lst.append(row[0]) # if there are more than 2 args, 1st arg is the string component\n",
    "        else: string_lst.append('Null') # otherwise, there are no string components\n",
    "    return string_lst\n",
    "        \n",
    "get_string_component_from_ticket(new_train_data)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the previous 2 chunks of codes, we can dissect the Ticket column into 2 components: the string component and the integer component. Let's write some code to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Title</th>\n",
       "      <th>Ticket_Int</th>\n",
       "      <th>Ticket_Str</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Mr</td>\n",
       "      <td>21171</td>\n",
       "      <td>A/5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>17599</td>\n",
       "      <td>PC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Miss</td>\n",
       "      <td>3101282</td>\n",
       "      <td>STON/O2.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>113803</td>\n",
       "      <td>Null</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Mr</td>\n",
       "      <td>373450</td>\n",
       "      <td>Null</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "\n",
       "                                                          Name     Sex   Age  \\\n",
       "PassengerId                                                                    \n",
       "1                                      Braund, Mr. Owen Harris    male  22.0   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n",
       "3                                       Heikkinen, Miss. Laina  female  26.0   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n",
       "5                                     Allen, Mr. William Henry    male  35.0   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Embarked Title  \\\n",
       "PassengerId                                                           \n",
       "1                1      0         A/5 21171   7.2500        S    Mr   \n",
       "2                1      0          PC 17599  71.2833        C   Mrs   \n",
       "3                0      0  STON/O2. 3101282   7.9250        S  Miss   \n",
       "4                1      0            113803  53.1000        S   Mrs   \n",
       "5                0      0            373450   8.0500        S    Mr   \n",
       "\n",
       "            Ticket_Int Ticket_Str  \n",
       "PassengerId                        \n",
       "1                21171        A/5  \n",
       "2                17599         PC  \n",
       "3              3101282   STON/O2.  \n",
       "4               113803       Null  \n",
       "5               373450       Null  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def dissect_ticket_column(data, column='Ticket'):\n",
    "    int_component = data[column].apply(lambda x: x.split(' ')[-1])\n",
    "    str_component = get_string_component_from_ticket(new_train_data)\n",
    "    \n",
    "    data['Ticket_Int'] = int_component # creates a new column that stores the integer component\n",
    "    data['Ticket_Str'] = str_component # creates a new column that stores the string component\n",
    "    \n",
    "    return data\n",
    "\n",
    "new_data = dissect_ticket_column(new_train_data)\n",
    "new_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like everything worked out fine! We have successfully split the ticket column into 2 different columns for us to carry out analysis on. It might be useful to conduct a similar analysis on the column, Name.\n",
    "\n",
    "---\n",
    "\n",
    "In-class assignment (You can define your own functions, or use functional programming):\n",
    "1. Split the name column of new_data in to 2 different columns, Last_Name, and First_Name. Hint: You can split on the comma punctuation (', ') - there's a space.  \n",
    "2. After splitting, do another split on the First_Name column on the following string: \". \" (note the space) to obtain the honorific name (Mr, Ms etc.) and the actual Last_Name. Name the columns: Honorific and True_Last_Name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def dissect_name_column(data, column='Name'):\n",
    "    ## Your code here\n",
    "    last_name = data[column].apply(lambda x: x.split(', ')[0]) # Split Name column by ', ' and take element 0\n",
    "    first_name = data[column].apply(lambda x: x.split(', ')[1]) # Split Name column by ', ' and take element 1\n",
    "    \n",
    "    title = first_name.apply(lambda x: x.split('. ')[0]) # Split first_name column by '. ' and take element 0\n",
    "    \n",
    "    data['True_Last_Name'] = last_name\n",
    "    data['Honorific'] = title\n",
    "    return data\n",
    "\n",
    "new_train_data = dissect_name_column(data=new_train_data, column='Name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Capt', 'Col', 'Don', 'Dr', 'Jonkheer', 'Lady', 'Major', 'Master',\n",
       "       'Miss', 'Mlle', 'Mme', 'Mr', 'Mrs', 'Ms', 'Rev', 'Sir',\n",
       "       'the Countess'], dtype=object)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(new_train_data['Honorific'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Title</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Capt</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Col</th>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Don</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dr</th>\n",
       "      <td>0.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jonkheer</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lady</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Major</th>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Master</th>\n",
       "      <td>0.575000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Miss</th>\n",
       "      <td>0.697802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mlle</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mme</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mr</th>\n",
       "      <td>0.156673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mrs</th>\n",
       "      <td>0.792000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ms</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rev</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sir</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the Countess</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Survived\n",
       "Title                 \n",
       "Capt          0.000000\n",
       "Col           0.500000\n",
       "Don           0.000000\n",
       "Dr            0.428571\n",
       "Jonkheer      0.000000\n",
       "Lady          1.000000\n",
       "Major         0.500000\n",
       "Master        0.575000\n",
       "Miss          0.697802\n",
       "Mlle          1.000000\n",
       "Mme           1.000000\n",
       "Mr            0.156673\n",
       "Mrs           0.792000\n",
       "Ms            1.000000\n",
       "Rev           0.000000\n",
       "Sir           1.000000\n",
       "the Countess  1.000000"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_train_data.pivot_table(values='Survived',\n",
    "                           index='Title',\n",
    "                           aggfunc='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going forward, it is useful to check what kinds of tickets (Ticket_Str) there are. To do so, we can use the `np.unique` function to do that (what other functions could we have used?). Let's do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['A./5.', 'A.5.', 'A/4', 'A/4.', 'A/5', 'A/5.', 'A/S', 'A4.', 'C',\n",
       "       'C.A.', 'C.A./SOTON', 'CA', 'CA.', 'F.C.', 'F.C.C.', 'Fa', 'Null',\n",
       "       'P/PP', 'PC', 'PP', 'S.C./A.4.', 'S.C./PARIS', 'S.O./P.P.',\n",
       "       'S.O.C.', 'S.O.P.', 'S.P.', 'S.W./PP', 'SC', 'SC/AH', 'SC/PARIS',\n",
       "       'SC/Paris', 'SCO/W', 'SO/C', 'SOTON/O.Q.', 'SOTON/O2', 'SOTON/OQ',\n",
       "       'STON/O', 'STON/O2.', 'SW/PP', 'W./C.', 'W.E.P.', 'W/C', 'WE/P'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(new_data['Ticket_Str'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that the data is slightly dirty - there appears to be many mistakes (we actually do not know if these are genuine errors or not). One way to verify this is to count the number of times each category appears. If a category only appears once, and is lexically close to another category, then it is very likely to be a recording error.\n",
    "\n",
    "One way to do so is through the use of a dictionary, looping over every value and keeping count of them. Alternatively, we can use the Series method (a Series is basically a column of a DataFrame), `value_counts`. A Series is basically a column of the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Null          665\n",
       "PC             60\n",
       "C.A.           27\n",
       "STON/O         12\n",
       "A/5            10\n",
       "W./C.           9\n",
       "CA.             8\n",
       "SOTON/O.Q.      8\n",
       "SOTON/OQ        7\n",
       "A/5.            7\n",
       "CA              6\n",
       "STON/O2.        6\n",
       "F.C.C.          5\n",
       "C               5\n",
       "SC/PARIS        5\n",
       "S.O.C.          5\n",
       "SC/Paris        4\n",
       "PP              3\n",
       "S.O./P.P.       3\n",
       "SC/AH           3\n",
       "A/4.            3\n",
       "A/4             3\n",
       "SOTON/O2        2\n",
       "WE/P            2\n",
       "S.C./PARIS      2\n",
       "P/PP            2\n",
       "A./5.           2\n",
       "A.5.            2\n",
       "F.C.            1\n",
       "C.A./SOTON      1\n",
       "SCO/W           1\n",
       "S.W./PP         1\n",
       "SW/PP           1\n",
       "SO/C            1\n",
       "S.C./A.4.       1\n",
       "W.E.P.          1\n",
       "A4.             1\n",
       "S.O.P.          1\n",
       "A/S             1\n",
       "Fa              1\n",
       "S.P.            1\n",
       "W/C             1\n",
       "SC              1\n",
       "Name: Ticket_Str, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data['Ticket_Str'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In-class Assignment:\n",
    "\n",
    "Use the dictionary method to generate the same table above. Helper code has been provided below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'A./5.': 2,\n",
       " 'A.5.': 2,\n",
       " 'A/4': 3,\n",
       " 'A/4.': 3,\n",
       " 'A/5': 10,\n",
       " 'A/5.': 7,\n",
       " 'A/S': 1,\n",
       " 'A4.': 1,\n",
       " 'C': 5,\n",
       " 'C.A.': 27,\n",
       " 'C.A./SOTON': 1,\n",
       " 'CA': 6,\n",
       " 'CA.': 8,\n",
       " 'F.C.': 1,\n",
       " 'F.C.C.': 5,\n",
       " 'Fa': 1,\n",
       " 'Null': 665,\n",
       " 'P/PP': 2,\n",
       " 'PC': 60,\n",
       " 'PP': 3,\n",
       " 'S.C./A.4.': 1,\n",
       " 'S.C./PARIS': 2,\n",
       " 'S.O./P.P.': 3,\n",
       " 'S.O.C.': 5,\n",
       " 'S.O.P.': 1,\n",
       " 'S.P.': 1,\n",
       " 'S.W./PP': 1,\n",
       " 'SC': 1,\n",
       " 'SC/AH': 3,\n",
       " 'SC/PARIS': 5,\n",
       " 'SC/Paris': 4,\n",
       " 'SCO/W': 1,\n",
       " 'SO/C': 1,\n",
       " 'SOTON/O.Q.': 8,\n",
       " 'SOTON/O2': 2,\n",
       " 'SOTON/OQ': 7,\n",
       " 'STON/O': 12,\n",
       " 'STON/O2.': 6,\n",
       " 'SW/PP': 1,\n",
       " 'W./C.': 9,\n",
       " 'W.E.P.': 1,\n",
       " 'W/C': 1,\n",
       " 'WE/P': 2}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def unique_values_dict(data, column='Ticket_Str'):\n",
    "    ticket_dict = {}\n",
    "    for ticket_value in new_data[column].values:\n",
    "        # Your code here\n",
    "        if ticket_value in ticket_dict.keys():\n",
    "            ticket_dict[ticket_value] += 1\n",
    "        else:\n",
    "            ticket_dict[ticket_value] = 1\n",
    "    return ticket_dict\n",
    "\n",
    "unique_values_dict(new_data, \"Ticket_Str\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There appears to be quite a number of mistakes, based on our analysis. For example, A./5., A.5., A/5 and A/5. seem to be referring to the same cabin type. Same goes for the types A/4 and A/4 and A/4.. To clean the data, we can write a simple function that removes all kinds of punctuations in the data.\n",
    "\n",
    "Let's search for [\"clean punctuations python\"](https://www.google.com/search?q=clean+punctuations+python&oq=clean+punctuations+python&aqs=chrome..69i57j0l4.2954j0j1&sourceid=chrome&ie=UTF-8) using Google's search engine.\n",
    "\n",
    "Google recommends using the library string, and module punctuation. Let's find out what's in `string.punctuation first'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string\n",
    "\n",
    "string.punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code was taken from [StackOverflow](https://stackoverflow.com/questions/265960/best-way-to-strip-punctuation-from-a-string). Essentially, the code takes a data structure as its input, loops over each element (each element here refers to a string) in the data structure, and removes all punctuation from the string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A5'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def clean_punctuations(data, column):\n",
    "    clean_data = []\n",
    "    for idx, text in enumerate(data[column].values):\n",
    "        interim_string = text\n",
    "        for punctuation in string.punctuation:\n",
    "            if punctuation in text:\n",
    "                interim_string = interim_string.replace(punctuation, '')\n",
    "        # clean_data.append(row.translate(str.maketrans('', '', string.punctuation)))\n",
    "        clean_data.append(interim_string)\n",
    "    return clean_data\n",
    "\n",
    "clean_punctuations(new_data, 'Ticket_Str')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All punctuation marks have been removed from the `Ticket_Str` column! This is a neat trick that you can use on any specific datasets to remove puncutations or any other character that may be uninformative or redundant. Now that we have cleaned up the dataset, let's take a look at the remaining unique values in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "43\n",
      "30\n"
     ]
    }
   ],
   "source": [
    "print(len(np.unique(new_data['Ticket_Str'])))\n",
    "print(len(np.unique(clean_punctuations(new_data, 'Ticket_Str'))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to lower case\n",
    "new_data['New_Ticket_Str'] = clean_punctuations(new_data, 'Ticket_Str')\n",
    "new_data['New_Ticket_Str'].apply(lambda x: x.lower())\n",
    "new_data['New_Ticket_Str'] = new_data['New_Ticket_Str'].apply(lambda x: x.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId\n",
       "1        a5\n",
       "2        pc\n",
       "3    stono2\n",
       "4      null\n",
       "5      null\n",
       "Name: New_Ticket_Str, dtype: object"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data['New_Ticket_Str'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Imputation for Categorical Variables\n",
    "\n",
    "Let's take a look at the missing values in the data once again. Note that we have 2 observations with missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived          0\n",
       "Pclass            0\n",
       "Name              0\n",
       "Sex               0\n",
       "Age               0\n",
       "SibSp             0\n",
       "Parch             0\n",
       "Ticket            0\n",
       "Fare              0\n",
       "Embarked          2\n",
       "Title             0\n",
       "Ticket_Int        0\n",
       "Ticket_Str        0\n",
       "True_Last_Name    0\n",
       "Honorific         0\n",
       "New_Ticket_Str    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(pd.isnull(new_train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Title</th>\n",
       "      <th>Ticket_Int</th>\n",
       "      <th>Ticket_Str</th>\n",
       "      <th>True_Last_Name</th>\n",
       "      <th>Honorific</th>\n",
       "      <th>New_Ticket_Str</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Icard, Miss. Amelie</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>113572</td>\n",
       "      <td>80.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Miss</td>\n",
       "      <td>113572</td>\n",
       "      <td>Null</td>\n",
       "      <td>Icard</td>\n",
       "      <td>Miss</td>\n",
       "      <td>null</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>830</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Stone, Mrs. George Nelson (Martha Evelyn)</td>\n",
       "      <td>female</td>\n",
       "      <td>62.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>113572</td>\n",
       "      <td>80.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>113572</td>\n",
       "      <td>Null</td>\n",
       "      <td>Stone</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>null</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass                                       Name  \\\n",
       "PassengerId                                                                \n",
       "62                  1       1                        Icard, Miss. Amelie   \n",
       "830                 1       1  Stone, Mrs. George Nelson (Martha Evelyn)   \n",
       "\n",
       "                Sex   Age  SibSp  Parch  Ticket  Fare Embarked Title  \\\n",
       "PassengerId                                                            \n",
       "62           female  38.0      0      0  113572  80.0      NaN  Miss   \n",
       "830          female  62.0      0      0  113572  80.0      NaN   Mrs   \n",
       "\n",
       "            Ticket_Int Ticket_Str True_Last_Name Honorific New_Ticket_Str  \n",
       "PassengerId                                                                \n",
       "62              113572       Null          Icard      Miss           null  \n",
       "830             113572       Null          Stone       Mrs           null  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I subset on observations where Embarked column is null\n",
    "new_train_data[pd.isnull(new_train_data['Embarked'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Embarked</th>\n",
       "      <th>C</th>\n",
       "      <th>Q</th>\n",
       "      <th>S</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>104.718529</td>\n",
       "      <td>90.000000</td>\n",
       "      <td>70.364862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25.358335</td>\n",
       "      <td>12.350000</td>\n",
       "      <td>20.327439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.214083</td>\n",
       "      <td>11.183393</td>\n",
       "      <td>14.644083</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Embarked           C          Q          S\n",
       "Pclass                                    \n",
       "1         104.718529  90.000000  70.364862\n",
       "2          25.358335  12.350000  20.327439\n",
       "3          11.214083  11.183393  14.644083"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Suppose I believe that I can embarkation point is related to fare\n",
    "new_train_data.pivot_table(values='Fare',\n",
    "                           index='Pclass',\n",
    "                           columns='Embarked',\n",
    "                           aggfunc='mean')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since 80 is closer to 70.36 than 90, I assign these 2 passengers \"S\"\n",
    "new_train_data['Embarked'].fillna('S', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived          0\n",
       "Pclass            0\n",
       "Name              0\n",
       "Sex               0\n",
       "Age               0\n",
       "SibSp             0\n",
       "Parch             0\n",
       "Ticket            0\n",
       "Fare              0\n",
       "Embarked          0\n",
       "Title             0\n",
       "Ticket_Int        0\n",
       "Ticket_Str        0\n",
       "True_Last_Name    0\n",
       "Honorific         0\n",
       "New_Ticket_Str    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(pd.isnull(new_train_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are faced with such a scenario, one thing we might do is to to input the most common value in the dataset. To do this, we can use the `describe` method on the dataframe once again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Title</th>\n",
       "      <th>Ticket_Int</th>\n",
       "      <th>Ticket_Str</th>\n",
       "      <th>True_Last_Name</th>\n",
       "      <th>Honorific</th>\n",
       "      <th>New_Ticket_Str</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>891</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>681</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>679</td>\n",
       "      <td>43</td>\n",
       "      <td>667</td>\n",
       "      <td>17</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Waelens, Mr. Achille</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1601</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>Mr</td>\n",
       "      <td>2343</td>\n",
       "      <td>Null</td>\n",
       "      <td>Andersson</td>\n",
       "      <td>Mr</td>\n",
       "      <td>null</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>577</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>646</td>\n",
       "      <td>517</td>\n",
       "      <td>7</td>\n",
       "      <td>665</td>\n",
       "      <td>9</td>\n",
       "      <td>517</td>\n",
       "      <td>665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.204208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.002015</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.693429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.910400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>35.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Survived      Pclass                  Name   Sex         Age  \\\n",
       "count   891.000000  891.000000                   891   891  891.000000   \n",
       "unique         NaN         NaN                   891     2         NaN   \n",
       "top            NaN         NaN  Waelens, Mr. Achille  male         NaN   \n",
       "freq           NaN         NaN                     1   577         NaN   \n",
       "mean      0.383838    2.308642                   NaN   NaN   29.699118   \n",
       "std       0.486592    0.836071                   NaN   NaN   13.002015   \n",
       "min       0.000000    1.000000                   NaN   NaN    0.420000   \n",
       "25%       0.000000    2.000000                   NaN   NaN   22.000000   \n",
       "50%       0.000000    3.000000                   NaN   NaN   29.699118   \n",
       "75%       1.000000    3.000000                   NaN   NaN   35.000000   \n",
       "max       1.000000    3.000000                   NaN   NaN   80.000000   \n",
       "\n",
       "             SibSp       Parch Ticket        Fare Embarked Title Ticket_Int  \\\n",
       "count   891.000000  891.000000    891  891.000000      891   891        891   \n",
       "unique         NaN         NaN    681         NaN        3    17        679   \n",
       "top            NaN         NaN   1601         NaN        S    Mr       2343   \n",
       "freq           NaN         NaN      7         NaN      646   517          7   \n",
       "mean      0.523008    0.381594    NaN   32.204208      NaN   NaN        NaN   \n",
       "std       1.102743    0.806057    NaN   49.693429      NaN   NaN        NaN   \n",
       "min       0.000000    0.000000    NaN    0.000000      NaN   NaN        NaN   \n",
       "25%       0.000000    0.000000    NaN    7.910400      NaN   NaN        NaN   \n",
       "50%       0.000000    0.000000    NaN   14.454200      NaN   NaN        NaN   \n",
       "75%       1.000000    0.000000    NaN   31.000000      NaN   NaN        NaN   \n",
       "max       8.000000    6.000000    NaN  512.329200      NaN   NaN        NaN   \n",
       "\n",
       "       Ticket_Str True_Last_Name Honorific New_Ticket_Str  \n",
       "count         891            891       891            891  \n",
       "unique         43            667        17             29  \n",
       "top          Null      Andersson        Mr           null  \n",
       "freq          665              9       517            665  \n",
       "mean          NaN            NaN       NaN            NaN  \n",
       "std           NaN            NaN       NaN            NaN  \n",
       "min           NaN            NaN       NaN            NaN  \n",
       "25%           NaN            NaN       NaN            NaN  \n",
       "50%           NaN            NaN       NaN            NaN  \n",
       "75%           NaN            NaN       NaN            NaN  \n",
       "max           NaN            NaN       NaN            NaN  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the Embarked column, there are 3 unique values, with \"S\" as the most common value, appearing 644 out of 889 times.\n",
    "\n",
    "---\n",
    "\n",
    "In-class assignment:\n",
    "\n",
    "1. Define a function that takes data and column as its input, and returns the most common value. You can assume that the column provided is a categorical or string column.\n",
    "2. Define another function that fills the NA value with the most common value (refer to first function), and returns the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'S'"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def most_common_category(data, col):\n",
    "    return data[col].value_counts().index[0]\n",
    "    \n",
    "    \n",
    "# The following code is equivalent to running train_data[\"Embarked\"].value_counts().index[0]\n",
    "most_common_category(train_data, \"Embarked\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Survived          0\n",
       "Pclass            0\n",
       "Name              0\n",
       "Sex               0\n",
       "Age               0\n",
       "SibSp             0\n",
       "Parch             0\n",
       "Ticket            0\n",
       "Fare              0\n",
       "Embarked          0\n",
       "Title             0\n",
       "Ticket_Int        0\n",
       "Ticket_Str        0\n",
       "True_Last_Name    0\n",
       "Honorific         0\n",
       "New_Ticket_Str    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(pd.isnull(new_train_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Correlation Analysis\n",
    "\n",
    "Finally, we can conduct some meaningful analysis. One question we might ask is what factors are highly correlated with the Survival factor? \n",
    "\n",
    "To answer this question, we can check the correlation across different features, using the `corr` method on `new_data`. It returns the correlation coefficient between any 2 features in the dataframe. Note that the correlation has to lie between -1 and 1, and the larger the magnitude of the coefficient, the more correlated the 2 features are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd # pd will refer to the pandas library\n",
    "import numpy as np # np will refer to the numpy library\n",
    "\n",
    "# We are using the original version data\n",
    "train_data = pd.read_csv(\"/Users/ngbolin/Downloads/PythonProgrammingWorkshop-master/data/titanic/train.csv\")\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId      0\n",
       "Survived         0\n",
       "Pclass           0\n",
       "Name             0\n",
       "Sex              0\n",
       "Age            177\n",
       "SibSp            0\n",
       "Parch            0\n",
       "Ticket           0\n",
       "Fare             0\n",
       "Cabin          687\n",
       "Embarked         2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This code returns all the missing values in the dataset\n",
    "np.sum(pd.isnull(train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.005007</td>\n",
       "      <td>-0.035144</td>\n",
       "      <td>0.036847</td>\n",
       "      <td>-0.057527</td>\n",
       "      <td>-0.001652</td>\n",
       "      <td>0.012658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <td>-0.005007</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.338481</td>\n",
       "      <td>-0.077221</td>\n",
       "      <td>-0.035322</td>\n",
       "      <td>0.081629</td>\n",
       "      <td>0.257307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <td>-0.035144</td>\n",
       "      <td>-0.338481</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.369226</td>\n",
       "      <td>0.083081</td>\n",
       "      <td>0.018443</td>\n",
       "      <td>-0.549500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>0.036847</td>\n",
       "      <td>-0.077221</td>\n",
       "      <td>-0.369226</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.308247</td>\n",
       "      <td>-0.189119</td>\n",
       "      <td>0.096067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SibSp</th>\n",
       "      <td>-0.057527</td>\n",
       "      <td>-0.035322</td>\n",
       "      <td>0.083081</td>\n",
       "      <td>-0.308247</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.414838</td>\n",
       "      <td>0.159651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Parch</th>\n",
       "      <td>-0.001652</td>\n",
       "      <td>0.081629</td>\n",
       "      <td>0.018443</td>\n",
       "      <td>-0.189119</td>\n",
       "      <td>0.414838</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.216225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fare</th>\n",
       "      <td>0.012658</td>\n",
       "      <td>0.257307</td>\n",
       "      <td>-0.549500</td>\n",
       "      <td>0.096067</td>\n",
       "      <td>0.159651</td>\n",
       "      <td>0.216225</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             PassengerId  Survived    Pclass       Age     SibSp     Parch  \\\n",
       "PassengerId     1.000000 -0.005007 -0.035144  0.036847 -0.057527 -0.001652   \n",
       "Survived       -0.005007  1.000000 -0.338481 -0.077221 -0.035322  0.081629   \n",
       "Pclass         -0.035144 -0.338481  1.000000 -0.369226  0.083081  0.018443   \n",
       "Age             0.036847 -0.077221 -0.369226  1.000000 -0.308247 -0.189119   \n",
       "SibSp          -0.057527 -0.035322  0.083081 -0.308247  1.000000  0.414838   \n",
       "Parch          -0.001652  0.081629  0.018443 -0.189119  0.414838  1.000000   \n",
       "Fare            0.012658  0.257307 -0.549500  0.096067  0.159651  0.216225   \n",
       "\n",
       "                 Fare  \n",
       "PassengerId  0.012658  \n",
       "Survived     0.257307  \n",
       "Pclass      -0.549500  \n",
       "Age          0.096067  \n",
       "SibSp        0.159651  \n",
       "Parch        0.216225  \n",
       "Fare         1.000000  "
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the correlation matrix, it appears that the features, Fare and Passenger Class are highly correlated with the Survival Probability of an individual. However, we also note that the features, Sex and Embarked do not show up in this matrix. This is due to the fact that they are categorical or ordinal factors. However, we can still use the `pivot_table` method to find out the likelihood of a person surviving given a particular gender.\n",
    "\n",
    "Below, we use the method, `pivot_table` to illustrate the relationship between survival likelihood against gender, passenger class and embarkation point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "train_data[\"Embarked\"] == \"S\"\n",
    "\n",
    "print(False == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>EmbarkedS</th>\n",
       "      <th>Male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex   Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                             Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                           Allen, Mr. William Henry    male  35.0      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  EmbarkedS   Male  \n",
       "0      0         A/5 21171   7.2500   NaN        S       True   True  \n",
       "1      0          PC 17599  71.2833   C85        C      False  False  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S       True  False  \n",
       "3      0            113803  53.1000  C123        S       True  False  \n",
       "4      0            373450   8.0500   NaN        S       True   True  "
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a column that verifies/checks whether the person embarked at S\n",
    "train_data[\"EmbarkedS\"] = train_data[\"Embarked\"] == \"S\"\n",
    "train_data[\"Male\"] = train_data[\"Sex\"] == \"male\"\n",
    "\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "      <th>EmbarkedS</th>\n",
       "      <th>Male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.005007</td>\n",
       "      <td>-0.035144</td>\n",
       "      <td>0.036847</td>\n",
       "      <td>-0.057527</td>\n",
       "      <td>-0.001652</td>\n",
       "      <td>0.012658</td>\n",
       "      <td>0.022148</td>\n",
       "      <td>0.042939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <td>-0.005007</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.338481</td>\n",
       "      <td>-0.077221</td>\n",
       "      <td>-0.035322</td>\n",
       "      <td>0.081629</td>\n",
       "      <td>0.257307</td>\n",
       "      <td>-0.155660</td>\n",
       "      <td>-0.543351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pclass</th>\n",
       "      <td>-0.035144</td>\n",
       "      <td>-0.338481</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.369226</td>\n",
       "      <td>0.083081</td>\n",
       "      <td>0.018443</td>\n",
       "      <td>-0.549500</td>\n",
       "      <td>0.081720</td>\n",
       "      <td>0.131900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>0.036847</td>\n",
       "      <td>-0.077221</td>\n",
       "      <td>-0.369226</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.308247</td>\n",
       "      <td>-0.189119</td>\n",
       "      <td>0.096067</td>\n",
       "      <td>-0.032523</td>\n",
       "      <td>0.093254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SibSp</th>\n",
       "      <td>-0.057527</td>\n",
       "      <td>-0.035322</td>\n",
       "      <td>0.083081</td>\n",
       "      <td>-0.308247</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.414838</td>\n",
       "      <td>0.159651</td>\n",
       "      <td>0.070941</td>\n",
       "      <td>-0.114631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Parch</th>\n",
       "      <td>-0.001652</td>\n",
       "      <td>0.081629</td>\n",
       "      <td>0.018443</td>\n",
       "      <td>-0.189119</td>\n",
       "      <td>0.414838</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.216225</td>\n",
       "      <td>0.063036</td>\n",
       "      <td>-0.245489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fare</th>\n",
       "      <td>0.012658</td>\n",
       "      <td>0.257307</td>\n",
       "      <td>-0.549500</td>\n",
       "      <td>0.096067</td>\n",
       "      <td>0.159651</td>\n",
       "      <td>0.216225</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.166603</td>\n",
       "      <td>-0.182333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EmbarkedS</th>\n",
       "      <td>0.022148</td>\n",
       "      <td>-0.155660</td>\n",
       "      <td>0.081720</td>\n",
       "      <td>-0.032523</td>\n",
       "      <td>0.070941</td>\n",
       "      <td>0.063036</td>\n",
       "      <td>-0.166603</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.125722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Male</th>\n",
       "      <td>0.042939</td>\n",
       "      <td>-0.543351</td>\n",
       "      <td>0.131900</td>\n",
       "      <td>0.093254</td>\n",
       "      <td>-0.114631</td>\n",
       "      <td>-0.245489</td>\n",
       "      <td>-0.182333</td>\n",
       "      <td>0.125722</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             PassengerId  Survived    Pclass       Age     SibSp     Parch  \\\n",
       "PassengerId     1.000000 -0.005007 -0.035144  0.036847 -0.057527 -0.001652   \n",
       "Survived       -0.005007  1.000000 -0.338481 -0.077221 -0.035322  0.081629   \n",
       "Pclass         -0.035144 -0.338481  1.000000 -0.369226  0.083081  0.018443   \n",
       "Age             0.036847 -0.077221 -0.369226  1.000000 -0.308247 -0.189119   \n",
       "SibSp          -0.057527 -0.035322  0.083081 -0.308247  1.000000  0.414838   \n",
       "Parch          -0.001652  0.081629  0.018443 -0.189119  0.414838  1.000000   \n",
       "Fare            0.012658  0.257307 -0.549500  0.096067  0.159651  0.216225   \n",
       "EmbarkedS       0.022148 -0.155660  0.081720 -0.032523  0.070941  0.063036   \n",
       "Male            0.042939 -0.543351  0.131900  0.093254 -0.114631 -0.245489   \n",
       "\n",
       "                 Fare  EmbarkedS      Male  \n",
       "PassengerId  0.012658   0.022148  0.042939  \n",
       "Survived     0.257307  -0.155660 -0.543351  \n",
       "Pclass      -0.549500   0.081720  0.131900  \n",
       "Age          0.096067  -0.032523  0.093254  \n",
       "SibSp        0.159651   0.070941 -0.114631  \n",
       "Parch        0.216225   0.063036 -0.245489  \n",
       "Fare         1.000000  -0.166603 -0.182333  \n",
       "EmbarkedS   -0.166603   1.000000  0.125722  \n",
       "Male        -0.182333   0.125722  1.000000  "
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it turns out, Females have a very high rate of survival. This might be due to many reasons (they might be younger, or they may have paid a higher fare for their tickets etc.). Let's explore this phenomenon in larger detail. We can use the `pivot_table` method to do so.\n",
    "\n",
    "---\n",
    "\n",
    "In-class assignment:\n",
    "1. Test our hypothesis that females are younger than males on the Titanic on average.\n",
    "2. Test our hypothesis that females pay more than males on average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>44.479818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>25.523893</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Fare\n",
       "Sex              \n",
       "female  44.479818\n",
       "male    25.523893"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Your code here\n",
    "new_train_data.pivot_table(values='Fare', index='Sex', aggfunc='mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Groupby\n",
    "One other thing that we can do with DataFrames is the `groupby` method. From the [groupby article on pandas](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html), a groupby operation involves some combination of splitting the object, applying a function, and combining the results. This can be used to group large amounts of data and compute operations on these groups.\n",
    "\n",
    "---\n",
    "\n",
    "As motivation, let's take a closer look at the \"Ticket\" column of the DataFrame. There are 2 ways to subset a column: `new_data['Ticket']` and `new_data.Ticket`. We prefer the first method since it allows for \"spaces\" in the variable name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>EmbarkedS</th>\n",
       "      <th>Male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "      <td>714.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>891</td>\n",
       "      <td>891.000000</td>\n",
       "      <td>204</td>\n",
       "      <td>889</td>\n",
       "      <td>891</td>\n",
       "      <td>891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>891</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>681</td>\n",
       "      <td>NaN</td>\n",
       "      <td>147</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gee, Mr. Arthur H</td>\n",
       "      <td>male</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>CA. 2343</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C23 C25 C27</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>577</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>644</td>\n",
       "      <td>644</td>\n",
       "      <td>577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.383838</td>\n",
       "      <td>2.308642</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29.699118</td>\n",
       "      <td>0.523008</td>\n",
       "      <td>0.381594</td>\n",
       "      <td>NaN</td>\n",
       "      <td>32.204208</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>257.353842</td>\n",
       "      <td>0.486592</td>\n",
       "      <td>0.836071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.526497</td>\n",
       "      <td>1.102743</td>\n",
       "      <td>0.806057</td>\n",
       "      <td>NaN</td>\n",
       "      <td>49.693429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.420000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>223.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.125000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.910400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>446.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.454200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>668.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>891.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>512.329200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        PassengerId    Survived      Pclass               Name   Sex  \\\n",
       "count    891.000000  891.000000  891.000000                891   891   \n",
       "unique          NaN         NaN         NaN                891     2   \n",
       "top             NaN         NaN         NaN  Gee, Mr. Arthur H  male   \n",
       "freq            NaN         NaN         NaN                  1   577   \n",
       "mean     446.000000    0.383838    2.308642                NaN   NaN   \n",
       "std      257.353842    0.486592    0.836071                NaN   NaN   \n",
       "min        1.000000    0.000000    1.000000                NaN   NaN   \n",
       "25%      223.500000    0.000000    2.000000                NaN   NaN   \n",
       "50%      446.000000    0.000000    3.000000                NaN   NaN   \n",
       "75%      668.500000    1.000000    3.000000                NaN   NaN   \n",
       "max      891.000000    1.000000    3.000000                NaN   NaN   \n",
       "\n",
       "               Age       SibSp       Parch    Ticket        Fare        Cabin  \\\n",
       "count   714.000000  891.000000  891.000000       891  891.000000          204   \n",
       "unique         NaN         NaN         NaN       681         NaN          147   \n",
       "top            NaN         NaN         NaN  CA. 2343         NaN  C23 C25 C27   \n",
       "freq           NaN         NaN         NaN         7         NaN            4   \n",
       "mean     29.699118    0.523008    0.381594       NaN   32.204208          NaN   \n",
       "std      14.526497    1.102743    0.806057       NaN   49.693429          NaN   \n",
       "min       0.420000    0.000000    0.000000       NaN    0.000000          NaN   \n",
       "25%      20.125000    0.000000    0.000000       NaN    7.910400          NaN   \n",
       "50%      28.000000    0.000000    0.000000       NaN   14.454200          NaN   \n",
       "75%      38.000000    1.000000    0.000000       NaN   31.000000          NaN   \n",
       "max      80.000000    8.000000    6.000000       NaN  512.329200          NaN   \n",
       "\n",
       "       Embarked EmbarkedS  Male  \n",
       "count       889       891   891  \n",
       "unique        3         2     2  \n",
       "top           S      True  True  \n",
       "freq        644       644   577  \n",
       "mean        NaN       NaN   NaN  \n",
       "std         NaN       NaN   NaN  \n",
       "min         NaN       NaN   NaN  \n",
       "25%         NaN       NaN   NaN  \n",
       "50%         NaN       NaN   NaN  \n",
       "75%         NaN       NaN   NaN  \n",
       "max         NaN       NaN   NaN  "
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.describe(include='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "681"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(np.unique(train_data['Ticket']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "891"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_data['Ticket'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>EmbarkedS</th>\n",
       "      <th>Male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>259</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Ward, Miss. Anna</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17755</td>\n",
       "      <td>512.3292</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>679</th>\n",
       "      <td>680</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cardeza, Mr. Thomas Drake Martinez</td>\n",
       "      <td>male</td>\n",
       "      <td>36.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>PC 17755</td>\n",
       "      <td>512.3292</td>\n",
       "      <td>B51 B53 B55</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>737</th>\n",
       "      <td>738</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Lesurer, Mr. Gustave J</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17755</td>\n",
       "      <td>512.3292</td>\n",
       "      <td>B101</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass                                Name  \\\n",
       "258          259         1       1                    Ward, Miss. Anna   \n",
       "679          680         1       1  Cardeza, Mr. Thomas Drake Martinez   \n",
       "737          738         1       1              Lesurer, Mr. Gustave J   \n",
       "\n",
       "        Sex   Age  SibSp  Parch    Ticket      Fare        Cabin Embarked  \\\n",
       "258  female  35.0      0      0  PC 17755  512.3292          NaN        C   \n",
       "679    male  36.0      0      1  PC 17755  512.3292  B51 B53 B55        C   \n",
       "737    male  35.0      0      0  PC 17755  512.3292         B101        C   \n",
       "\n",
       "     EmbarkedS   Male  \n",
       "258      False  False  \n",
       "679      False   True  \n",
       "737      False   True  "
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[train_data[\"Fare\"] > 300]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "53 out of 891 observations pay $100 for a ticket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that some of the passengers shared tickets: while there were only 681 unique tickets, there were a total of 891 passengers. This begs the question: who are the passengers who shared these tickets, and how many tickets were shared? \n",
    "\n",
    "As it turns out, we can answer this question using the DataFrame method, `new_data.groupby`. In our case, we can create a new Series that returns 1 when the ticket is shared by only 1 passenger, and 2 if it is shared by 2 passengers etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x7f9e181461c0>"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We are \"splitting\" on the Ticket column of the train_data\n",
    "groupby_ticket = train_data.groupby('Ticket')\n",
    "groupby_ticket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      1\n",
       "1      1\n",
       "2      1\n",
       "3      2\n",
       "4      1\n",
       "      ..\n",
       "886    1\n",
       "887    1\n",
       "888    2\n",
       "889    1\n",
       "890    1\n",
       "Name: Pclass, Length: 891, dtype: int64"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We are now \"applying\" an aggregation function to the groupby_ticket from earlier\n",
    "# I am interested in the # of times each ticket appears in the column, so I use \"count\"\n",
    "count_ticket = groupby_ticket.transform('count')\n",
    "count_ticket[\"Pclass\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Ticket Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Ticket  Fare  Ticket Count\n",
       "0      1    10             3\n",
       "1      2    20             2\n",
       "2      3    15             1\n",
       "3      1    10             3\n",
       "4      2    20             2\n",
       "5      1    10             3"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'Ticket': ['1', '2', '3', '1', '2', '1'],\n",
    "                   'Fare': [10, 20, 15, 10, 20, 10]})\n",
    "\n",
    "groupby_ticket = df.groupby(\"Ticket\")\n",
    "count_ticket   = groupby_ticket.transform(\"count\")\n",
    "df[\"Ticket Count\"]  = count_ticket\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ticket</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Fare\n",
       "Ticket      \n",
       "1         10\n",
       "2         20\n",
       "3         15"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot_table(values=\"Fare\",\n",
    "               index = \"Ticket\",\n",
    "               aggfunc=\"mean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a working understanding of the DataFrame method, `groupby`, use it to solve the following questions:\n",
    "\n",
    "1. Several passengers are sharing a ticket with one another. Using the 'groupby' method, construct a column called \"AdjustedFare\" where the \"Fare\" column is adjusted for the number of people sharing the same ticket.\n",
    "\n",
    "Hint: You can do vector/column multiplication or division simply by using the following:\n",
    "data[column1]/data[column2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 1\n",
    "## Your code here\n",
    "train_data[\"AdjustedFare\"] = train_data[\"Fare\"]/count_ticket[\"Pclass\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>EmbarkedS</th>\n",
       "      <th>Male</th>\n",
       "      <th>AdjustedFare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>26.5500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>887</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Montvila, Rev. Juozas</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211536</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>888</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Graham, Miss. Margaret Edith</td>\n",
       "      <td>female</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>112053</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>B42</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>30.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>888</th>\n",
       "      <td>889</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Johnston, Miss. Catherine Helen \"Carrie\"</td>\n",
       "      <td>female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>W./C. 6607</td>\n",
       "      <td>23.4500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>11.7250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>889</th>\n",
       "      <td>890</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Behr, Mr. Karl Howell</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>111369</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>C148</td>\n",
       "      <td>C</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>30.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>891</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Dooley, Mr. Patrick</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>370376</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Q</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>7.7500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>891 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     PassengerId  Survived  Pclass  \\\n",
       "0              1         0       3   \n",
       "1              2         1       1   \n",
       "2              3         1       3   \n",
       "3              4         1       1   \n",
       "4              5         0       3   \n",
       "..           ...       ...     ...   \n",
       "886          887         0       2   \n",
       "887          888         1       1   \n",
       "888          889         0       3   \n",
       "889          890         1       1   \n",
       "890          891         0       3   \n",
       "\n",
       "                                                  Name     Sex   Age  SibSp  \\\n",
       "0                              Braund, Mr. Owen Harris    male  22.0      1   \n",
       "1    Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n",
       "2                               Heikkinen, Miss. Laina  female  26.0      0   \n",
       "3         Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n",
       "4                             Allen, Mr. William Henry    male  35.0      0   \n",
       "..                                                 ...     ...   ...    ...   \n",
       "886                              Montvila, Rev. Juozas    male  27.0      0   \n",
       "887                       Graham, Miss. Margaret Edith  female  19.0      0   \n",
       "888           Johnston, Miss. Catherine Helen \"Carrie\"  female   NaN      1   \n",
       "889                              Behr, Mr. Karl Howell    male  26.0      0   \n",
       "890                                Dooley, Mr. Patrick    male  32.0      0   \n",
       "\n",
       "     Parch            Ticket     Fare Cabin Embarked  EmbarkedS   Male  \\\n",
       "0        0         A/5 21171   7.2500   NaN        S       True   True   \n",
       "1        0          PC 17599  71.2833   C85        C      False  False   \n",
       "2        0  STON/O2. 3101282   7.9250   NaN        S       True  False   \n",
       "3        0            113803  53.1000  C123        S       True  False   \n",
       "4        0            373450   8.0500   NaN        S       True   True   \n",
       "..     ...               ...      ...   ...      ...        ...    ...   \n",
       "886      0            211536  13.0000   NaN        S       True   True   \n",
       "887      0            112053  30.0000   B42        S       True  False   \n",
       "888      2        W./C. 6607  23.4500   NaN        S       True  False   \n",
       "889      0            111369  30.0000  C148        C      False   True   \n",
       "890      0            370376   7.7500   NaN        Q      False   True   \n",
       "\n",
       "     AdjustedFare  \n",
       "0          7.2500  \n",
       "1         71.2833  \n",
       "2          7.9250  \n",
       "3         26.5500  \n",
       "4          8.0500  \n",
       "..            ...  \n",
       "886       13.0000  \n",
       "887       30.0000  \n",
       "888       11.7250  \n",
       "889       30.0000  \n",
       "890        7.7500  \n",
       "\n",
       "[891 rows x 15 columns]"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can save the dataset using the method `.to_csv()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "new_data.to_csv('data/newdata.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Embarked</th>\n",
       "      <th>Title</th>\n",
       "      <th>Ticket_Int</th>\n",
       "      <th>Ticket_Str</th>\n",
       "      <th>True_Last_Name</th>\n",
       "      <th>Honorific</th>\n",
       "      <th>New_Ticket_Str</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PassengerId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Mr</td>\n",
       "      <td>21171</td>\n",
       "      <td>A/5</td>\n",
       "      <td>Braund</td>\n",
       "      <td>Mr</td>\n",
       "      <td>a5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>17599</td>\n",
       "      <td>PC</td>\n",
       "      <td>Cumings</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>pc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Miss</td>\n",
       "      <td>3101282</td>\n",
       "      <td>STON/O2.</td>\n",
       "      <td>Heikkinen</td>\n",
       "      <td>Miss</td>\n",
       "      <td>stono2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>113803</td>\n",
       "      <td>Null</td>\n",
       "      <td>Futrelle</td>\n",
       "      <td>Mrs</td>\n",
       "      <td>null</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Mr</td>\n",
       "      <td>373450</td>\n",
       "      <td>Null</td>\n",
       "      <td>Allen</td>\n",
       "      <td>Mr</td>\n",
       "      <td>null</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Survived  Pclass  \\\n",
       "PassengerId                     \n",
       "1                   0       3   \n",
       "2                   1       1   \n",
       "3                   1       3   \n",
       "4                   1       1   \n",
       "5                   0       3   \n",
       "\n",
       "                                                          Name     Sex   Age  \\\n",
       "PassengerId                                                                    \n",
       "1                                      Braund, Mr. Owen Harris    male  22.0   \n",
       "2            Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n",
       "3                                       Heikkinen, Miss. Laina  female  26.0   \n",
       "4                 Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n",
       "5                                     Allen, Mr. William Henry    male  35.0   \n",
       "\n",
       "             SibSp  Parch            Ticket     Fare Embarked Title  \\\n",
       "PassengerId                                                           \n",
       "1                1      0         A/5 21171   7.2500        S    Mr   \n",
       "2                1      0          PC 17599  71.2833        C   Mrs   \n",
       "3                0      0  STON/O2. 3101282   7.9250        S  Miss   \n",
       "4                1      0            113803  53.1000        S   Mrs   \n",
       "5                0      0            373450   8.0500        S    Mr   \n",
       "\n",
       "            Ticket_Int Ticket_Str True_Last_Name Honorific New_Ticket_Str  \n",
       "PassengerId                                                                \n",
       "1                21171        A/5         Braund        Mr             a5  \n",
       "2                17599         PC        Cumings       Mrs             pc  \n",
       "3              3101282   STON/O2.      Heikkinen      Miss         stono2  \n",
       "4               113803       Null       Futrelle       Mrs           null  \n",
       "5               373450       Null          Allen        Mr           null  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Manipulation with MovieLens\n",
    "\n",
    "The following example is taken from the book, \"Python for Data Analysis\" written by Wes McKinney, the inventor of Pandas. If you're interested in buying the book, you can find it [here](https://www.bookdepository.com/Python-for-Data-Analysis-2e-Wes-McKinney/9781491957660?ref=grid-view&qid=1564376975506&sr=1-1).\n",
    "\n",
    "If you've taken a closer look at the MovieLens dataset, you would have noticed that the extension for the dataset is \".dat\". However, it turns out that we can still use the `pd.read_csv` function for this. Alternatively, we can use the `pd.read_table` function for this as well. pandas has the ability to read different kinds of data files, and \".dat\" is one of them. \n",
    "\n",
    "The MovieLens dataset has 3 different tables: \n",
    "\n",
    "1. \"unames\", a table that contains the usernames, gender, age and occupation etc. of users\n",
    "2. \"rnames\", a table that contains the ratings, movie ids and timestamp etc.\n",
    "3. \"mnames\", a table that contains the movie titles, movie ids and genres of the movies\n",
    "\n",
    "We can add an argument, `names` to the functions `pd.read_table` or `pd.read_csv` to tell Python which table we are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import user data table\n",
    "unames = ['user_id', 'gender', 'age', 'occupation', 'zip']\n",
    "users = pd.read_csv('data/movielens/ml-1m/users.dat', engine='python', sep='::', header=None, names=unames)\n",
    "\n",
    "# Import ratings table\n",
    "rnames = ['user_id', 'movie_id', 'rating', 'timestamp']\n",
    "ratings = pd.read_csv('data/movielens/ml-1m/ratings.dat', engine='python', sep='::', header=None, names=rnames)\n",
    "\n",
    "# Import mnames table\n",
    "mnames = ['movie_id', 'title', 'genres']\n",
    "movies = pd.read_csv('data/movielens/ml-1m/movies.dat', engine='python', sep='::', header=None, names=mnames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is almost canon to take a look at the headers of the dataframes you have read after you have imported them into Python, so that's what we'll do in the next few code blocks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id gender  age  occupation    zip\n",
       "0        1      F    1          10  48067\n",
       "1        2      M   56          16  70072\n",
       "2        3      M   25          15  55117\n",
       "3        4      M   45           7  02460\n",
       "4        5      M   25          20  55455"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp\n",
       "0        1      1193       5  978300760\n",
       "1        1       661       3  978302109\n",
       "2        1       914       3  978301968\n",
       "3        1      3408       4  978300275\n",
       "4        1      2355       5  978824291"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                               title                        genres\n",
       "0         1                    Toy Story (1995)   Animation|Children's|Comedy\n",
       "1         2                      Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "2         3             Grumpier Old Men (1995)                Comedy|Romance\n",
       "3         4            Waiting to Exhale (1995)                  Comedy|Drama\n",
       "4         5  Father of the Bride Part II (1995)                        Comedy"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A keen observer would have noted that there are some dataframes that we can combine using columns that we can merge on. For example, one can merge the users and rating dataframes using the columns, user_id. Alternatively, one can also merge the ratings and movies dataset using the column, movie_id. \n",
    "\n",
    "As it turns out, we can use the function, `pd.merge` from pandas to merge 2 different datasets. Below, we merge the users and ratings dataframes on a common column, 'user_id'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000204</th>\n",
       "      <td>6040</td>\n",
       "      <td>1091</td>\n",
       "      <td>1</td>\n",
       "      <td>956716541</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>11106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000205</th>\n",
       "      <td>6040</td>\n",
       "      <td>1094</td>\n",
       "      <td>5</td>\n",
       "      <td>956704887</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>11106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000206</th>\n",
       "      <td>6040</td>\n",
       "      <td>562</td>\n",
       "      <td>5</td>\n",
       "      <td>956704746</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>11106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000207</th>\n",
       "      <td>6040</td>\n",
       "      <td>1096</td>\n",
       "      <td>4</td>\n",
       "      <td>956715648</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>11106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000208</th>\n",
       "      <td>6040</td>\n",
       "      <td>1097</td>\n",
       "      <td>4</td>\n",
       "      <td>956715569</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>11106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000209 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         user_id  movie_id  rating  timestamp gender  age  occupation    zip\n",
       "0              1      1193       5  978300760      F    1          10  48067\n",
       "1              1       661       3  978302109      F    1          10  48067\n",
       "2              1       914       3  978301968      F    1          10  48067\n",
       "3              1      3408       4  978300275      F    1          10  48067\n",
       "4              1      2355       5  978824291      F    1          10  48067\n",
       "...          ...       ...     ...        ...    ...  ...         ...    ...\n",
       "1000204     6040      1091       1  956716541      M   25           6  11106\n",
       "1000205     6040      1094       5  956704887      M   25           6  11106\n",
       "1000206     6040       562       5  956704746      M   25           6  11106\n",
       "1000207     6040      1096       4  956715648      M   25           6  11106\n",
       "1000208     6040      1097       4  956715569      M   25           6  11106\n",
       "\n",
       "[1000209 rows x 8 columns]"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dat = pd.merge(ratings, users, left_on=\"user_id\", right_on=\"user_id\")\n",
    "dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the merged dataframe. Following the \"general\" principle of data analysis, we take a look at the first 5 rows of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id gender  age  occupation    zip  movie_id  rating  timestamp\n",
       "0        1      F    1          10  48067      1193       5  978300760\n",
       "1        1      F    1          10  48067       661       3  978302109\n",
       "2        1      F    1          10  48067       914       3  978301968\n",
       "3        1      F    1          10  48067      3408       4  978300275\n",
       "4        1      F    1          10  48067      2355       5  978824291"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dat.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now merge this dataframe with the movies dataframe using the column, 'movie_id'. Let's do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "dat = pd.merge(dat, movies, on='movie_id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978298413</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978220179</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>12</td>\n",
       "      <td>32793</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978199279</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>7</td>\n",
       "      <td>22903</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978158471</td>\n",
       "      <td>M</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>95350</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp gender  age  occupation    zip  \\\n",
       "0        1      1193       5  978300760      F    1          10  48067   \n",
       "1        2      1193       5  978298413      M   56          16  70072   \n",
       "2       12      1193       4  978220179      M   25          12  32793   \n",
       "3       15      1193       4  978199279      M   25           7  22903   \n",
       "4       17      1193       5  978158471      M   50           1  95350   \n",
       "\n",
       "                                    title genres  \n",
       "0  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "1  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "2  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "3  One Flew Over the Cuckoo's Nest (1975)  Drama  \n",
       "4  One Flew Over the Cuckoo's Nest (1975)  Drama  "
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000209, 10)"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dat.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We managed to combine the 3 dataframes into 1, using the function `pd.merge`. This function takes as argument 2 dataframes to combine together, as well as some other arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "?pd.merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some following questions we can ask:\n",
    "\n",
    "1. What are the most popular titles?\n",
    "2. Which user rated the most movies? \n",
    "3. Which users are likely to give a very high rating on the movies that they rate?\n",
    "4. What areas are users most likely to stay at?\n",
    "5. Do older users tend to watch different genres of movies from the younger users?\n",
    "\n",
    "It turns out that we can answer these questions fairly easily with the use of pandas and Numpy. Below, we answer each question in a sequential manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For question 3, we want to have a dataframe with 2 columns. 1st column - user_id, and the second column will be average ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "title\n",
       "American Beauty (1999)                                   3428\n",
       "Star Wars: Episode IV - A New Hope (1977)                2991\n",
       "Star Wars: Episode V - The Empire Strikes Back (1980)    2990\n",
       "Star Wars: Episode VI - Return of the Jedi (1983)        2883\n",
       "Jurassic Park (1993)                                     2672\n",
       "Saving Private Ryan (1998)                               2653\n",
       "Terminator 2: Judgment Day (1991)                        2649\n",
       "Matrix, The (1999)                                       2590\n",
       "Back to the Future (1985)                                2583\n",
       "Silence of the Lambs, The (1991)                         2578\n",
       "dtype: int64"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Group dataset using titles and count using .size() - size method is similar to aggfunc = \"count\"\n",
    "# This returns a data column\n",
    "title_count = dat.groupby('title').size()\n",
    "\n",
    "title_count.sort_values(ascending=False)[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "title\n",
       "American Beauty (1999)                                   3428\n",
       "Star Wars: Episode IV - A New Hope (1977)                2991\n",
       "Star Wars: Episode V - The Empire Strikes Back (1980)    2990\n",
       "Star Wars: Episode VI - Return of the Jedi (1983)        2883\n",
       "Jurassic Park (1993)                                     2672\n",
       "dtype: int64"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Popular Titles\n",
    "def top_n_titles(n):\n",
    "    '''\n",
    "    This function takes an integer as the argument, and returns the top n titles.\n",
    "    '''\n",
    "    # .size() returns the number of times a title appears in the dataset\n",
    "    titles = dat.groupby('title').size()\n",
    "    return titles.sort_values(ascending=False)[:n] # we sort the number of counts in a descending order\n",
    "\n",
    "top_n_titles(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that most of the users are fans of the movies, American Beauty, Star Wars and Jurassic Park. Now, which users rated the most movies? We can define a function that returns these users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id\n",
       "4169    2314\n",
       "1680    1850\n",
       "4277    1743\n",
       "1941    1595\n",
       "1181    1521\n",
       "889     1518\n",
       "3618    1344\n",
       "2063    1323\n",
       "1150    1302\n",
       "1015    1286\n",
       "dtype: int64"
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def most_active_users(n):\n",
    "    active_users = dat.groupby('user_id').size() # we count the number of times a user appears in the dataset\n",
    "    return active_users.sort_values(ascending=False)[:n] \n",
    "\n",
    "most_active_users(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One may be interested in users who are very critical or lax in their reviews of movies. If a user has a high mean rating, then it could be that his reviews are less informative than someone who is more critical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id\n",
       "283     4.962963\n",
       "2339    4.956522\n",
       "3324    4.904762\n",
       "3902    4.890909\n",
       "446     4.843137\n",
       "          ...   \n",
       "5850    1.844828\n",
       "4539    1.815126\n",
       "2744    1.304348\n",
       "4486    1.058824\n",
       "3598    1.015385\n",
       "Name: rating, Length: 6040, dtype: float64"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pivot Table here\n",
    "pivot_rating = dat.pivot_table(values=\"rating\",\n",
    "                               index=\"user_id\",\n",
    "                               aggfunc=\"mean\")\n",
    "\n",
    "pivot_rating['rating'].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1560.5471698113208"
      ]
     },
     "execution_count": 240,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(dat[dat[\"user_id\"] == 1][\"movie_id\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id\n",
       "283     4.962963\n",
       "2339    4.956522\n",
       "3324    4.904762\n",
       "3902    4.890909\n",
       "446     4.843137\n",
       "          ...   \n",
       "5850    1.844828\n",
       "4539    1.815126\n",
       "2744    1.304348\n",
       "4486    1.058824\n",
       "3598    1.015385\n",
       "Name: rating, Length: 6040, dtype: float64"
      ]
     },
     "execution_count": 242,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Argument in groupby is column we are grouping by is \"user_id\"\n",
    "# Column we are interested in performing an operation (mean/avg) is \"ratings\"\n",
    "# Operation is \"mean\"\n",
    "dat.groupby('user_id').mean()[\"rating\"].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id\n",
       "283     4.962963\n",
       "2339    4.956522\n",
       "3324    4.904762\n",
       "3902    4.890909\n",
       "446     4.843137\n",
       "447     4.837838\n",
       "4649    4.818182\n",
       "4634    4.813725\n",
       "1131    4.796117\n",
       "4925    4.761905\n",
       "Name: rating, dtype: float64"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def user_average_ratings(n):\n",
    "    # \"user_id\" are my groups\n",
    "    # \"rating\" is my column of interest\n",
    "    # \"mean\" is my function\n",
    "    user_ratings = dat.groupby('user_id')['rating'].mean()\n",
    "    return user_ratings.sort_values(ascending=False)[:n]\n",
    "\n",
    "user_average_ratings(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that users 4169 and 1680 rated the most movies out of all the users. In addition, it appears that user 283 and 2339 are very lenient with their ratings, as their average rating comes to about 4.95 and above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "zip\n",
       "94110    3802\n",
       "60640    3430\n",
       "98103    3204\n",
       "95616    3079\n",
       "02138    3019\n",
       "55408    2787\n",
       "48135    2725\n",
       "97401    2663\n",
       "10025    2632\n",
       "10024    2594\n",
       "dtype: int64"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def most_populous_area(n):\n",
    "    user_zip = dat.groupby('zip').size()\n",
    "    return user_zip.sort_values(ascending=False)[:n]\n",
    "\n",
    "most_populous_area(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the zip codes, it appears that most users stay in the area of San Francisco and Illinois. Let's find the average age of the movie reviewers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29.73831369243828"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(dat['age'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One may hypothesize that perhaps there is a relationship between a user's age, and the movie genres that he/she may enjoy. \n",
    "\n",
    "On average, the age of a reviewer is about 30. Let's categories the older reviewers as 1.5 times the average age, 45."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>movie_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>gender</th>\n",
       "      <th>age</th>\n",
       "      <th>occupation</th>\n",
       "      <th>zip</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "      <th>older</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978298413</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978220179</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>12</td>\n",
       "      <td>32793</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>15</td>\n",
       "      <td>1193</td>\n",
       "      <td>4</td>\n",
       "      <td>978199279</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>7</td>\n",
       "      <td>22903</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978158471</td>\n",
       "      <td>M</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>95350</td>\n",
       "      <td>One Flew Over the Cuckoo's Nest (1975)</td>\n",
       "      <td>Drama</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  movie_id  rating  timestamp gender  age  occupation    zip  \\\n",
       "0        1      1193       5  978300760      F    1          10  48067   \n",
       "1        2      1193       5  978298413      M   56          16  70072   \n",
       "2       12      1193       4  978220179      M   25          12  32793   \n",
       "3       15      1193       4  978199279      M   25           7  22903   \n",
       "4       17      1193       5  978158471      M   50           1  95350   \n",
       "\n",
       "                                    title genres  older  \n",
       "0  One Flew Over the Cuckoo's Nest (1975)  Drama  False  \n",
       "1  One Flew Over the Cuckoo's Nest (1975)  Drama   True  \n",
       "2  One Flew Over the Cuckoo's Nest (1975)  Drama  False  \n",
       "3  One Flew Over the Cuckoo's Nest (1975)  Drama  False  \n",
       "4  One Flew Over the Cuckoo's Nest (1975)  Drama   True  "
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a binary variable, \"older\" such that if age > 45, returns True\n",
    "dat['older'] = dat['age'] > 40\n",
    "dat.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next code block, we subset the data by 2 different conditions, and then count the number of times a particular genre appears for each subset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [],
   "source": [
    "def most_popular_genres(n):\n",
    "    old_df = dat[dat['older'] == True]\n",
    "    young_df = dat[dat['older'] == False]\n",
    "    \n",
    "    old_genres = old_df.groupby('genres').size()\n",
    "    young_genres = young_df.groupby('genres').size()\n",
    "    \n",
    "    top_old_genres = old_genres.sort_values(ascending=False)[:n]\n",
    "    top_young_genres = young_genres.sort_values(ascending=False)[:n]\n",
    "    \n",
    "    return top_old_genres, top_young_genres\n",
    "\n",
    "top_old_genres, top_young_genres = most_popular_genres(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "genres\n",
       "Drama              26230\n",
       "Comedy             18664\n",
       "Comedy|Romance      8532\n",
       "Comedy|Drama        8191\n",
       "Drama|Romance       6591\n",
       "Action|Thriller     4098\n",
       "Drama|War           3860\n",
       "Horror              3703\n",
       "Drama|Thriller      3647\n",
       "Thriller            3490\n",
       "dtype: int64"
      ]
     },
     "execution_count": 251,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_old_genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "genres\n",
       "Comedy                     98219\n",
       "Drama                      85193\n",
       "Comedy|Romance             34180\n",
       "Comedy|Drama               34054\n",
       "Action|Thriller            22661\n",
       "Drama|Romance              22579\n",
       "Horror                     18860\n",
       "Drama|Thriller             14601\n",
       "Action|Adventure|Sci-Fi    14591\n",
       "Thriller                   14361\n",
       "dtype: int64"
      ]
     },
     "execution_count": 252,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_young_genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([   3,    6,    7, ..., 3697, 3702, 3705]),)"
      ]
     },
     "execution_count": 247,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    movies_groupby = dat.groupby('title').size()\n",
    "    greater_n_ratings = np.where(dat.groupby('title').size() >= 250)\n",
    "    \n",
    "    greater_n_ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the following problem:\n",
    "\n",
    "The Academy Awards (Oscars) has contacted us to select the top films of each year according to the movie genre. We have been given the following criteria for selection:\n",
    "\n",
    "1. For each category, we are interested in movies with 250 ratings and above.\n",
    "2. For each selected film, our tolerance for 1-star ratings is 1%.\n",
    "\n",
    "\n",
    "In the following code block, 4 functions have been defined to help us achieve our objective of selecting the top movie that fulfils both requirements:\n",
    "\n",
    "1. `popular_movies` - This function takes 2 arguments, a dataframe and an integer, and returns a dataframe of movies with more than n ratings.\n",
    "2. `good_review_movies` - This function takes 2 arguments, a dataframe and a floating point, and returns movies with number of 1-star ratings < x%.\n",
    "3. `best_movies` - This function takes 2 arguments, a dataframe and the genre, and returns the top movie for the genre given the dataframe.\n",
    "4. `oscar_winners` - This function takes 3 arguments, a dataframe, an integer and a floating point, and returns a dictionary with genres of movies as the keys and the movie names as the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [],
   "source": [
    "def popular_movies(data, n=250):\n",
    "    '''\n",
    "    Input: \n",
    "        1. data, a pd.DataFrame\n",
    "        2. n, an integer\n",
    "    \n",
    "    Output: movies, pd.DataFrame\n",
    "    \n",
    "    Returns movies with more than n ratings\n",
    "    '''\n",
    "    # Groupby movies data to get movies that have at least N ratings\n",
    "    movies_groupby = data.groupby('title').size()\n",
    "    greater_n_ratings = np.where(data.groupby('title').size() >= n)\n",
    "\n",
    "    top_movies = list(movies_groupby.iloc[greater_n_ratings].index)\n",
    "    \n",
    "    # After getting these top movies, create a new column that returns True if the movie is a top movie\n",
    "    data['top_movies'] = data.title.apply(lambda x: x in top_movies)\n",
    "    return data[data.top_movies == True]\n",
    "\n",
    "def good_reviews_movies(data, tolerance=0.01):\n",
    "    '''\n",
    "    Input: \n",
    "        1. data, a pd.DataFrame\n",
    "        2. tolerance, a floating point\n",
    "    \n",
    "    Output: good_movies, a pd.DataFrame\n",
    "    \n",
    "    Returns movies with number of 1-star rating < 1%\n",
    "    '''\n",
    "    # Group the movie ratings by the number of ratings each movie has\n",
    "    movie_ratings_count = data.groupby('title').count()['movie_id']\n",
    "    \n",
    "    # Subset dataframe into users who rated 1-star\n",
    "    poor_df = data[data.rating == 1]\n",
    "    poor_ratings_count = poor_df.groupby('title').count()['movie_id']\n",
    "    \n",
    "    # Check for the proportion of 1-star ratings across all ratings\n",
    "    tolerance_dict = dict()\n",
    "    for movie in movie_ratings_count.index:\n",
    "        if movie in poor_ratings_count.index:\n",
    "            # Values are proportion of 1* ratings\n",
    "            tolerance_dict[movie] = poor_ratings_count.loc[movie]/movie_ratings_count.loc[movie]\n",
    "        else:\n",
    "            tolerance_dict[movie] = 0\n",
    "    \n",
    "    # Check if each movie passes tolerance rate by iterating over movies in tolerance_dict\n",
    "    good_review_movies = []\n",
    "    for movie in tolerance_dict.keys():\n",
    "        if tolerance_dict[movie] < tolerance: good_review_movies.append(movie)\n",
    "    \n",
    "    data['good_movies'] = data.title.apply(lambda x: x in good_review_movies)\n",
    "    return data[data['good_movies'] == True]\n",
    "\n",
    "def best_movie(data, genre):\n",
    "    '''\n",
    "    Input:\n",
    "        1. data, a pd.DataFrame\n",
    "        2. genre, a string\n",
    "        \n",
    "    Output:\n",
    "        1. top_movie, a string\n",
    "    \n",
    "    Returns the top movie given a genre.\n",
    "    '''\n",
    "    # Subsets data by genre and calculates average ratings\n",
    "    subset = data[data.genres == genre]\n",
    "    movie_ratings = subset.groupby('title')['rating'].mean()\n",
    "    \n",
    "    # Returns top movie based on ratings\n",
    "    top_movie = movie_ratings.sort_values(ascending=False).index[0]\n",
    "    return top_movie\n",
    "    \n",
    "def oscar_winners(data, n=250, tolerance=0.01):\n",
    "    '''\n",
    "    Input:\n",
    "        1. data, a pd.DataFrame\n",
    "        2. n, an integer for the minimum number of ratings\n",
    "        3. tolerance rate, a floating point, for proportion of 1-star ratings\n",
    "        \n",
    "    Output: top_movies, a dicttionary, key-value pairs are genre: movies\n",
    "    \n",
    "    Returns the Oscar winners for each category.\n",
    "    '''\n",
    "    # Get ratings of top movies with at least n reviews and has less than tolerance level\n",
    "    movies_with_n_ratings = popular_movies(data, n)\n",
    "    movies_with_good_reviews = good_reviews_movies(movies_with_n_ratings, tolerance)\n",
    "    \n",
    "    # Generate the Oscar Winners (based on our conditions)\n",
    "    top_movies = dict()\n",
    "    for genre in np.unique(data.genres.values):\n",
    "        try: \n",
    "            oscar = best_movie(movies_with_good_reviews, genre)\n",
    "            top_movies[genre] = oscar\n",
    "        except: pass\n",
    "    return top_movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-253-0b0ea19de782>:52: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data['good_movies'] = data.title.apply(lambda x: x in good_review_movies)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Action': 'From Russia with Love (1963)',\n",
       " 'Action|Adventure': 'Raiders of the Lost Ark (1981)',\n",
       " 'Action|Adventure|Comedy|Crime': 'Midnight Run (1988)',\n",
       " 'Action|Adventure|Comedy|Romance': 'Princess Bride, The (1987)',\n",
       " 'Action|Adventure|Drama': 'Ben-Hur (1959)',\n",
       " 'Action|Adventure|Drama|Sci-Fi|War': 'Star Wars: Episode V - The Empire Strikes Back (1980)',\n",
       " 'Action|Adventure|Fantasy|Sci-Fi': 'Star Wars: Episode IV - A New Hope (1977)',\n",
       " 'Action|Adventure|Romance|War': 'African Queen, The (1951)',\n",
       " 'Action|Adventure|Thriller': 'Clear and Present Danger (1994)',\n",
       " 'Action|Comedy|Western': 'Butch Cassidy and the Sundance Kid (1969)',\n",
       " 'Action|Crime|Drama': 'Godfather, The (1972)',\n",
       " 'Action|Crime|Drama|Thriller': 'French Connection, The (1971)',\n",
       " 'Action|Drama': 'Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954)',\n",
       " 'Action|Drama|Mystery|Romance|Thriller': 'Diva (1981)',\n",
       " 'Action|Drama|War': 'Saving Private Ryan (1998)',\n",
       " 'Action|Sci-Fi|Thriller': 'Terminator, The (1984)',\n",
       " 'Action|Thriller': 'Die Hard (1988)',\n",
       " 'Action|War': 'Dirty Dozen, The (1967)',\n",
       " 'Adventure': 'Treasure of the Sierra Madre, The (1948)',\n",
       " 'Adventure|Comedy|Drama': 'Stand by Me (1986)',\n",
       " 'Adventure|War': 'Lawrence of Arabia (1962)',\n",
       " \"Animation|Children's|Comedy\": 'Toy Story (1995)',\n",
       " 'Animation|Comedy': 'Wrong Trousers, The (1993)',\n",
       " 'Animation|Comedy|Thriller': 'Close Shave, A (1995)',\n",
       " 'Comedy': 'His Girl Friday (1940)',\n",
       " 'Comedy|Crime': 'Sting, The (1973)',\n",
       " 'Comedy|Crime|Drama': 'Dog Day Afternoon (1975)',\n",
       " 'Comedy|Drama': 'Cool Hand Luke (1967)',\n",
       " 'Comedy|Drama|Romance': 'Cinema Paradiso (1988)',\n",
       " 'Comedy|Drama|War': 'Mister Roberts (1955)',\n",
       " 'Comedy|Drama|Western': 'Little Big Man (1970)',\n",
       " 'Comedy|Mystery|Romance|Thriller': 'Charade (1963)',\n",
       " 'Comedy|Mystery|Thriller': 'Arsenic and Old Lace (1944)',\n",
       " 'Comedy|Romance': 'Philadelphia Story, The (1940)',\n",
       " 'Comedy|Romance|Thriller': 'To Catch a Thief (1955)',\n",
       " 'Comedy|Sci-Fi': 'Back to the Future (1985)',\n",
       " 'Crime|Drama': 'On the Waterfront (1954)',\n",
       " 'Crime|Drama|Film-Noir|Thriller': 'Key Largo (1948)',\n",
       " 'Crime|Drama|Romance|Thriller': 'Professional, The (a.k.a. Leon: The Professional) (1994)',\n",
       " 'Crime|Drama|Sci-Fi': 'Sneakers (1992)',\n",
       " 'Crime|Film-Noir': 'Double Indemnity (1944)',\n",
       " 'Crime|Film-Noir|Mystery': 'Laura (1944)',\n",
       " 'Crime|Film-Noir|Mystery|Thriller': 'L.A. Confidential (1997)',\n",
       " 'Crime|Film-Noir|Thriller': 'M (1931)',\n",
       " 'Crime|Thriller': 'Usual Suspects, The (1995)',\n",
       " 'Documentary': 'When We Were Kings (1996)',\n",
       " 'Drama': 'Shawshank Redemption, The (1994)',\n",
       " 'Drama|Film-Noir': 'Blood Simple (1984)',\n",
       " 'Drama|Mystery': 'Conversation, The (1974)',\n",
       " 'Drama|Romance': 'Graduate, The (1967)',\n",
       " 'Drama|Romance|Thriller': 'Witness (1985)',\n",
       " 'Drama|Romance|War': 'Casablanca (1942)',\n",
       " 'Drama|Sci-Fi': 'Day the Earth Stood Still, The (1951)',\n",
       " 'Drama|Thriller': 'North by Northwest (1959)',\n",
       " 'Drama|War': \"Schindler's List (1993)\",\n",
       " 'Film-Noir': 'Sunset Blvd. (a.k.a. Sunset Boulevard) (1950)',\n",
       " 'Film-Noir|Mystery': 'Maltese Falcon, The (1941)',\n",
       " 'Film-Noir|Mystery|Thriller': 'Chinatown (1974)',\n",
       " 'Film-Noir|Romance|Thriller': 'Notorious (1946)',\n",
       " 'Film-Noir|Sci-Fi': 'Blade Runner (1982)',\n",
       " 'Film-Noir|Thriller': 'Manchurian Candidate, The (1962)',\n",
       " 'Mystery': 'Thin Man, The (1934)',\n",
       " 'Mystery|Thriller': 'Rear Window (1954)',\n",
       " 'Romance|Thriller': 'Rebecca (1940)',\n",
       " 'Sci-Fi|War': 'Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963)',\n",
       " 'Thriller': 'Sixth Sense, The (1999)',\n",
       " 'Western': 'Pale Rider (1985)'}"
      ]
     },
     "execution_count": 254,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oscar_winners(dat, n=250, tolerance=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After writing our helper functions, let's proceed to find the Oscar Winners for each category!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "oscar_winners = oscar_winners(dat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the Oscar winners!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "oscar_winners"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the dictionary, we can answer the following questions:\n",
    "1. Which movies won the most Oscars?\n",
    "2. Which movie, out of all the Oscar Winners, had the highest rating?\n",
    "3. How many movies won more than 1 Oscar?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "oscars_won = {}\n",
    "for movie in oscar_winners.values():\n",
    "    # Have I seen the movie before in the oscars_won dictionary?\n",
    "    if movie in oscars_won.keys():\n",
    "        # If so, add 1 to the number of oscars it has already won\n",
    "        oscars_won[movie] += 1\n",
    "    else:\n",
    "        # If not, initialize the first entry in the dictionary with 1\n",
    "        oscars_won[movie] = 1\n",
    "\n",
    "oscars_won\n",
    "\n",
    "import operator\n",
    "\n",
    "max(oscars_won.items(), key=operator.itemgetter(1))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_movie = 'NA'\n",
    "best_rating = 0\n",
    "\n",
    "subset = dat.groupby('title')['rating'].mean()\n",
    "\n",
    "for movie in oscars_won.keys():\n",
    "    movie_rating = subset[movie]\n",
    "    if movie_rating > best_rating:\n",
    "        best_movie = movie\n",
    "        best_rating = movie_rating\n",
    "\n",
    "best_movie, best_rating"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}