intermediate_machine_learning/03-categorical-variables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Intermediate Machine Learning Home Page](https://www.kaggle.com/learn/intermediate-machine-learning)**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By encoding **categorical variables**, you'll obtain your best results thus far!\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "# Set up code checking\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.ml_intermediate.ex3 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, you will work with data from the [Housing Prices Competition for Kaggle Learn Users](https://www.kaggle.com/c/home-data-for-ml-course). \n",
    "\n",
    "![Ames Housing dataset image](https://i.imgur.com/lTJVG4e.png)\n",
    "\n",
    "Run the next code cell without changes to load the training and validation sets in `X_train`, `X_valid`, `y_train`, and `y_valid`.  The test set is loaded in `X_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data\n",
    "X = pd.read_csv('../input/train.csv', index_col='Id') \n",
    "X_test = pd.read_csv('../input/test.csv', index_col='Id')\n",
    "\n",
    "# Remove rows with missing target, separate target from predictors\n",
    "X.dropna(axis=0, subset=['SalePrice'], inplace=True)\n",
    "y = X.SalePrice\n",
    "X.drop(['SalePrice'], axis=1, inplace=True)\n",
    "\n",
    "# To keep things simple, we'll drop columns with missing values\n",
    "cols_with_missing = [col for col in X.columns if X[col].isnull().any()] \n",
    "X.drop(cols_with_missing, axis=1, inplace=True)\n",
    "X_test.drop(cols_with_missing, axis=1, inplace=True)\n",
    "\n",
    "# Break off validation set from training data\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y,\n",
    "                                                      train_size=0.8, test_size=0.2,\n",
    "                                                      random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the next code cell to print the first five rows of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>...</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>619</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>11694</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NridgHt</td>\n",
       "      <td>...</td>\n",
       "      <td>108</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>260</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2007</td>\n",
       "      <td>New</td>\n",
       "      <td>Partial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>6600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NAmes</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>30</td>\n",
       "      <td>RL</td>\n",
       "      <td>13360</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>HLS</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Crawfor</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2009</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>818</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>13265</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>CulDSac</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Mitchel</td>\n",
       "      <td>...</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>13704</td>\n",
       "      <td>Pave</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>...</td>\n",
       "      <td>81</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 60 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     MSSubClass MSZoning  LotArea Street LotShape LandContour Utilities  \\\n",
       "Id                                                                        \n",
       "619          20       RL    11694   Pave      Reg         Lvl    AllPub   \n",
       "871          20       RL     6600   Pave      Reg         Lvl    AllPub   \n",
       "93           30       RL    13360   Pave      IR1         HLS    AllPub   \n",
       "818          20       RL    13265   Pave      IR1         Lvl    AllPub   \n",
       "303          20       RL    13704   Pave      IR1         Lvl    AllPub   \n",
       "\n",
       "    LotConfig LandSlope Neighborhood  ... OpenPorchSF EnclosedPorch 3SsnPorch  \\\n",
       "Id                                    ...                                       \n",
       "619    Inside       Gtl      NridgHt  ...         108             0         0   \n",
       "871    Inside       Gtl        NAmes  ...           0             0         0   \n",
       "93     Inside       Gtl      Crawfor  ...           0            44         0   \n",
       "818   CulDSac       Gtl      Mitchel  ...          59             0         0   \n",
       "303    Corner       Gtl      CollgCr  ...          81             0         0   \n",
       "\n",
       "    ScreenPorch  PoolArea  MiscVal  MoSold  YrSold SaleType SaleCondition  \n",
       "Id                                                                         \n",
       "619         260         0        0       7    2007      New       Partial  \n",
       "871           0         0        0       8    2009       WD        Normal  \n",
       "93            0         0        0       8    2009       WD        Normal  \n",
       "818           0         0        0       7    2008       WD        Normal  \n",
       "303           0         0        0       1    2006       WD        Normal  \n",
       "\n",
       "[5 rows x 60 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the dataset contains both numerical and categorical variables.  You'll need to encode the categorical data before training a model.\n",
    "\n",
    "To compare different models, you'll use the same `score_dataset()` function from the tutorial.  This function reports the [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error) (MAE) from a random forest model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "# function for comparing different approaches\n",
    "def score_dataset(X_train, X_valid, y_train, y_valid):\n",
    "    model = RandomForestRegressor(n_estimators=100, random_state=0)\n",
    "    model.fit(X_train, y_train)\n",
    "    preds = model.predict(X_valid)\n",
    "    return mean_absolute_error(y_valid, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Drop columns with categorical data\n",
    "\n",
    "You'll get started with the most straightforward approach.  Use the code cell below to preprocess the data in `X_train` and `X_valid` to remove columns with categorical data.  Set the preprocessed DataFrames to `drop_X_train` and `drop_X_valid`, respectively.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_Drop\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the lines below: drop columns in training and validation data\n",
    "drop_X_train = X_train.select_dtypes(exclude=['object'])\n",
    "drop_X_valid = X_valid.select_dtypes(exclude=['object'])\n",
    "\n",
    "# Check your answers\n",
    "step_1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_1.hint()\n",
    "#step_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to get the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE from Approach 1 (Drop categorical variables):\n",
      "17837.82570776256\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE from Approach 1 (Drop categorical variables):\")\n",
    "print(score_dataset(drop_X_train, drop_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Label encoding\n",
    "\n",
    "Before jumping into label encoding, we'll investigate the dataset.  Specifically, we'll look at the `'Condition2'` column.  The code cell below prints the unique entries in both the training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique values in 'Condition2' column in training data: ['Norm' 'PosA' 'Feedr' 'PosN' 'Artery' 'RRAe']\n",
      "\n",
      "Unique values in 'Condition2' column in validation data: ['Norm' 'RRAn' 'RRNn' 'Artery' 'Feedr' 'PosN']\n"
     ]
    }
   ],
   "source": [
    "print(\"Unique values in 'Condition2' column in training data:\", X_train['Condition2'].unique())\n",
    "print(\"\\nUnique values in 'Condition2' column in validation data:\", X_valid['Condition2'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you now write code to: \n",
    "- fit a label encoder to the training data, and then \n",
    "- use it to transform both the training and validation data, \n",
    "\n",
    "you'll get an error.  Can you see why this is the case?  (_You'll need  to use the above output to answer this question._)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# step_2.a.hint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"2.1_LabelA\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> Fitting a label encoder to a column in the training data creates a corresponding integer-valued label for each unique value **that appears in the training data**. In the case that the validation data contains values that don't also appear in the training data, the encoder will throw an error, because these values won't have an integer assigned to them.  Notice that the `'Condition2'` column in the validation data contains the values `'RRAn'` and `'RRNn'`, but these don't appear in the training data -- thus, if we try to use a label encoder with scikit-learn, the code will throw an error."
      ],
      "text/plain": [
       "Solution: Fitting a label encoder to a column in the training data creates a corresponding integer-valued label for each unique value **that appears in the training data**. In the case that the validation data contains values that don't also appear in the training data, the encoder will throw an error, because these values won't have an integer assigned to them.  Notice that the `'Condition2'` column in the validation data contains the values `'RRAn'` and `'RRNn'`, but these don't appear in the training data -- thus, if we try to use a label encoder with scikit-learn, the code will throw an error."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "step_2.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a common problem that you'll encounter with real-world data, and there are many approaches to fixing this issue.  For instance, you can write a custom label encoder to deal with new categories.  The simplest approach, however, is to drop the problematic categorical columns.  \n",
    "\n",
    "Run the code cell below to save the problematic columns to a Python list `bad_label_cols`.  Likewise, columns that can be safely label encoded are stored in `good_label_cols`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical columns that will be label encoded: ['MSZoning', 'Street', 'LotShape', 'LandContour', 'LotConfig', 'BldgType', 'HouseStyle', 'ExterQual', 'CentralAir', 'KitchenQual', 'PavedDrive', 'SaleCondition']\n",
      "\n",
      "Categorical columns that will be dropped from the dataset: ['Neighborhood', 'RoofStyle', 'ExterCond', 'Functional', 'Condition2', 'Heating', 'Exterior2nd', 'RoofMatl', 'Utilities', 'Foundation', 'LandSlope', 'HeatingQC', 'Condition1', 'SaleType', 'Exterior1st']\n"
     ]
    }
   ],
   "source": [
    "# All categorical columns\n",
    "object_cols = [col for col in X_train.columns if X_train[col].dtype == \"object\"]\n",
    "\n",
    "# Columns that can be safely label encoded\n",
    "good_label_cols = [col for col in object_cols if \n",
    "                   set(X_train[col]) == set(X_valid[col])]\n",
    "        \n",
    "# Problematic columns that will be dropped from the dataset\n",
    "bad_label_cols = list(set(object_cols)-set(good_label_cols))\n",
    "        \n",
    "print('Categorical columns that will be label encoded:', good_label_cols)\n",
    "print('\\nCategorical columns that will be dropped from the dataset:', bad_label_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the next code cell to label encode the data in `X_train` and `X_valid`.  Set the preprocessed DataFrames to `label_X_train` and `label_X_valid`, respectively.  \n",
    "- We have provided code below to drop the categorical columns in `bad_label_cols` from the dataset. \n",
    "- You should label encode the categorical columns in `good_label_cols`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2.2_LabelB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# Drop categorical columns that will not be encoded\n",
    "label_X_train = X_train.drop(bad_label_cols, axis=1)\n",
    "label_X_valid = X_valid.drop(bad_label_cols, axis=1)\n",
    "\n",
    "# Apply label encoder \n",
    "# Your code here\n",
    "labelEncoder = LabelEncoder()\n",
    "for col in good_label_cols:\n",
    "    label_X_train[col] = labelEncoder.fit_transform(X_train[col])\n",
    "    label_X_valid[col] = labelEncoder.transform(X_valid[col])\n",
    "    \n",
    "# Check your answer\n",
    "step_2.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_2.b.hint()\n",
    "#step_2.b.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to get the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE from Approach 2 (Label Encoding):\n",
      "17575.291883561644\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE from Approach 2 (Label Encoding):\") \n",
    "print(score_dataset(label_X_train, label_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Investigating cardinality\n",
    "\n",
    "So far, you've tried two different approaches to dealing with categorical variables.  And, you've seen that encoding categorical data yields better results than removing columns from the dataset.\n",
    "\n",
    "Soon, you'll try one-hot encoding.  Before then, there's one additional topic we need to cover.  Begin by running the next code cell without changes.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Street', 2),\n",
       " ('Utilities', 2),\n",
       " ('CentralAir', 2),\n",
       " ('LandSlope', 3),\n",
       " ('PavedDrive', 3),\n",
       " ('LotShape', 4),\n",
       " ('LandContour', 4),\n",
       " ('ExterQual', 4),\n",
       " ('KitchenQual', 4),\n",
       " ('MSZoning', 5),\n",
       " ('LotConfig', 5),\n",
       " ('BldgType', 5),\n",
       " ('ExterCond', 5),\n",
       " ('HeatingQC', 5),\n",
       " ('Condition2', 6),\n",
       " ('RoofStyle', 6),\n",
       " ('Foundation', 6),\n",
       " ('Heating', 6),\n",
       " ('Functional', 6),\n",
       " ('SaleCondition', 6),\n",
       " ('RoofMatl', 7),\n",
       " ('HouseStyle', 8),\n",
       " ('Condition1', 9),\n",
       " ('SaleType', 9),\n",
       " ('Exterior1st', 15),\n",
       " ('Exterior2nd', 16),\n",
       " ('Neighborhood', 25)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get number of unique entries in each column with categorical data\n",
    "object_nunique = list(map(lambda col: X_train[col].nunique(), object_cols))\n",
    "d = dict(zip(object_cols, object_nunique))\n",
    "\n",
    "# Print number of unique entries by column, in ascending order\n",
    "sorted(d.items(), key=lambda x: x[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows, for each column with categorical data, the number of unique values in the column.  For instance, the `'Street'` column in the training data has two unique values: `'Grvl'` and `'Pave'`, corresponding to a gravel road and a paved road, respectively.\n",
    "\n",
    "We refer to the number of unique entries of a categorical variable as the **cardinality** of that categorical variable.  For instance, the `'Street'` variable has cardinality 2.\n",
    "\n",
    "Use the output above to answer the questions below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"3.1_CardinalityA\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: How many categorical variables in the training data\n",
    "# have cardinality greater than 10?\n",
    "high_cardinality_numcols = 3\n",
    "\n",
    "# Fill in the line below: How many columns are needed to one-hot encode the \n",
    "# 'Neighborhood' variable in the training data?\n",
    "num_cols_neighborhood = 25\n",
    "\n",
    "# Check your answers\n",
    "step_3.a.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_3.a.hint()\n",
    "#step_3.a.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For large datasets with many rows, one-hot encoding can greatly expand the size of the dataset.  For this reason, we typically will only one-hot encode columns with relatively low cardinality.  Then, high cardinality columns can either be dropped from the dataset, or we can use label encoding.\n",
    "\n",
    "As an example, consider a dataset with 10,000 rows, and containing one categorical column with 100 unique entries.  \n",
    "- If this column is replaced with the corresponding one-hot encoding, how many entries are added to the dataset?  \n",
    "- If we instead replace the column with the label encoding, how many entries are added?  \n",
    "\n",
    "Use your answers to fill in the lines below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"3.2_CardinalityB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fill in the line below: How many entries are added to the dataset by \n",
    "# replacing the column with a one-hot encoding?\n",
    "OH_entries_added = 10000 * 99\n",
    "\n",
    "# Fill in the line below: How many entries are added to the dataset by\n",
    "# replacing the column with a label encoding?\n",
    "label_entries_added = 0\n",
    "\n",
    "# Check your answers\n",
    "step_3.b.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_3.b.hint()\n",
    "#step_3.b.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: One-hot encoding\n",
    "\n",
    "In this step, you'll experiment with one-hot encoding.  But, instead of encoding all of the categorical variables in the dataset, you'll only create a one-hot encoding for columns with cardinality less than 10.\n",
    "\n",
    "Run the code cell below without changes to set `low_cardinality_cols` to a Python list containing the columns that will be one-hot encoded.  Likewise, `high_cardinality_cols` contains a list of categorical columns that will be dropped from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical columns that will be one-hot encoded: ['MSZoning', 'Street', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'ExterQual', 'ExterCond', 'Foundation', 'Heating', 'HeatingQC', 'CentralAir', 'KitchenQual', 'Functional', 'PavedDrive', 'SaleType', 'SaleCondition']\n",
      "\n",
      "Categorical columns that will be dropped from the dataset: ['Neighborhood', 'Exterior2nd', 'Exterior1st']\n"
     ]
    }
   ],
   "source": [
    "# Columns that will be one-hot encoded\n",
    "low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]\n",
    "\n",
    "# Columns that will be dropped from the dataset\n",
    "high_cardinality_cols = list(set(object_cols)-set(low_cardinality_cols))\n",
    "\n",
    "print('Categorical columns that will be one-hot encoded:', low_cardinality_cols)\n",
    "print('\\nCategorical columns that will be dropped from the dataset:', high_cardinality_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the next code cell to one-hot encode the data in `X_train` and `X_valid`.  Set the preprocessed DataFrames to `OH_X_train` and `OH_X_valid`, respectively.  \n",
    "- The full list of categorical columns in the dataset can be found in the Python list `object_cols`.\n",
    "- You should only one-hot encode the categorical columns in `low_cardinality_cols`.  All other categorical columns should be dropped from the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4_OneHot\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Use as many lines of code as you need!\n",
    "OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)\n",
    "OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))\n",
    "OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[low_cardinality_cols]))\n",
    "\n",
    "# One-hot encoding removed index; put it back\n",
    "OH_cols_train.index = X_train.index\n",
    "OH_cols_valid.index = X_valid.index\n",
    "\n",
    "# Remove categorical columns (will replace with one-hot encoding)\n",
    "num_X_train = X_train.drop(object_cols, axis=1)\n",
    "num_X_valid = X_valid.drop(object_cols, axis=1)\n",
    "\n",
    "# Add one-hot encoded columns to numerical features\n",
    "OH_X_train = pd.concat([num_X_train, OH_cols_train], axis=1)\n",
    "OH_X_valid = pd.concat([num_X_valid, OH_cols_valid], axis=1)\n",
    "\n",
    "# Check your answer\n",
    "step_4.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#step_4.hint()\n",
    "#step_4.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to get the MAE for this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE from Approach 3 (One-Hot Encoding):\n",
      "17525.345719178084\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE from Approach 3 (One-Hot Encoding):\") \n",
    "print(score_dataset(OH_X_train, OH_X_valid, y_train, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5: Generate test predictions and submit your results\n",
    "\n",
    "After you complete Step 4, if you'd like to use what you've learned to submit your results to the leaderboard, you'll need to preprocess the test data before generating predictions.\n",
    "\n",
    "**This step is completely optional, and you do not need to submit results to the leaderboard to successfully complete the exercise.**\n",
    "\n",
    "Check out the previous exercise if you need help with remembering how to [join the competition](https://www.kaggle.com/c/home-data-for-ml-course) or save your results to CSV.  Once you have generated a file with your results, follow the instructions below:\n",
    "- Begin by clicking on the blue **COMMIT** button in the top right corner.  This will generate a pop-up window.  \n",
    "- After your code has finished running, click on the blue **Open Version** button in the top right of the pop-up window.  This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n",
    "- Click on the **Output** tab on the left of the screen.  Then, click on the **Submit to Competition** button to submit your results to the leaderboard.\n",
    "- If you want to keep working to improve your performance, select the blue **Edit** button in the top right of the screen. Then you can change your model and repeat the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical columns that will be label encoded: ['Street', 'LotShape', 'LandContour', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'ExterQual', 'ExterCond', 'Foundation', 'Heating', 'HeatingQC', 'CentralAir', 'PavedDrive', 'SaleCondition']\n",
      "\n",
      "Categorical columns that will be dropped from the dataset: ['MSZoning', 'KitchenQual', 'Functional', 'Exterior2nd', 'Utilities', 'SaleType', 'Exterior1st']\n"
     ]
    }
   ],
   "source": [
    "X_test.fillna(0, inplace=True)\n",
    "\n",
    "# All categorical columns\n",
    "object_cols = [col for col in X.columns if X[col].dtype == \"object\"]\n",
    "\n",
    "# Columns that can be safely label encoded\n",
    "good_label_cols = [col for col in object_cols if \n",
    "                   set(X_test[col]).issubset(set(X[col]))]\n",
    "        \n",
    "# Problematic columns that will be dropped from the dataset\n",
    "bad_label_cols = list(set(object_cols)-set(good_label_cols))\n",
    "        \n",
    "print('Categorical columns that will be label encoded:', good_label_cols)\n",
    "print('\\nCategorical columns that will be dropped from the dataset:', bad_label_cols)\n",
    "\n",
    "# Drop categorical columns that will not be encoded\n",
    "label_X = X.drop(bad_label_cols, axis=1)\n",
    "label_X_test = X_test.drop(bad_label_cols, axis=1)\n",
    "\n",
    "# Apply label encoder \n",
    "# Your code here\n",
    "labelEncoder = LabelEncoder()\n",
    "for col in good_label_cols:\n",
    "    label_X[col] = labelEncoder.fit_transform(X[col])\n",
    "    label_X_test[col] = labelEncoder.transform(X_test[col])\n",
    "\n",
    "# Model with 200 estimators, MAE: 15923.57616\n",
    "model = RandomForestRegressor(n_estimators=200, random_state=0)\n",
    "model.fit(label_X, y)\n",
    "\n",
    "# Get test predictions \n",
    "preds_test = model.predict(label_X_test)\n",
    "\n",
    "# Save test predictions to file\n",
    "output = pd.DataFrame({'Id': label_X_test.index,\n",
    "                       'SalePrice': preds_test})\n",
    "output.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keep going\n",
    "\n",
    "With missing value handling and categorical encoding, your modeling process is getting complex. This complexity gets worse when you want to save your model to use in the future. The key to managing this complexity is something called **pipelines**. \n",
    "\n",
    "**[Learn to use pipelines](https://www.kaggle.com/alexisbcook/pipelines)** to preprocess datasets with categorical variables, missing values and any other messiness your data throws at you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Intermediate Machine Learning Home Page](https://www.kaggle.com/learn/intermediate-machine-learning)**\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}