section-04-research-and-development/05-machine-learning-pipeline-scoring-new-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Pipeline - Scoring New Data\n",
    "\n",
    "Let's imagine that a colleague from the business department comes and asks us to score the data from last months customers. They want to be sure that our model is working appropriately in the most recent data that the organization has.\n",
    "\n",
    "**How would you go about to score the new data?** Try to give it a go. There is more than 1 way of doing it.\n",
    "\n",
    "Below we present one potential solution.\n",
    "\n",
    "What could we have done better?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to handle datasets\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# for plotting\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# for the yeo-johnson transformation\n",
    "import scipy.stats as stats\n",
    "\n",
    "# to save the model\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1459, 80)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1461</td>\n",
       "      <td>20</td>\n",
       "      <td>RH</td>\n",
       "      <td>80.0</td>\n",
       "      <td>11622</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>120</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1462</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>81.0</td>\n",
       "      <td>14267</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gar2</td>\n",
       "      <td>12500</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1463</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>74.0</td>\n",
       "      <td>13830</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1464</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>78.0</td>\n",
       "      <td>9978</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1465</td>\n",
       "      <td>120</td>\n",
       "      <td>RL</td>\n",
       "      <td>43.0</td>\n",
       "      <td>5005</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>HLS</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>144</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 80 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0  1461          20       RH         80.0    11622   Pave   NaN      Reg   \n",
       "1  1462          20       RL         81.0    14267   Pave   NaN      IR1   \n",
       "2  1463          60       RL         74.0    13830   Pave   NaN      IR1   \n",
       "3  1464          60       RL         78.0     9978   Pave   NaN      IR1   \n",
       "4  1465         120       RL         43.0     5005   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities  ... ScreenPorch PoolArea PoolQC  Fence MiscFeature  \\\n",
       "0         Lvl    AllPub  ...         120        0    NaN  MnPrv         NaN   \n",
       "1         Lvl    AllPub  ...           0        0    NaN    NaN        Gar2   \n",
       "2         Lvl    AllPub  ...           0        0    NaN  MnPrv         NaN   \n",
       "3         Lvl    AllPub  ...           0        0    NaN    NaN         NaN   \n",
       "4         HLS    AllPub  ...         144        0    NaN    NaN         NaN   \n",
       "\n",
       "  MiscVal MoSold  YrSold  SaleType  SaleCondition  \n",
       "0       0      6    2010        WD         Normal  \n",
       "1   12500      6    2010        WD         Normal  \n",
       "2       0      3    2010        WD         Normal  \n",
       "3       0      6    2010        WD         Normal  \n",
       "4       0      1    2010        WD         Normal  \n",
       "\n",
       "[5 rows x 80 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the unseen / new dataset\n",
    "data = pd.read_csv('test.csv')\n",
    "\n",
    "# rows and columns of the data\n",
    "print(data.shape)\n",
    "\n",
    "# visualise the dataset\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1459, 79)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop the id variable\n",
    "\n",
    "data.drop('Id', axis=1, inplace=True)\n",
    "\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering\n",
    "\n",
    "First we need to transform the data. Below the list of transformations that we did during the Feature Engineering phase:\n",
    "\n",
    "1. Missing values\n",
    "2. Temporal variables\n",
    "3. Non-Gaussian distributed variables\n",
    "4. Categorical variables: remove rare labels\n",
    "5. Categorical variables: convert strings to numbers\n",
    "6. Put the variables in a similar scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing values\n",
    "\n",
    "### Categorical variables\n",
    "\n",
    "- Replace missing values with the string \"missing\" in those variables with a lot of missing data. \n",
    "- Replace missing data with the most frequent category in those variables that contain fewer observations without values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first we needed to cast MSSubClass as object\n",
    "\n",
    "data['MSSubClass'] = data['MSSubClass'].astype('O')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list of different groups of categorical variables\n",
    "\n",
    "with_string_missing = ['Alley', 'FireplaceQu',\n",
    "                       'PoolQC', 'Fence', 'MiscFeature']\n",
    "\n",
    "# ==================\n",
    "# we copy this dictionary from the Feature-engineering notebook\n",
    "# note that we needed to hard-code this by hand\n",
    "\n",
    "# the key is the variable and the value is its most frequent category\n",
    "\n",
    "# what if we re-train the model and the below values change?\n",
    "# ==================\n",
    "\n",
    "with_frequent_category = {\n",
    "    'MasVnrType': 'None',\n",
    "    'BsmtQual': 'TA',\n",
    "    'BsmtCond': 'TA',\n",
    "    'BsmtExposure': 'No',\n",
    "    'BsmtFinType1': 'Unf',\n",
    "    'BsmtFinType2': 'Unf',\n",
    "    'Electrical': 'SBrkr',\n",
    "    'GarageType': 'Attchd',\n",
    "    'GarageFinish': 'Unf',\n",
    "    'GarageQual': 'TA',\n",
    "    'GarageCond': 'TA',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace missing values with new label: \"Missing\"\n",
    "\n",
    "data[with_string_missing] = data[with_string_missing].fillna('Missing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace missing values with the most frequent category\n",
    "\n",
    "for var in with_frequent_category.keys():\n",
    "    data[var].fillna(with_frequent_category[var], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numerical variables\n",
    "\n",
    "To engineer missing values in numerical variables, we will:\n",
    "\n",
    "- add a binary missing value indicator variable\n",
    "- and then replace the missing values in the original variable with the mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this is the dictionary of numerical variable with missing data\n",
    "# and its mean, as determined from the training set in the\n",
    "# Feature Engineering notebook\n",
    "\n",
    "# note how we needed to hard code the values\n",
    "\n",
    "vars_with_na = {\n",
    "    'LotFrontage': 69.87974098057354,\n",
    "    'MasVnrArea': 103.7974006116208,\n",
    "    'GarageYrBlt': 1978.2959677419356,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0\n",
       "MasVnrArea     0\n",
       "GarageYrBlt    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace missing values as we described above\n",
    "\n",
    "for var in vars_with_na.keys():\n",
    "\n",
    "    # add binary missing indicator (in train and test)\n",
    "    data[var + '_na'] = np.where(data[var].isnull(), 1, 0)\n",
    "\n",
    "    # replace missing values by the mean\n",
    "    # (in train and test)\n",
    "    data[var].fillna(vars_with_na[var], inplace=True)\n",
    "\n",
    "data[vars_with_na].isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage_na</th>\n",
       "      <th>MasVnrArea_na</th>\n",
       "      <th>GarageYrBlt_na</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage_na  MasVnrArea_na  GarageYrBlt_na\n",
       "0               0              0               0\n",
       "1               0              0               0\n",
       "2               0              0               0\n",
       "3               0              0               0\n",
       "4               0              0               0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the binary missing indicator variables\n",
    "\n",
    "data[['LotFrontage_na', 'MasVnrArea_na', 'GarageYrBlt_na']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal variables\n",
    "\n",
    "### Capture elapsed time\n",
    "\n",
    "We need to capture the time elapsed between those variables and the year in which the house was sold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def elapsed_years(df, var):\n",
    "    # capture difference between the year variable\n",
    "    # and the year in which the house was sold\n",
    "    df[var] = df['YrSold'] - df[var]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "for var in ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt']:\n",
    "    data = elapsed_years(data, var)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we drop YrSold\n",
    "data.drop(['YrSold'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numerical variable transformation\n",
    "\n",
    "### Logarithmic transformation\n",
    "\n",
    "We will transform with the logarithm the positive numerical variables in order to get a more Gaussian-like distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "for var in [\"LotFrontage\", \"1stFlrSF\", \"GrLivArea\"]:\n",
    "    data[var] = np.log(data[var])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Yeo-Johnson transformation\n",
    "\n",
    "We will apply the Yeo-Johnson transformation to LotArea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note how we use the lambda that we learned from the train set\n",
    "# in the notebook on Feature Engineering.\n",
    "\n",
    "# Note that we need to hard code this value\n",
    "\n",
    "data['LotArea'] = stats.yeojohnson(data['LotArea'], lmbda=-12.55283001172003)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binarize skewed variables\n",
    "\n",
    "There were a few variables very skewed, we would transform those into binary variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "skewed = [\n",
    "    'BsmtFinSF2', 'LowQualFinSF', 'EnclosedPorch',\n",
    "    '3SsnPorch', 'ScreenPorch', 'MiscVal'\n",
    "]\n",
    "\n",
    "for var in skewed:\n",
    "    \n",
    "    # map the variable values into 0 and 1\n",
    "    data[var] = np.where(data[var]==0, 0, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical variables\n",
    "\n",
    "### Apply mappings\n",
    "\n",
    "We remap variables with specific meanings into a numerical scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# re-map strings to numbers, which determine quality\n",
    "\n",
    "qual_mappings = {'Po': 1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex': 5, 'Missing': 0, 'NA': 0}\n",
    "\n",
    "qual_vars = ['ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond',\n",
    "             'HeatingQC', 'KitchenQual', 'FireplaceQu',\n",
    "             'GarageQual', 'GarageCond',\n",
    "            ]\n",
    "\n",
    "for var in qual_vars:\n",
    "    data[var] = data[var].map(qual_mappings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "exposure_mappings = {'No': 1, 'Mn': 2, 'Av': 3, 'Gd': 4}\n",
    "\n",
    "var = 'BsmtExposure'\n",
    "\n",
    "data[var] = data[var].map(exposure_mappings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "finish_mappings = {'Missing': 0, 'NA': 0, 'Unf': 1, 'LwQ': 2, 'Rec': 3, 'BLQ': 4, 'ALQ': 5, 'GLQ': 6}\n",
    "\n",
    "finish_vars = ['BsmtFinType1', 'BsmtFinType2']\n",
    "\n",
    "for var in finish_vars:\n",
    "    data[var] = data[var].map(finish_mappings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "garage_mappings = {'Missing': 0, 'NA': 0, 'Unf': 1, 'RFn': 2, 'Fin': 3}\n",
    "\n",
    "var = 'GarageFinish'\n",
    "\n",
    "data[var] = data[var].map(garage_mappings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "fence_mappings = {'Missing': 0, 'NA': 0, 'MnWw': 1, 'GdWo': 2, 'MnPrv': 3, 'GdPrv': 4}\n",
    "\n",
    "var = 'Fence'\n",
    "\n",
    "data[var] = data[var].map(fence_mappings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MSZoning',\n",
       " 'Utilities',\n",
       " 'Exterior1st',\n",
       " 'Exterior2nd',\n",
       " 'BsmtFinSF1',\n",
       " 'BsmtUnfSF',\n",
       " 'TotalBsmtSF',\n",
       " 'BsmtFullBath',\n",
       " 'BsmtHalfBath',\n",
       " 'KitchenQual',\n",
       " 'Functional',\n",
       " 'GarageCars',\n",
       " 'GarageArea',\n",
       " 'SaleType']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check absence of na in the data set\n",
    "\n",
    "with_null = [var for var in data.columns if data[var].isnull().sum() > 0]\n",
    "\n",
    "with_null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Surprise**\n",
    "\n",
    "There are quite a few variables with missing data!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# did those have missing data in the train set?\n",
    "\n",
    "[var for var in with_null if var in list(\n",
    "    with_frequent_category.keys())+with_string_missing+list(vars_with_na.keys())]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IMPORTANT**\n",
    "\n",
    "In the new data, we have a bunch of variables that contain missing information, that we did not anticipate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Rare Labels\n",
    "\n",
    "For the remaining categorical variables, we will group those categories that are present in less than 1% of the observations into a \"Rare\" string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a dictionary with the most frequent categories per variable\n",
    "\n",
    "# note the amount of hard coding that I need to do.\n",
    "\n",
    "# Can you think of an alternative? Perhaps we could have save this as a numpy pickle\n",
    "# and load it here, instead of hard-coding.\n",
    "\n",
    "# But that means that we need to go back to the Feature Engineering notebook, and change\n",
    "# the code so that we store the pickle. So there is still some code changes that we need\n",
    "\n",
    "frequent_ls = {\n",
    "    'MSZoning': ['FV', 'RH', 'RL', 'RM'],\n",
    "    'Street': ['Pave'],\n",
    "    'Alley': ['Grvl', 'Missing', 'Pave'],\n",
    "    'LotShape': ['IR1', 'IR2', 'Reg'],\n",
    "    'LandContour': ['Bnk', 'HLS', 'Low', 'Lvl'],\n",
    "    'Utilities': ['AllPub'],\n",
    "    'LotConfig': ['Corner', 'CulDSac', 'FR2', 'Inside'],\n",
    "    'LandSlope': ['Gtl', 'Mod'],\n",
    "    'Neighborhood': ['Blmngtn', 'BrDale', 'BrkSide', 'ClearCr', 'CollgCr', 'Crawfor',\n",
    "                     'Edwards', 'Gilbert', 'IDOTRR', 'MeadowV', 'Mitchel', 'NAmes', 'NWAmes',\n",
    "                     'NoRidge', 'NridgHt', 'OldTown', 'SWISU', 'Sawyer', 'SawyerW',\n",
    "                     'Somerst', 'StoneBr', 'Timber'],\n",
    "\n",
    "    'Condition1': ['Artery', 'Feedr', 'Norm', 'PosN', 'RRAn'],\n",
    "    'Condition2': ['Norm'],\n",
    "    'BldgType': ['1Fam', '2fmCon', 'Duplex', 'Twnhs', 'TwnhsE'],\n",
    "    'HouseStyle': ['1.5Fin', '1Story', '2Story', 'SFoyer', 'SLvl'],\n",
    "    'RoofStyle': ['Gable', 'Hip'],\n",
    "    'RoofMatl': ['CompShg'],\n",
    "    'Exterior1st': ['AsbShng', 'BrkFace', 'CemntBd', 'HdBoard', 'MetalSd', 'Plywood',\n",
    "                    'Stucco', 'VinylSd', 'Wd Sdng', 'WdShing'],\n",
    "\n",
    "    'Exterior2nd': ['AsbShng', 'BrkFace', 'CmentBd', 'HdBoard', 'MetalSd', 'Plywood',\n",
    "                    'Stucco', 'VinylSd', 'Wd Sdng', 'Wd Shng'],\n",
    "\n",
    "    'MasVnrType': ['BrkFace', 'None', 'Stone'],\n",
    "    'Foundation': ['BrkTil', 'CBlock', 'PConc', 'Slab'],\n",
    "    'Heating': ['GasA', 'GasW'],\n",
    "    'CentralAir': ['N', 'Y'],\n",
    "    'Electrical': ['FuseA', 'FuseF', 'SBrkr'],\n",
    "    'Functional': ['Min1', 'Min2', 'Mod', 'Typ'],\n",
    "    'GarageType': ['Attchd', 'Basment', 'BuiltIn', 'Detchd'],\n",
    "    'PavedDrive': ['N', 'P', 'Y'],\n",
    "    'PoolQC': ['Missing'],\n",
    "    'MiscFeature': ['Missing', 'Shed'],\n",
    "    'SaleType': ['COD', 'New', 'WD'],\n",
    "    'SaleCondition': ['Abnorml', 'Family', 'Normal', 'Partial'],\n",
    "    'MSSubClass': ['20', '30', '50', '60', '70', '75', '80', '85', '90', '120', '160', '190'],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "for var in frequent_ls.keys():\n",
    "    \n",
    "    # replace rare categories by the string \"Rare\"\n",
    "    data[var] = np.where(data[var].isin(\n",
    "        frequent_ls), data[var], 'Rare')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding of categorical variables\n",
    "\n",
    "Next, we need to transform the strings of the categorical variables into numbers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we need the mappings learned from the train set. Otherwise, our model is going\n",
    "# to produce inaccurate results\n",
    "\n",
    "# note the amount of hard coding that we need to do.\n",
    "\n",
    "# Can you think of an alternative? \n",
    "\n",
    "# Perhaps we could have save this as a numpy pickle\n",
    "# and load it here, instead of hard-coding.\n",
    "\n",
    "# But that means that we need to go back to the Feature Engineering notebook, and change\n",
    "# the code so that we store the pickle. So there is still some code changes that we need\n",
    "\n",
    "ordinal_mappings = {\n",
    "    'MSZoning': {'Rare': 0, 'RM': 1, 'RH': 2, 'RL': 3, 'FV': 4},\n",
    "    'Street': {'Rare': 0, 'Pave': 1},\n",
    "    'Alley': {'Grvl': 0, 'Pave': 1, 'Missing': 2},\n",
    "    'LotShape': {'Reg': 0, 'IR1': 1, 'Rare': 2, 'IR2': 3},\n",
    "    'LandContour': {'Bnk': 0, 'Lvl': 1, 'Low': 2, 'HLS': 3},\n",
    "    'Utilities': {'Rare': 0, 'AllPub': 1},\n",
    "    'LotConfig': {'Inside': 0, 'FR2': 1, 'Corner': 2, 'Rare': 3, 'CulDSac': 4},\n",
    "    'LandSlope': {'Gtl': 0, 'Mod': 1, 'Rare': 2},\n",
    "    'Neighborhood': {'IDOTRR': 0, 'MeadowV': 1, 'BrDale': 2, 'Edwards': 3,\n",
    "                     'BrkSide': 4, 'OldTown': 5, 'Sawyer': 6, 'SWISU': 7,\n",
    "                     'NAmes': 8, 'Mitchel': 9, 'SawyerW': 10, 'Rare': 11,\n",
    "                     'NWAmes': 12, 'Gilbert': 13, 'Blmngtn': 14, 'CollgCr': 15,\n",
    "                     'Crawfor': 16, 'ClearCr': 17, 'Somerst': 18, 'Timber': 19,\n",
    "                     'StoneBr': 20, 'NridgHt': 21, 'NoRidge': 22},\n",
    "    \n",
    "    'Condition1': {'Artery': 0, 'Feedr': 1, 'Norm': 2, 'RRAn': 3, 'Rare': 4, 'PosN': 5},\n",
    "    'Condition2': {'Rare': 0, 'Norm': 1},\n",
    "    'BldgType': {'2fmCon': 0, 'Duplex': 1, 'Twnhs': 2, '1Fam': 3, 'TwnhsE': 4},\n",
    "    'HouseStyle': {'SFoyer': 0, '1.5Fin': 1, 'Rare': 2, '1Story': 3, 'SLvl': 4, '2Story': 5},\n",
    "    'RoofStyle': {'Gable': 0, 'Rare': 1, 'Hip': 2},\n",
    "    'RoofMatl': {'CompShg': 0, 'Rare': 1},\n",
    "    'Exterior1st': {'AsbShng': 0, 'Wd Sdng': 1, 'WdShing': 2, 'MetalSd': 3,\n",
    "                    'Stucco': 4, 'Rare': 5, 'HdBoard': 6, 'Plywood': 7,\n",
    "                    'BrkFace': 8, 'CemntBd': 9, 'VinylSd': 10},\n",
    "    \n",
    "    'Exterior2nd': {'AsbShng': 0, 'Wd Sdng': 1, 'MetalSd': 2, 'Wd Shng': 3,\n",
    "                    'Stucco': 4, 'Rare': 5, 'HdBoard': 6, 'Plywood': 7,\n",
    "                    'BrkFace': 8, 'CmentBd': 9, 'VinylSd': 10},\n",
    "    \n",
    "    'MasVnrType': {'Rare': 0, 'None': 1, 'BrkFace': 2, 'Stone': 3},\n",
    "    'Foundation': {'Slab': 0, 'BrkTil': 1, 'CBlock': 2, 'Rare': 3, 'PConc': 4},\n",
    "    'Heating': {'Rare': 0, 'GasW': 1, 'GasA': 2},\n",
    "    'CentralAir': {'N': 0, 'Y': 1},\n",
    "    'Electrical': {'Rare': 0, 'FuseF': 1, 'FuseA': 2, 'SBrkr': 3},\n",
    "    'Functional': {'Rare': 0, 'Min2': 1, 'Mod': 2, 'Min1': 3, 'Typ': 4},\n",
    "    'GarageType': {'Rare': 0, 'Detchd': 1, 'Basment': 2, 'Attchd': 3, 'BuiltIn': 4},\n",
    "    'PavedDrive': {'N': 0, 'P': 1, 'Y': 2},\n",
    "    'PoolQC': {'Missing': 0, 'Rare': 1},\n",
    "    'MiscFeature': {'Rare': 0, 'Shed': 1, 'Missing': 2},\n",
    "    'SaleType': {'COD': 0, 'Rare': 1, 'WD': 2, 'New': 3},\n",
    "    'SaleCondition': {'Rare': 0, 'Abnorml': 1, 'Family': 2, 'Normal': 3, 'Partial': 4},\n",
    "    'MSSubClass': {'30': 0, 'Rare': 1, '190': 2, '90': 3, '160': 4, '50': 5, '85': 6,\n",
    "                   '70': 7, '80': 8, '20': 9, '75': 10, '120': 11, '60': 12},\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "for var in ordinal_mappings.keys():\n",
    "\n",
    "    ordinal_label = ordinal_mappings[var]\n",
    "\n",
    "    # use the dictionary to replace the categorical strings by integers\n",
    "    data[var] = data[var].map(ordinal_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check absence of na in the data set\n",
    "\n",
    "with_null = [var for var in data.columns if data[var].isnull().sum() > 0]\n",
    "\n",
    "len(with_null)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# there is missing data in a lot of the variables.\n",
    "\n",
    "# unfortunately, the scaler wil not work with missing data, so\n",
    "# we need to fill those values\n",
    "\n",
    "# in the real world, we would try to understand where they are coming from\n",
    "# and why they were not present in the training set\n",
    "\n",
    "# here I will just fill them in quickly to proceed with the demo\n",
    "\n",
    "data.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Scaling\n",
    "\n",
    "We will scale features to the minimum and maximum values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the scaler we saved in the notebook on Feature Engineering\n",
    "\n",
    "# fortunataly, we were smart and we saved it, but this is an easy step\n",
    "# to forget\n",
    "\n",
    "scaler = joblib.load('minmax_scaler.joblib') \n",
    "\n",
    "data = pd.DataFrame(\n",
    "    scaler.transform(data),\n",
    "    columns=data.columns\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>LotFrontage_na</th>\n",
       "      <th>MasVnrArea_na</th>\n",
       "      <th>GarageYrBlt_na</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.495064</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.454545</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.499662</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.454545</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.466207</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.485693</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.454545</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.265271</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   MSSubClass  MSZoning  LotFrontage  LotArea  Street  Alley  LotShape  \\\n",
       "0    0.083333       0.0     0.495064      0.0     0.0    0.0  0.666667   \n",
       "1    0.083333       0.0     0.499662      0.0     0.0    0.0  0.666667   \n",
       "2    0.083333       0.0     0.466207      0.0     0.0    0.0  0.666667   \n",
       "3    0.083333       0.0     0.485693      0.0     0.0    0.0  0.666667   \n",
       "4    0.083333       0.0     0.265271      0.0     0.0    0.0  0.666667   \n",
       "\n",
       "   LandContour  Utilities  LotConfig  ...  PoolQC  Fence  MiscFeature  \\\n",
       "0          0.0        0.0       0.75  ...     1.0   0.75          0.0   \n",
       "1          0.0        0.0       0.75  ...     1.0   0.00          0.0   \n",
       "2          0.0        0.0       0.75  ...     1.0   0.75          0.0   \n",
       "3          0.0        0.0       0.75  ...     1.0   0.00          0.0   \n",
       "4          0.0        0.0       0.75  ...     1.0   0.00          0.0   \n",
       "\n",
       "   MiscVal    MoSold  SaleType  SaleCondition  LotFrontage_na  MasVnrArea_na  \\\n",
       "0      0.0  0.454545  0.333333            0.0             0.0            0.0   \n",
       "1      1.0  0.454545  0.333333            0.0             0.0            0.0   \n",
       "2      0.0  0.181818  0.333333            0.0             0.0            0.0   \n",
       "3      0.0  0.454545  0.333333            0.0             0.0            0.0   \n",
       "4      0.0  0.000000  0.333333            0.0             0.0            0.0   \n",
       "\n",
       "   GarageYrBlt_na  \n",
       "0             0.0  \n",
       "1             0.0  \n",
       "2             0.0  \n",
       "3             0.0  \n",
       "4             0.0  \n",
       "\n",
       "[5 rows x 81 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1459, 36)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the pre-selected features\n",
    "# ==============================\n",
    "\n",
    "features = pd.read_csv('selected_features.csv')\n",
    "features = features['0'].to_list() \n",
    "\n",
    "# reduce the train and test set to the selected features\n",
    "data = data[features]\n",
    "\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we engineered so many variables, when we are actually going to feed only 31 to the model.\n",
    "\n",
    "**What could we do differently?**\n",
    "\n",
    "We could have, of course, engineered only the variables that we are going to use in the model. But that means:\n",
    "\n",
    "- identifying which variables we need\n",
    "- identifying which transformation we need per variable\n",
    "- redefining our dictionaries accordingly\n",
    "- retraining the MinMaxScaler only on the selected variables (at the moment, it is trained on the entire dataset)\n",
    "\n",
    "That means, that we need to create extra code to train the scaler only on the selected variables. Probably removing the scaler from the Feature Engineering notebook and passing it onto the Feature Selection one.\n",
    "\n",
    "We need to be really careful in re-writing the code here to make sure we do not forget or engineer wrongly any of the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAVKElEQVR4nO3dcYyc9X3n8ff3IBCSzdkmzq0s27qlLUpEcZviPUqUKNotd6kDVU0lFIFQY3KcrLYk5a6uCjTSJfcHOvdONErUuyRu4aCXiIXSVHC0OUId9qJIh1M7JdhACA5xGlvGbhpwuilq6973/phnk2GZ9e7M88zuPD+/X9Jon/k9z/yezzye/fq3v3nmmchMJEll+WerHUCS1DyLuyQVyOIuSQWyuEtSgSzuklSgc1c7AMD69etzYmKiVh8/+MEPeOMb39hMoBVk7pXTxszQztxtzAzty33gwIHvZuZbeq0bieI+MTHB/v37a/UxOzvL1NRUM4FWkLlXThszQztztzEztC93RHx7sXVOy0hSgZYs7hFxd0ScjIhDPdbtioiMiPXV/YiIT0TE4Yh4KiIuG0ZoSdKZLWfkfg+wbWFjRGwG3gP8VVfze4GLq9tO4JP1I0qS+rVkcc/MLwHf67HqY8BvAd3XL9gO/GF2PAGsjYgNjSSVJC3bQG+oRsR24Fhmfi0iuldtBL7Tdf9o1Xa8Rx876YzuGR8fZ3Z2dpAoPzQ3N1e7j9Vg7pXTxszQztxtzAztzd1TZi55AyaAQ9XyG4B9wJrq/hFgfbX8CPCursftBSaX6n/r1q1Z1+OPP167j9Vg7pXTxsyZ7czdxsyZ7csN7M9F6uogI/cfBy4C5kftm4CvRsTlwDFgc9e2m6o2SdIK6vtUyMw8mJn/IjMnMnOCztTLZZn5IvAw8P7qrJkrgFOZ+ZopGUnScC3nVMj7gP8LvDUijkbETWfY/M+AF4DDwO8Dv9ZISklSX5aclsnM65dYP9G1nMDN9WMJYOK2P+3ZfmT31SucRFLb+AlVSSqQxV2SCmRxl6QCWdwlqUAWd0kqkMVdkgpkcZekAlncJalAI/E1e2e7xT6sJEmDcuQuSQWyuEtSgSzuklQgi7skFcjiLkkFsrhLUoEs7pJUIIu7JBXI4i5JBbK4S1KBLO6SVCCvLdNCfnG2pKUsOXKPiLsj4mREHOpq+68R8fWIeCoi/iQi1natuz0iDkfEcxHx80PKLUk6g+VMy9wDbFvQ9hhwaWb+FPAN4HaAiLgEuA74yeox/z0izmksrSRpWZYs7pn5JeB7C9q+kJmnq7tPAJuq5e3ATGb+fWZ+CzgMXN5gXknSMkRmLr1RxATwSGZe2mPd/wLuz8zPRMTvAU9k5meqdXcBn8/MB3s8biewE2B8fHzrzMxMrScyNzfH2NhYrT5Ww9zcHN869U+N9LVl45pG+lmONh7vNmaGduZuY2ZoX+7p6ekDmTnZa12tN1Qj4sPAaeCz/T42M/cAewAmJydzamqqThRmZ2ep28dqmJ2d5c4v/6CRvo7cMNVIP8vRxuPdxszQztxtzAztzd3LwMU9Im4EfgG4Mn80/D8GbO7abFPVphXgWTSS5g10nntEbAN+C/jFzPy7rlUPA9dFxPkRcRFwMfCV+jElSf1YcuQeEfcBU8D6iDgKfITO2THnA49FBHTm2X8lM5+OiAeAZ+hM19ycmc1MKEuSlm3J4p6Z1/dovusM298B3FEnlCSpHi8/IEkFsrhLUoEs7pJUIIu7JBXI4i5JBbK4S1KBLO6SVCCLuyQVyOIuSQWyuEtSgSzuklQgvyB7CLz0rqTV5shdkgrkyH0F9RrR79pyGv8ZJDXNkbskFcjiLkkFsrhLUoEs7pJUIIu7JBXI4i5JBbK4S1KBlizuEXF3RJyMiENdbRdGxGMR8Xz1c13VHhHxiYg4HBFPRcRlwwwvSeptOSP3e4BtC9puA/Zm5sXA3uo+wHuBi6vbTuCTzcSUJPVjyeKemV8CvregeTtwb7V8L3BNV/sfZscTwNqI2NBQVknSMg065z6emcer5ReB8Wp5I/Cdru2OVm2SpBUUmbn0RhETwCOZeWl1/+XMXNu1/qXMXBcRjwC7M/PLVfte4NbM3N+jz510pm4YHx/fOjMzU+uJzM3NMTY2VquPphw8dmrZ245fACdeGWIYYMvGNY33OUrHe7namBnambuNmaF9uaenpw9k5mSvdYNesepERGzIzOPVtMvJqv0YsLlru01V22tk5h5gD8Dk5GROTU0NGKVjdnaWun005cZFLvnby64tp7nz4HAvHHbkhqnG+xyl471cbcwM7czdxszQ3ty9DDot8zCwo1reATzU1f7+6qyZK4BTXdM3kqQVsuSQMSLuA6aA9RFxFPgIsBt4ICJuAr4NvK/a/M+Aq4DDwN8BHxhCZknSEpYs7pl5/SKrruyxbQI31w0lSarHT6hKUoEs7pJUIIu7JBXI4i5JBfKbmc8Cvb6Ye96R3VevYBJJK8WRuyQVyOIuSQWyuEtSgSzuklQgi7skFcjiLkkFsrhLUoEs7pJUIIu7JBXI4i5JBbK4S1KBLO6SVCCLuyQVyOIuSQWyuEtSgSzuklQgi7skFahWcY+I/xART0fEoYi4LyJeHxEXRcS+iDgcEfdHxHlNhZUkLc/AX7MXERuBXwcuycxXIuIB4DrgKuBjmTkTEZ8CbgI+2UhaNW6xr+Dz6/ekdqs7LXMucEFEnAu8ATgO/BzwYLX+XuCamvuQJPUpMnPwB0fcAtwBvAJ8AbgFeCIzf6Javxn4fGZe2uOxO4GdAOPj41tnZmYGzgEwNzfH2NhYrT6acvDYqWVvO34BnHhliGEGtGXjmjOuH6XjvVxtzAztzN3GzNC+3NPT0wcyc7LXujrTMuuA7cBFwMvAHwHblvv4zNwD7AGYnJzMqampQaMAMDs7S90+mnLjIlMdvezacpo7Dw78zzA0R26YOuP6UTrey9XGzNDO3G3MDO3N3UudaZl/DXwrM/86M/8R+BzwTmBtNU0DsAk4VjOjJKlPdYr7XwFXRMQbIiKAK4FngMeBa6ttdgAP1YsoSerXwMU9M/fReeP0q8DBqq89wK3Ab0TEYeDNwF0N5JQk9aHWZG9mfgT4yILmF4DL6/QrSarHT6hKUoFG7zSNFlnsA0CStNocuUtSgSzuklQgi7skFcjiLkkFsrhLUoEs7pJUIIu7JBXI4i5JBbK4S1KBLO6SVCCLuyQVyOIuSQWyuEtSgSzuklQgi7skFcjiLkkFsrhLUoEs7pJUIIu7JBWoVnGPiLUR8WBEfD0ino2Id0TEhRHxWEQ8X/1c11RYSdLy1B25fxz435n5NuCngWeB24C9mXkxsLe6L0laQQMX94hYA7wbuAsgM/8hM18GtgP3VpvdC1xTL6IkqV+RmYM9MOLtwB7gGTqj9gPALcCxzFxbbRPAS/P3Fzx+J7ATYHx8fOvMzMxAOebNzc0xNjZWq49+HTx2qnYf4xfAiVcaCNOwLRvX9Gyff84Lcy+2/ShZjddIE9qYu42ZoX25p6enD2TmZK91dYr7JPAE8M7M3BcRHwe+D3you5hHxEuZecZ598nJydy/f/9AOebNzs4yNTVVq49+Tdz2p7X72LXlNHcePLeBNM06svvqnu3zz3lh7sW2HyWr8RppQhtztzEztC93RCxa3OvMuR8Fjmbmvur+g8BlwImI2FDteANwssY+JEkDGHjImJkvRsR3IuKtmfkccCWdKZpngB3A7urnQ40k1Uhb7K+YNozopRLVnQ/4EPDZiDgPeAH4AJ2/Bh6IiJuAbwPvq7kPSVKfahX3zHwS6DXfc2WdfiVJ9fgJVUkqkMVdkgpkcZekAlncJalAFndJKpDFXZIKZHGXpAJZ3CWpQBZ3SSrQ6F2OUCOhiSteSlo9jtwlqUAWd0kqkMVdkgpkcZekAlncJalAFndJKpDFXZIKZHGXpAJZ3CWpQBZ3SSqQlx/QUC12GYMju69e4STS2aX2yD0izomIv4yIR6r7F0XEvog4HBH3R8R59WNKkvrRxLTMLcCzXfd/B/hYZv4E8BJwUwP7kCT1oVZxj4hNwNXAH1T3A/g54MFqk3uBa+rsQ5LUv8jMwR8c8SDwn4E3Ab8J3Ag8UY3aiYjNwOcz89Iej90J7AQYHx/fOjMzM3AOgLm5OcbGxmr10a+Dx07V7mP8AjjxSgNhVljd3Fs2rmkuzDKtxmukCW3M3cbM0L7c09PTBzJzste6gd9QjYhfAE5m5oGImOr38Zm5B9gDMDk5mVNTfXfxKrOzs9Tto183NnDN811bTnPnwfa9r10395EbppoLs0yr8RppQhtztzEztDd3L3WqyjuBX4yIq4DXA/8c+DiwNiLOzczTwCbgWP2YkqR+DDznnpm3Z+amzJwArgO+mJk3AI8D11ab7QAeqp1SktSXYXyI6VbgNyLiMPBm4K4h7EOSdAaNTPZm5iwwWy2/AFzeRL+SpMF4+QFJKpDFXZIK1L5z8FbBYtdHkaRRZXHXSPFCY1IznJaRpAJZ3CWpQBZ3SSqQxV2SCmRxl6QCWdwlqUAWd0kqkMVdkgpkcZekAlncJalAFndJKpDFXZIKZHGXpAJ5VUi1mleRlHpz5C5JBbK4S1KBnJaRKk7xqCQDj9wjYnNEPB4Rz0TE0xFxS9V+YUQ8FhHPVz/XNRdXkrQcdaZlTgO7MvMS4Arg5oi4BLgN2JuZFwN7q/uSpBU08LRMZh4HjlfLfxsRzwIbge3AVLXZvcAscGutlDrr+SXlUn8aeUM1IiaAnwH2AeNV4Qd4ERhvYh+SpOWLzKzXQcQY8H+AOzLzcxHxcmau7Vr/Uma+Zt49InYCOwHGx8e3zszM1MoxNzfH2NhYrT4Wc/DYqaH0CzB+AZx4ZWjdD03d3Fs2runZ3tSx7tX/Uq+Rxfa9WNaVMszX9rC0MTO0L/f09PSBzJzsta5WcY+I1wGPAI9m5u9Wbc8BU5l5PCI2ALOZ+dYz9TM5OZn79+8fOAfA7OwsU1NTtfpYzDCnBHZtOc2dB9t30lLd3IudgTLsY/2hG7Yvun5Uz5YZ5mt7WNqYGdqXOyIWLe51zpYJ4C7g2fnCXnkY2FEt7wAeGnQfkqTB1BkyvhP4ZeBgRDxZtf02sBt4ICJuAr4NvK9WQklS3+qcLfNlIBZZfeWg/UqS6vPyA5JUIIu7JBWofadpqAh+KEkaLkfuklSgs3Lk7qhRUukcuUtSgSzuklQgi7skFeisnHPX2cv3W3S2cOQuSQVy5C4tod/R/mpfRVICR+6SVCSLuyQVqOhpGd88k3S2Krq4S6NkVL/pSWVyWkaSCmRxl6QCOS0jrbKl3hvateU0N3Zt4zSOlsORuyQVqPUj9/lRz8LRjSSdzVpf3CU160zTRE4JtYfTMpJUoKGN3CNiG/Bx4BzgDzJz97D2JY2SYX94zmvdaDmGUtwj4hzgvwH/BjgK/EVEPJyZzwxjf5L65ye4V9ZKf4htWNMylwOHM/OFzPwHYAbYPqR9SZIWiMxsvtOIa4Ftmfnvqvu/DPxsZn6wa5udwM7q7luB52rudj3w3Zp9rAZzr5w2ZoZ25m5jZmhf7n+ZmW/ptWLVzpbJzD3Anqb6i4j9mTnZVH8rxdwrp42ZoZ2525gZ2pu7l2FNyxwDNnfd31S1SZJWwLCK+18AF0fERRFxHnAd8PCQ9iVJWmAo0zKZeToiPgg8SudUyLsz8+lh7KtLY1M8K8zcK6eNmaGduduYGdqb+zWG8oaqJGl1+QlVSSqQxV2SSpSZI3cDjgAHgSeB/VXbhcBjwPPVz3VVewCfAA4DTwGXdfWzo9r+eWBHV/vWqv/D1WNjgIx3AyeBQ11tQ8+42D5q5v4onbOZnqxuV3Wtu73K8Bzw813t26q2w8BtXe0XAfuq9vuB86r286v7h6v1E31k3gw8DjwDPA3cMurH+wyZR/1Yvx74CvC1Kvd/GnRfTT2fmrnvAb7VdbzfPiqvkWHfVj3AIv9QR4D1C9r+y/wLAbgN+J1q+Srg89U/1hXAvq4D/kL1c121PP/L/5Vq26ge+94BMr4buIxXF8mhZ1xsHzVzfxT4zR7bXlL9spxf/eJ9k84b5OdUyz8GnFdtc0n1mAeA66rlTwG/Wi3/GvCpavk64P4+Mm+Y/+UD3gR8o8o2ssf7DJlH/VgHMFYtv45Osb2i3301+Xxq5r4HuLbH9qv+Ghn2bdUDLPIPdYTXFvfngA1dvzjPVcufBq5fuB1wPfDprvZPV20bgK93tb9quz5zTvDqIjn0jIvto2buj9K74NwO3N51/1HgHdXt0YXbVS/67wLnVu0/3G7+sdXyudV2ff/FVD3+ITrXLWrF8V6QuTXHGngD8FXgZ/vdV5PPp2bue+hd3EfuNdL0bVTn3BP4QkQcqC5TADCemcer5ReB8Wp5I/CdrscerdrO1H60R3sTViLjYvuo64MR8VRE3B0R6wbM/Wbg5cw83SP3Dx9TrT9Vbd+XiJgAfobOyKwVx3tBZhjxYx0R50TEk3Sm7x6jM9Lud19NPp+Bcmfm/PG+ozreH4uI8xfmXma+1fidrGVUi/u7MvMy4L3AzRHx7u6V2fkvMlcl2TKtRMYG9/FJ4MeBtwPHgTsb6LNxETEG/DHw7zPz+93rRvV498g88sc6M/8pM99O55PllwNvW91Ey7Mwd0RcSuevgrcB/4rOVMutQ84wMrVpJIt7Zh6rfp4E/oTOC+xERGwAqH6erDZf7FIHZ2rf1KO9CSuRcbF9DCwzT1S/GP8P+H06x3uQ3H8DrI2Icxe0v6qvav2aavtliYjX0SmSn83Mz1XNI328e2Vuw7Gel5kv03lT+B0D7KvJ5zNo7m2ZeTw7/h74Hwx+vFf0d7IJI1fcI+KNEfGm+WXgPcAhOpcv2FFttoPOHCZV+/uj4wrgVPUn0qPAeyJiXfWn73vozOEdB74fEVdERADv7+qrrpXIuNg+Bjb/wqz8Ep3jPb+v6yLi/Ii4CLiYzptKPS8vUY1aHgeuXeQYzOe+Fvhitf1y8gVwF/BsZv5u16qRPd6LZW7BsX5LRKytli+g8z7BswPsq8nnM2jur3cV3QCu4dXHe2R/Jxux2pP+C2903kX/Gj86penDVfubgb10Tjf6c+DCqj3ofDHIN+mcpjTZ1de/pXPa0mHgA13tk3T+kb8J/B6Dvdl0H50/q/+RzvzbTSuRcbF91Mz9P6tcT9F5oW7o2v7DVYbn6DqriM7ZBt+o1n14wb/fV6rn80fA+VX766v7h6v1P9ZH5nfR+VP3KbpOIRzl432GzKN+rH8K+Msq3yHgPw66r6aeT83cX6yO9yHgM/zojJpVf40M++blBySpQCM3LSNJqs/iLkkFsrhLUoEs7pJUIIu7JBXI4i5JBbK4S1KB/j/vV6YlOvICDwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now let's load the trained model\n",
    "\n",
    "lin_model = joblib.load('linear_regression.joblib') \n",
    "\n",
    "# let's obtain the predictions\n",
    "pred = lin_model.predict(data)\n",
    "\n",
    "# let's plot the predicted sale prices\n",
    "pd.Series(np.exp(pred)).hist(bins=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "What shortcomings, inconvenience and problems did you find when scoring new data?\n",
    "\n",
    "# List of problems\n",
    "\n",
    "- re-wrote a lot of code ==> repetitive\n",
    "- hard coded a lot of parameters ==> if these change we need to re-write them again\n",
    "- engineered a lot of variables that we actually do not need for the model\n",
    "- additional variables present missing data, we do not know what to do with them\n",
    "\n",
    "We can minimize these hurdles by using Open-source. And we will see how in the next videos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "feml",
   "language": "python",
   "name": "feml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "583px",
    "left": "0px",
    "right": "1324px",
    "top": "107px",
    "width": "212px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}