feature_engineering_2019/01-baseline-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Feature Engineering Home Page](https://www.kaggle.com/learn/feature-engineering)**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this exercise, you will develop a baseline model for predicting if a customer will buy an app after clicking on an ad. With this baseline model, you'll be able to see how your feature engineering and selection efforts improve the model's performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip</th>\n",
       "      <th>app</th>\n",
       "      <th>device</th>\n",
       "      <th>os</th>\n",
       "      <th>channel</th>\n",
       "      <th>click_time</th>\n",
       "      <th>attributed_time</th>\n",
       "      <th>is_attributed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89489</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:13:23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>204158</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>21</td>\n",
       "      <td>2017-11-06 15:41:07</td>\n",
       "      <td>2017-11-07 08:17:19</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3437</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>459</td>\n",
       "      <td>2017-11-06 15:42:32</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>167543</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:56:17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>147509</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:57:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>71421</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>153</td>\n",
       "      <td>2017-11-06 16:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>76953</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 16:00:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>187909</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>477</td>\n",
       "      <td>2017-11-06 16:00:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>116779</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>150</td>\n",
       "      <td>2017-11-06 16:00:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>47857</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>205</td>\n",
       "      <td>2017-11-06 16:00:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       ip  app  device  os  channel          click_time      attributed_time  \\\n",
       "0   89489    3       1  13      379 2017-11-06 15:13:23                  NaN   \n",
       "1  204158   35       1  13       21 2017-11-06 15:41:07  2017-11-07 08:17:19   \n",
       "2    3437    6       1  13      459 2017-11-06 15:42:32                  NaN   \n",
       "3  167543    3       1  13      379 2017-11-06 15:56:17                  NaN   \n",
       "4  147509    3       1  13      379 2017-11-06 15:57:01                  NaN   \n",
       "5   71421   15       1  13      153 2017-11-06 16:00:00                  NaN   \n",
       "6   76953   14       1  13      379 2017-11-06 16:00:01                  NaN   \n",
       "7  187909    2       1  25      477 2017-11-06 16:00:01                  NaN   \n",
       "8  116779    1       1   8      150 2017-11-06 16:00:01                  NaN   \n",
       "9   47857    3       1  15      205 2017-11-06 16:00:01                  NaN   \n",
       "\n",
       "   is_attributed  \n",
       "0              0  \n",
       "1              1  \n",
       "2              0  \n",
       "3              0  \n",
       "4              0  \n",
       "5              0  \n",
       "6              0  \n",
       "7              0  \n",
       "8              0  \n",
       "9              0  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set up code checking\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.feature_engineering.ex1 import *\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "click_data = pd.read_csv('../input/feature-engineering-data/train_sample.csv',\n",
    "                         parse_dates=['click_time'])\n",
    "click_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline Model\n",
    "\n",
    "The first thing you need to do is construct a baseline model. All new features, processing, encodings, and feature selection should improve upon this baseline model. First you need to do a bit of feature engineering before training the model itself.\n",
    "\n",
    "### 1) Features from timestamps\n",
    "From the timestamps, create features for the day, hour, minute and second. Store these as new integer columns `day`, `hour`, `minute`, and `second` in a new DataFrame `clicks`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.25, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"1_TimestampFeatures\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip</th>\n",
       "      <th>app</th>\n",
       "      <th>device</th>\n",
       "      <th>os</th>\n",
       "      <th>channel</th>\n",
       "      <th>click_time</th>\n",
       "      <th>attributed_time</th>\n",
       "      <th>is_attributed</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>minute</th>\n",
       "      <th>second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89489</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:13:23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>204158</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>21</td>\n",
       "      <td>2017-11-06 15:41:07</td>\n",
       "      <td>2017-11-07 08:17:19</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3437</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>459</td>\n",
       "      <td>2017-11-06 15:42:32</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>42</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>167543</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:56:17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>56</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>147509</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:57:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300556</th>\n",
       "      <td>32457</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>477</td>\n",
       "      <td>2017-11-09 15:59:59</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>59</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300557</th>\n",
       "      <td>20266</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>446</td>\n",
       "      <td>2017-11-09 15:59:59</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>59</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300558</th>\n",
       "      <td>49383</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>178</td>\n",
       "      <td>2017-11-09 16:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300559</th>\n",
       "      <td>34894</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>145</td>\n",
       "      <td>2017-11-09 16:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300560</th>\n",
       "      <td>232000</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>134</td>\n",
       "      <td>2017-11-09 16:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2300561 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             ip  app  device  os  channel          click_time  \\\n",
       "0         89489    3       1  13      379 2017-11-06 15:13:23   \n",
       "1        204158   35       1  13       21 2017-11-06 15:41:07   \n",
       "2          3437    6       1  13      459 2017-11-06 15:42:32   \n",
       "3        167543    3       1  13      379 2017-11-06 15:56:17   \n",
       "4        147509    3       1  13      379 2017-11-06 15:57:01   \n",
       "...         ...  ...     ...  ..      ...                 ...   \n",
       "2300556   32457    2       1  19      477 2017-11-09 15:59:59   \n",
       "2300557   20266   14       1  13      446 2017-11-09 15:59:59   \n",
       "2300558   49383   12       2  17      178 2017-11-09 16:00:00   \n",
       "2300559   34894   12       1  15      145 2017-11-09 16:00:00   \n",
       "2300560  232000    9       1  19      134 2017-11-09 16:00:00   \n",
       "\n",
       "             attributed_time  is_attributed  day  hour  minute  second  \n",
       "0                        NaN              0    6    15      13      23  \n",
       "1        2017-11-07 08:17:19              1    6    15      41       7  \n",
       "2                        NaN              0    6    15      42      32  \n",
       "3                        NaN              0    6    15      56      17  \n",
       "4                        NaN              0    6    15      57       1  \n",
       "...                      ...            ...  ...   ...     ...     ...  \n",
       "2300556                  NaN              0    9    15      59      59  \n",
       "2300557                  NaN              0    9    15      59      59  \n",
       "2300558                  NaN              0    9    16       0       0  \n",
       "2300559                  NaN              0    9    16       0       0  \n",
       "2300560                  NaN              0    9    16       0       0  \n",
       "\n",
       "[2300561 rows x 12 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add new columns for timestamp features day, hour, minute, and second\n",
    "clicks = click_data.copy()\n",
    "clicks['day'] = clicks['click_time'].dt.day.astype('uint8')\n",
    "# Fill in the rest\n",
    "clicks['hour'] = clicks['click_time'].dt.hour.astype('uint8')\n",
    "clicks['minute'] = clicks['click_time'].dt.minute.astype('uint8')\n",
    "clicks['second'] = clicks['click_time'].dt.second.astype('uint8')\n",
    "\n",
    "q_1.check()\n",
    "clicks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment these if you need guidance\n",
    "#q_1.hint()\n",
    "#q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Label Encoding\n",
    "For each of the categorical features `['ip', 'app', 'device', 'os', 'channel']`, use scikit-learn's `LabelEncoder` to create new features in the `clicks` DataFrame. The new column names should be the original column name with `'_labels'` appended, like `ip_labels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.25, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"2_LabelEncoding\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "cat_features = ['ip', 'app', 'device', 'os', 'channel']\n",
    "\n",
    "encoder = preprocessing.LabelEncoder()\n",
    "# Create new columns in clicks using preprocessing.LabelEncoder()\n",
    "for feature in cat_features:\n",
    "    clicks[f'{feature}_labels'] = encoder.fit_transform(clicks[feature])\n",
    "\n",
    "q_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment these if you need guidance\n",
    "# q_2.hint()\n",
    "# q_2.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip</th>\n",
       "      <th>app</th>\n",
       "      <th>device</th>\n",
       "      <th>os</th>\n",
       "      <th>channel</th>\n",
       "      <th>click_time</th>\n",
       "      <th>attributed_time</th>\n",
       "      <th>is_attributed</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>minute</th>\n",
       "      <th>second</th>\n",
       "      <th>ip_labels</th>\n",
       "      <th>app_labels</th>\n",
       "      <th>device_labels</th>\n",
       "      <th>os_labels</th>\n",
       "      <th>channel_labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>89489</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:13:23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>23</td>\n",
       "      <td>27226</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>204158</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>21</td>\n",
       "      <td>2017-11-06 15:41:07</td>\n",
       "      <td>2017-11-07 08:17:19</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "      <td>7</td>\n",
       "      <td>110007</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3437</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>459</td>\n",
       "      <td>2017-11-06 15:42:32</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>42</td>\n",
       "      <td>32</td>\n",
       "      <td>1047</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>167543</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:56:17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>56</td>\n",
       "      <td>17</td>\n",
       "      <td>76270</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>147509</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>379</td>\n",
       "      <td>2017-11-06 15:57:01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>57862</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       ip  app  device  os  channel          click_time      attributed_time  \\\n",
       "0   89489    3       1  13      379 2017-11-06 15:13:23                  NaN   \n",
       "1  204158   35       1  13       21 2017-11-06 15:41:07  2017-11-07 08:17:19   \n",
       "2    3437    6       1  13      459 2017-11-06 15:42:32                  NaN   \n",
       "3  167543    3       1  13      379 2017-11-06 15:56:17                  NaN   \n",
       "4  147509    3       1  13      379 2017-11-06 15:57:01                  NaN   \n",
       "\n",
       "   is_attributed  day  hour  minute  second  ip_labels  app_labels  \\\n",
       "0              0    6    15      13      23      27226           3   \n",
       "1              1    6    15      41       7     110007          35   \n",
       "2              0    6    15      42      32       1047           6   \n",
       "3              0    6    15      56      17      76270           3   \n",
       "4              0    6    15      57       1      57862           3   \n",
       "\n",
       "   device_labels  os_labels  channel_labels  \n",
       "0              1         13             120  \n",
       "1              1         13              10  \n",
       "2              1         13             157  \n",
       "3              1         13             120  \n",
       "4              1         13             120  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clicks.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) One-hot Encoding\n",
    "\n",
    "Now you have label encoded features, does it make sense to use one-hot encoding for the categorical variables ip, app, device, os, or channel?\n",
    "\n",
    "Uncomment the following line after you've decided your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"3_OnehotEncoding\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "    The `ip` column has 58,000 values, which means it will create an extremely \n",
       "    sparse matrix with 58,000 columns. This many columns will make your model run\n",
       "    very slow, so in general you want to avoid one-hot encoding features with many\n",
       "    levels. LightGBM models work with label encoded features, so you don't actually need to \n",
       "    one-hot encode the categorical features.\n",
       "\n",
       "    "
      ],
      "text/plain": [
       "Solution: \n",
       "    The `ip` column has 58,000 values, which means it will create an extremely \n",
       "    sparse matrix with 58,000 columns. This many columns will make your model run\n",
       "    very slow, so in general you want to avoid one-hot encoding features with many\n",
       "    levels. LightGBM models work with label encoded features, so you don't actually need to \n",
       "    one-hot encode the categorical features.\n",
       "\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q_3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train, validation, and test sets\n",
    "With our baseline features ready, we need to split our data into training and validation sets. We should also hold out a test set to measure the final accuracy of the model.\n",
    "\n",
    "### 4) Train/test splits with time series data\n",
    "This is time series data. Are they any special considerations when creating train/test splits for time series? If so, what and why?\n",
    "\n",
    "Uncomment the following line after you've decided your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"4_TrainTestSplits\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "    Since our model is meant to predict events in the future, we must also validate the\n",
       "    model on events in the future. If the data is mixed up between the training and test \n",
       "    sets, then future data will leak in to the model and our validation results will \n",
       "    overestimate the performance on new data.\n",
       "    "
      ],
      "text/plain": [
       "Solution: \n",
       "    Since our model is meant to predict events in the future, we must also validate the\n",
       "    model on events in the future. If the data is mixed up between the training and test \n",
       "    sets, then future data will leak in to the model and our validation results will \n",
       "    overestimate the performance on new data.\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q_4.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create train/validation/test splits\n",
    "\n",
    "Here we'll create training, validation, and test splits. First, `clicks` DataFrame is sorted in order of increasing time. The first 80% of the rows are the train set, the next 10% are the validation set, and the last 10% are the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = ['day', 'hour', 'minute', 'second', \n",
    "                'ip_labels', 'app_labels', 'device_labels',\n",
    "                'os_labels', 'channel_labels']\n",
    "\n",
    "valid_fraction = 0.1\n",
    "clicks_srt = clicks.sort_values('click_time')\n",
    "valid_rows = int(len(clicks_srt) * valid_fraction)\n",
    "train = clicks_srt[:-valid_rows * 2]\n",
    "# valid size == test size, last two sections of the data\n",
    "valid = clicks_srt[-valid_rows * 2:-valid_rows]\n",
    "test = clicks_srt[-valid_rows:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train with LightGBM\n",
    "\n",
    "Now we can create LightGBM dataset objects for each of the smaller datasets and train the baseline model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tvalid_0's auc: 0.948979\n",
      "Training until validation scores don't improve for 10 rounds\n",
      "[2]\tvalid_0's auc: 0.949235\n",
      "[3]\tvalid_0's auc: 0.950126\n",
      "[4]\tvalid_0's auc: 0.950072\n",
      "[5]\tvalid_0's auc: 0.950536\n",
      "[6]\tvalid_0's auc: 0.950943\n",
      "[7]\tvalid_0's auc: 0.951453\n",
      "[8]\tvalid_0's auc: 0.951518\n",
      "[9]\tvalid_0's auc: 0.952385\n",
      "[10]\tvalid_0's auc: 0.952434\n",
      "[11]\tvalid_0's auc: 0.952465\n",
      "[12]\tvalid_0's auc: 0.952638\n",
      "[13]\tvalid_0's auc: 0.95266\n",
      "[14]\tvalid_0's auc: 0.952766\n",
      "[15]\tvalid_0's auc: 0.953203\n",
      "[16]\tvalid_0's auc: 0.953503\n",
      "[17]\tvalid_0's auc: 0.953793\n",
      "[18]\tvalid_0's auc: 0.953966\n",
      "[19]\tvalid_0's auc: 0.954184\n",
      "[20]\tvalid_0's auc: 0.9543\n",
      "[21]\tvalid_0's auc: 0.954305\n",
      "[22]\tvalid_0's auc: 0.954536\n",
      "[23]\tvalid_0's auc: 0.954748\n",
      "[24]\tvalid_0's auc: 0.955142\n",
      "[25]\tvalid_0's auc: 0.955493\n",
      "[26]\tvalid_0's auc: 0.955611\n",
      "[27]\tvalid_0's auc: 0.955708\n",
      "[28]\tvalid_0's auc: 0.955795\n",
      "[29]\tvalid_0's auc: 0.956172\n",
      "[30]\tvalid_0's auc: 0.95623\n",
      "[31]\tvalid_0's auc: 0.956477\n",
      "[32]\tvalid_0's auc: 0.956606\n",
      "[33]\tvalid_0's auc: 0.956864\n",
      "[34]\tvalid_0's auc: 0.957204\n",
      "[35]\tvalid_0's auc: 0.957327\n",
      "[36]\tvalid_0's auc: 0.957408\n",
      "[37]\tvalid_0's auc: 0.957524\n",
      "[38]\tvalid_0's auc: 0.957659\n",
      "[39]\tvalid_0's auc: 0.957846\n",
      "[40]\tvalid_0's auc: 0.958042\n",
      "[41]\tvalid_0's auc: 0.958146\n",
      "[42]\tvalid_0's auc: 0.958181\n",
      "[43]\tvalid_0's auc: 0.958285\n",
      "[44]\tvalid_0's auc: 0.958433\n",
      "[45]\tvalid_0's auc: 0.95854\n",
      "[46]\tvalid_0's auc: 0.958625\n",
      "[47]\tvalid_0's auc: 0.958756\n",
      "[48]\tvalid_0's auc: 0.958863\n",
      "[49]\tvalid_0's auc: 0.958938\n",
      "[50]\tvalid_0's auc: 0.959046\n",
      "[51]\tvalid_0's auc: 0.95908\n",
      "[52]\tvalid_0's auc: 0.959147\n",
      "[53]\tvalid_0's auc: 0.9592\n",
      "[54]\tvalid_0's auc: 0.959259\n",
      "[55]\tvalid_0's auc: 0.959311\n",
      "[56]\tvalid_0's auc: 0.959324\n",
      "[57]\tvalid_0's auc: 0.959348\n",
      "[58]\tvalid_0's auc: 0.959435\n",
      "[59]\tvalid_0's auc: 0.959463\n",
      "[60]\tvalid_0's auc: 0.95949\n",
      "[61]\tvalid_0's auc: 0.959562\n",
      "[62]\tvalid_0's auc: 0.959721\n",
      "[63]\tvalid_0's auc: 0.959729\n",
      "[64]\tvalid_0's auc: 0.959773\n",
      "[65]\tvalid_0's auc: 0.959809\n",
      "[66]\tvalid_0's auc: 0.959868\n",
      "[67]\tvalid_0's auc: 0.959921\n",
      "[68]\tvalid_0's auc: 0.959994\n",
      "[69]\tvalid_0's auc: 0.960065\n",
      "[70]\tvalid_0's auc: 0.96011\n",
      "[71]\tvalid_0's auc: 0.960133\n",
      "[72]\tvalid_0's auc: 0.960275\n",
      "[73]\tvalid_0's auc: 0.960299\n",
      "[74]\tvalid_0's auc: 0.960336\n",
      "[75]\tvalid_0's auc: 0.960365\n",
      "[76]\tvalid_0's auc: 0.960411\n",
      "[77]\tvalid_0's auc: 0.960488\n",
      "[78]\tvalid_0's auc: 0.960523\n",
      "[79]\tvalid_0's auc: 0.960563\n",
      "[80]\tvalid_0's auc: 0.960624\n",
      "[81]\tvalid_0's auc: 0.960665\n",
      "[82]\tvalid_0's auc: 0.960724\n",
      "[83]\tvalid_0's auc: 0.960724\n",
      "[84]\tvalid_0's auc: 0.960751\n",
      "[85]\tvalid_0's auc: 0.960799\n",
      "[86]\tvalid_0's auc: 0.960853\n",
      "[87]\tvalid_0's auc: 0.960876\n",
      "[88]\tvalid_0's auc: 0.960934\n",
      "[89]\tvalid_0's auc: 0.961012\n",
      "[90]\tvalid_0's auc: 0.961012\n",
      "[91]\tvalid_0's auc: 0.961065\n",
      "[92]\tvalid_0's auc: 0.961095\n",
      "[93]\tvalid_0's auc: 0.961131\n",
      "[94]\tvalid_0's auc: 0.961136\n",
      "[95]\tvalid_0's auc: 0.961155\n",
      "[96]\tvalid_0's auc: 0.961191\n",
      "[97]\tvalid_0's auc: 0.961189\n",
      "[98]\tvalid_0's auc: 0.961189\n",
      "[99]\tvalid_0's auc: 0.961224\n",
      "[100]\tvalid_0's auc: 0.961228\n",
      "[101]\tvalid_0's auc: 0.96125\n",
      "[102]\tvalid_0's auc: 0.961259\n",
      "[103]\tvalid_0's auc: 0.961289\n",
      "[104]\tvalid_0's auc: 0.961309\n",
      "[105]\tvalid_0's auc: 0.961309\n",
      "[106]\tvalid_0's auc: 0.96134\n",
      "[107]\tvalid_0's auc: 0.961373\n",
      "[108]\tvalid_0's auc: 0.961382\n",
      "[109]\tvalid_0's auc: 0.961391\n",
      "[110]\tvalid_0's auc: 0.961402\n",
      "[111]\tvalid_0's auc: 0.961449\n",
      "[112]\tvalid_0's auc: 0.96145\n",
      "[113]\tvalid_0's auc: 0.961482\n",
      "[114]\tvalid_0's auc: 0.961481\n",
      "[115]\tvalid_0's auc: 0.961492\n",
      "[116]\tvalid_0's auc: 0.961513\n",
      "[117]\tvalid_0's auc: 0.961531\n",
      "[118]\tvalid_0's auc: 0.961539\n",
      "[119]\tvalid_0's auc: 0.961563\n",
      "[120]\tvalid_0's auc: 0.961563\n",
      "[121]\tvalid_0's auc: 0.961568\n",
      "[122]\tvalid_0's auc: 0.961588\n",
      "[123]\tvalid_0's auc: 0.961599\n",
      "[124]\tvalid_0's auc: 0.961605\n",
      "[125]\tvalid_0's auc: 0.961605\n",
      "[126]\tvalid_0's auc: 0.96161\n",
      "[127]\tvalid_0's auc: 0.961626\n",
      "[128]\tvalid_0's auc: 0.961626\n",
      "[129]\tvalid_0's auc: 0.96163\n",
      "[130]\tvalid_0's auc: 0.961646\n",
      "[131]\tvalid_0's auc: 0.961678\n",
      "[132]\tvalid_0's auc: 0.961672\n",
      "[133]\tvalid_0's auc: 0.961673\n",
      "[134]\tvalid_0's auc: 0.96171\n",
      "[135]\tvalid_0's auc: 0.96171\n",
      "[136]\tvalid_0's auc: 0.961724\n",
      "[137]\tvalid_0's auc: 0.961723\n",
      "[138]\tvalid_0's auc: 0.961726\n",
      "[139]\tvalid_0's auc: 0.961731\n",
      "[140]\tvalid_0's auc: 0.961736\n",
      "[141]\tvalid_0's auc: 0.961751\n",
      "[142]\tvalid_0's auc: 0.961759\n",
      "[143]\tvalid_0's auc: 0.961777\n",
      "[144]\tvalid_0's auc: 0.961777\n",
      "[145]\tvalid_0's auc: 0.961779\n",
      "[146]\tvalid_0's auc: 0.961782\n",
      "[147]\tvalid_0's auc: 0.961782\n",
      "[148]\tvalid_0's auc: 0.961796\n",
      "[149]\tvalid_0's auc: 0.961799\n",
      "[150]\tvalid_0's auc: 0.961806\n",
      "[151]\tvalid_0's auc: 0.961804\n",
      "[152]\tvalid_0's auc: 0.961805\n",
      "[153]\tvalid_0's auc: 0.961794\n",
      "[154]\tvalid_0's auc: 0.961802\n",
      "[155]\tvalid_0's auc: 0.961805\n",
      "[156]\tvalid_0's auc: 0.961821\n",
      "[157]\tvalid_0's auc: 0.961853\n",
      "[158]\tvalid_0's auc: 0.96187\n",
      "[159]\tvalid_0's auc: 0.961875\n",
      "[160]\tvalid_0's auc: 0.961877\n",
      "[161]\tvalid_0's auc: 0.961889\n",
      "[162]\tvalid_0's auc: 0.961894\n",
      "[163]\tvalid_0's auc: 0.961898\n",
      "[164]\tvalid_0's auc: 0.961901\n",
      "[165]\tvalid_0's auc: 0.961911\n",
      "[166]\tvalid_0's auc: 0.961911\n",
      "[167]\tvalid_0's auc: 0.961915\n",
      "[168]\tvalid_0's auc: 0.961925\n",
      "[169]\tvalid_0's auc: 0.961925\n",
      "[170]\tvalid_0's auc: 0.961929\n",
      "[171]\tvalid_0's auc: 0.961949\n",
      "[172]\tvalid_0's auc: 0.961945\n",
      "[173]\tvalid_0's auc: 0.961945\n",
      "[174]\tvalid_0's auc: 0.961944\n",
      "[175]\tvalid_0's auc: 0.961946\n",
      "[176]\tvalid_0's auc: 0.961952\n",
      "[177]\tvalid_0's auc: 0.961956\n",
      "[178]\tvalid_0's auc: 0.961958\n",
      "[179]\tvalid_0's auc: 0.961971\n",
      "[180]\tvalid_0's auc: 0.961998\n",
      "[181]\tvalid_0's auc: 0.961998\n",
      "[182]\tvalid_0's auc: 0.962014\n",
      "[183]\tvalid_0's auc: 0.962018\n",
      "[184]\tvalid_0's auc: 0.962016\n",
      "[185]\tvalid_0's auc: 0.962022\n",
      "[186]\tvalid_0's auc: 0.962031\n",
      "[187]\tvalid_0's auc: 0.96203\n",
      "[188]\tvalid_0's auc: 0.962021\n",
      "[189]\tvalid_0's auc: 0.962021\n",
      "[190]\tvalid_0's auc: 0.962022\n",
      "[191]\tvalid_0's auc: 0.962026\n",
      "[192]\tvalid_0's auc: 0.962038\n",
      "[193]\tvalid_0's auc: 0.962042\n",
      "[194]\tvalid_0's auc: 0.962041\n",
      "[195]\tvalid_0's auc: 0.962035\n",
      "[196]\tvalid_0's auc: 0.962037\n",
      "[197]\tvalid_0's auc: 0.962048\n",
      "[198]\tvalid_0's auc: 0.962054\n",
      "[199]\tvalid_0's auc: 0.962052\n",
      "[200]\tvalid_0's auc: 0.962054\n",
      "[201]\tvalid_0's auc: 0.962041\n",
      "[202]\tvalid_0's auc: 0.962041\n",
      "[203]\tvalid_0's auc: 0.962052\n",
      "[204]\tvalid_0's auc: 0.962051\n",
      "[205]\tvalid_0's auc: 0.962056\n",
      "[206]\tvalid_0's auc: 0.962056\n",
      "[207]\tvalid_0's auc: 0.962069\n",
      "[208]\tvalid_0's auc: 0.962072\n",
      "[209]\tvalid_0's auc: 0.962072\n",
      "[210]\tvalid_0's auc: 0.962062\n",
      "[211]\tvalid_0's auc: 0.962064\n",
      "[212]\tvalid_0's auc: 0.962066\n",
      "[213]\tvalid_0's auc: 0.962066\n",
      "[214]\tvalid_0's auc: 0.962066\n",
      "[215]\tvalid_0's auc: 0.962064\n",
      "[216]\tvalid_0's auc: 0.96206\n",
      "[217]\tvalid_0's auc: 0.962059\n",
      "[218]\tvalid_0's auc: 0.962059\n",
      "Early stopping, best iteration is:\n",
      "[208]\tvalid_0's auc: 0.962072\n"
     ]
    }
   ],
   "source": [
    "import lightgbm as lgb\n",
    "\n",
    "dtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])\n",
    "dvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])\n",
    "dtest = lgb.Dataset(test[feature_cols], label=test['is_attributed'])\n",
    "\n",
    "param = {'num_leaves': 64, 'objective': 'binary'}\n",
    "param['metric'] = 'auc'\n",
    "num_round = 1000\n",
    "bst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], early_stopping_rounds=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model\n",
    "Finally, with the model trained, I'll evaluate it's performance on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.9726727334566094\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "ypred = bst.predict(test[feature_cols])\n",
    "score = metrics.roc_auc_score(test['is_attributed'], ypred)\n",
    "print(f\"Test score: {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will be our baseline score for the model. When we transform features, add new ones, or perform feature selection, we should be improving on this score. However, since this is the test set, we only want to look at it at the end of all our manipulations. At the very end of this course you'll look at the test score again to see if you improved on the baseline model.\n",
    "\n",
    "# Keep Going\n",
    "Now that you have a baseline model, you are ready to learn **[Categorical Encoding Techniques](https://www.kaggle.com/matleonard/categorical-encodings)** to improve it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Feature Engineering Home Page](https://www.kaggle.com/learn/feature-engineering)**\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}