feature_engineering_2019/02-categorical-encodings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Feature Engineering Home Page](https://www.kaggle.com/learn/feature-engineering)**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this exercise you'll apply more advanced encodings to encode the categorical variables ito improve your classifier model. The encodings you will implement are:\n",
    "\n",
    "- Count Encoding\n",
    "- Target Encoding\n",
    "- Leave-one-out Encoding\n",
    "- CatBoost Encoding\n",
    "- Feature embedding with SVD \n",
    "\n",
    "You'll refit the classifier after each encoding to check its performance on hold-out data. First, run the next cell to repeat the work you did in the last exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import preprocessing, metrics\n",
    "import lightgbm as lgb\n",
    "\n",
    "# Set up code checking\n",
    "# This can take a few seconds, thanks for your patience\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.feature_engineering.ex2 import *\n",
    "\n",
    "clicks = pd.read_parquet('../input/feature-engineering-data/baseline_data.pqt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I'll define a couple functions to help test the new encodings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data_splits(dataframe, valid_fraction=0.1):\n",
    "    \"\"\" Splits a dataframe into train, validation, and test sets. First, orders by \n",
    "        the column 'click_time'. Set the size of the validation and test sets with\n",
    "        the valid_fraction keyword argument.\n",
    "    \"\"\"\n",
    "\n",
    "    dataframe = dataframe.sort_values('click_time')\n",
    "    valid_rows = int(len(dataframe) * valid_fraction)\n",
    "    train = dataframe[:-valid_rows * 2]\n",
    "    # valid size == test size, last two sections of the data\n",
    "    valid = dataframe[-valid_rows * 2:-valid_rows]\n",
    "    test = dataframe[-valid_rows:]\n",
    "    \n",
    "    return train, valid, test\n",
    "\n",
    "def train_model(train, valid, test=None, feature_cols=None):\n",
    "    if feature_cols is None:\n",
    "        feature_cols = train.columns.drop(['click_time', 'attributed_time',\n",
    "                                           'is_attributed'])\n",
    "    dtrain = lgb.Dataset(train[feature_cols], label=train['is_attributed'])\n",
    "    dvalid = lgb.Dataset(valid[feature_cols], label=valid['is_attributed'])\n",
    "    \n",
    "    param = {'num_leaves': 64, 'objective': 'binary', \n",
    "             'metric': 'auc', 'seed': 7}\n",
    "    num_round = 1000\n",
    "    print(\"Training model!\")\n",
    "    bst = lgb.train(param, dtrain, num_round, valid_sets=[dvalid], \n",
    "                    early_stopping_rounds=20, verbose_eval=False)\n",
    "    \n",
    "    valid_pred = bst.predict(valid[feature_cols])\n",
    "    valid_score = metrics.roc_auc_score(valid['is_attributed'], valid_pred)\n",
    "    print(f\"Validation AUC score: {valid_score}\")\n",
    "    \n",
    "    if test is not None: \n",
    "        test_pred = bst.predict(test[feature_cols])\n",
    "        test_score = metrics.roc_auc_score(test['is_attributed'], test_pred)\n",
    "        return bst, valid_score, test_score\n",
    "    else:\n",
    "        return bst, valid_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run this cell to get a baseline score. If your encodings do better than this, you can keep them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline model\n",
      "Training model!\n",
      "Validation AUC score: 0.9622743228943659\n"
     ]
    }
   ],
   "source": [
    "print(\"Baseline model\")\n",
    "train, valid, test = get_data_splits(clicks)\n",
    "_ = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Categorical encodings and leakage\n",
    "\n",
    "These encodings are all based on statistics calculated from the dataset like counts and means. Considering this, what data should you be using to calculate the encodings?\n",
    "\n",
    "Uncomment the following line after you've decided your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"1_LeakageQuestion\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> You should calculate the encodings from the training set only. If you include data from the validation and test sets into the encodings, you'll overestimate the model's performance. You should in general be vigilant to avoid leakage, that is, including any information from the validation and test sets into the model. For a review on this topic, see our lesson on [data leakage](https://www.kaggle.com/alexisbcook/data-leakage)"
      ],
      "text/plain": [
       "Solution: You should calculate the encodings from the training set only. If you include data from the validation and test sets into the encodings, you'll overestimate the model's performance. You should in general be vigilant to avoid leakage, that is, including any information from the validation and test sets into the model. For a review on this topic, see our lesson on [data leakage](https://www.kaggle.com/alexisbcook/data-leakage)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Count encodings\n",
    "\n",
    "Here, encode the categorical features `['ip', 'app', 'device', 'os', 'channel']` using the count of each value in the data set. Using `CountEncoder` from the `category_encoders` library, fit the encoding using the categorical feature columns defined in `cat_features`. Then apply the encodings to the train and validation sets, adding them as new columns with names suffixed `\"_count\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.125, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"2_CountEncodings\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip</th>\n",
       "      <th>app</th>\n",
       "      <th>device</th>\n",
       "      <th>os</th>\n",
       "      <th>channel</th>\n",
       "      <th>click_time</th>\n",
       "      <th>attributed_time</th>\n",
       "      <th>is_attributed</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>minute</th>\n",
       "      <th>second</th>\n",
       "      <th>ip_count</th>\n",
       "      <th>app_count</th>\n",
       "      <th>device_count</th>\n",
       "      <th>os_count</th>\n",
       "      <th>channel_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27226</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:13:23</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>23</td>\n",
       "      <td>68</td>\n",
       "      <td>292254</td>\n",
       "      <td>1648091</td>\n",
       "      <td>370652</td>\n",
       "      <td>26760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110007</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "      <td>2017-11-06 15:41:07</td>\n",
       "      <td>2017-11-07 08:17:19</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>60114</td>\n",
       "      <td>1648091</td>\n",
       "      <td>370652</td>\n",
       "      <td>41256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1047</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>157</td>\n",
       "      <td>2017-11-06 15:42:32</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>42</td>\n",
       "      <td>32</td>\n",
       "      <td>118</td>\n",
       "      <td>19564</td>\n",
       "      <td>1648091</td>\n",
       "      <td>370652</td>\n",
       "      <td>31221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>76270</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:56:17</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>56</td>\n",
       "      <td>17</td>\n",
       "      <td>29</td>\n",
       "      <td>292254</td>\n",
       "      <td>1648091</td>\n",
       "      <td>370652</td>\n",
       "      <td>26760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57862</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:57:01</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>292254</td>\n",
       "      <td>1648091</td>\n",
       "      <td>370652</td>\n",
       "      <td>26760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840445</th>\n",
       "      <td>11718</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>26</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>28</td>\n",
       "      <td>129507</td>\n",
       "      <td>1648091</td>\n",
       "      <td>87078</td>\n",
       "      <td>39586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840449</th>\n",
       "      <td>32849</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>75</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>51</td>\n",
       "      <td>140369</td>\n",
       "      <td>1648091</td>\n",
       "      <td>420998</td>\n",
       "      <td>84548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840446</th>\n",
       "      <td>249422</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>103</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:53:15</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>845</td>\n",
       "      <td>1648091</td>\n",
       "      <td>420998</td>\n",
       "      <td>7403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840443</th>\n",
       "      <td>232256</td>\n",
       "      <td>19</td>\n",
       "      <td>6</td>\n",
       "      <td>21</td>\n",
       "      <td>59</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:50:59</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>104701</td>\n",
       "      <td>492</td>\n",
       "      <td>15303</td>\n",
       "      <td>107747</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840442</th>\n",
       "      <td>227929</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>20</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:50:48</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>1</td>\n",
       "      <td>33072</td>\n",
       "      <td>1648091</td>\n",
       "      <td>15804</td>\n",
       "      <td>45870</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1840449 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             ip  app  device  os  channel          click_time  \\\n",
       "0         27226    3       1  13      120 2017-11-06 15:13:23   \n",
       "1        110007   35       1  13       10 2017-11-06 15:41:07   \n",
       "2          1047    6       1  13      157 2017-11-06 15:42:32   \n",
       "3         76270    3       1  13      120 2017-11-06 15:56:17   \n",
       "4         57862    3       1  13      120 2017-11-06 15:57:01   \n",
       "...         ...  ...     ...  ..      ...                 ...   \n",
       "1840445   11718   18       1  17       26 2017-11-09 04:50:14   \n",
       "1840449   32849   15       1  19       75 2017-11-09 04:50:14   \n",
       "1840446  249422   48       1  19      103 2017-11-09 04:50:14   \n",
       "1840443  232256   19       6  21       59 2017-11-09 04:50:14   \n",
       "1840442  227929   10       1  46       20 2017-11-09 04:50:14   \n",
       "\n",
       "             attributed_time  is_attributed  day  hour  minute  second  \\\n",
       "0                       None              0    6    15      13      23   \n",
       "1        2017-11-07 08:17:19              1    6    15      41       7   \n",
       "2                       None              0    6    15      42      32   \n",
       "3                       None              0    6    15      56      17   \n",
       "4                       None              0    6    15      57       1   \n",
       "...                      ...            ...  ...   ...     ...     ...   \n",
       "1840445                 None              0    9     4      50      14   \n",
       "1840449                 None              0    9     4      50      14   \n",
       "1840446  2017-11-09 04:53:15              1    9     4      50      14   \n",
       "1840443  2017-11-09 04:50:59              1    9     4      50      14   \n",
       "1840442  2017-11-09 04:50:48              1    9     4      50      14   \n",
       "\n",
       "         ip_count  app_count  device_count  os_count  channel_count  \n",
       "0              68     292254       1648091    370652          26760  \n",
       "1               4      60114       1648091    370652          41256  \n",
       "2             118      19564       1648091    370652          31221  \n",
       "3              29     292254       1648091    370652          26760  \n",
       "4              31     292254       1648091    370652          26760  \n",
       "...           ...        ...           ...       ...            ...  \n",
       "1840445        28     129507       1648091     87078          39586  \n",
       "1840449        51     140369       1648091    420998          84548  \n",
       "1840446         1        845       1648091    420998           7403  \n",
       "1840443         1     104701           492     15303         107747  \n",
       "1840442         1      33072       1648091     15804          45870  \n",
       "\n",
       "[1840449 rows x 17 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import category_encoders as ce\n",
    "\n",
    "cat_features = ['ip', 'app', 'device', 'os', 'channel']\n",
    "train, valid, test = get_data_splits(clicks)\n",
    "\n",
    "# Create the count encoder\n",
    "count_enc = ce.CountEncoder(cols=cat_features)\n",
    "\n",
    "# Learn encoding from the training set\n",
    "count_enc.fit(train[cat_features])\n",
    "\n",
    "# Apply encoding to the train and validation sets as new columns\n",
    "# Make sure to add `_count` as a suffix to the new columns\n",
    "train_encoded = train.join(count_enc.transform(train[cat_features]).add_suffix(\"_count\"))\n",
    "valid_encoded = valid.join(count_enc.transform(valid[cat_features]).add_suffix(\"_count\"))\n",
    "\n",
    "q_2.check()\n",
    "train_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment if you need some guidance\n",
    "#q_2.hint()\n",
    "#q_2.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.9653051135205329\n"
     ]
    }
   ],
   "source": [
    "# Train the model on the encoded datasets\n",
    "# This can take around 30 seconds to complete\n",
    "_ = train_model(train_encoded, valid_encoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Count encoding improved our model's score!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Why is count encoding effective?\n",
    "At first glance, it could be surprising that Count Encoding helps make accurate models. \n",
    "Why do you think is count encoding is a good idea, or how does it improve the model score?\n",
    "\n",
    "Uncomment the following line after you've decided your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"3_CountEncodingEffectiveness\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "    Rare values tend to have similar counts (with values like 1 or 2), so you can classify rare \n",
       "    values together at prediction time. Common values with large counts are unlikely to have \n",
       "    the same exact count as other values. So, the common/important values get their own \n",
       "    grouping.\n",
       "    "
      ],
      "text/plain": [
       "Solution: \n",
       "    Rare values tend to have similar counts (with values like 1 or 2), so you can classify rare \n",
       "    values together at prediction time. Common values with large counts are unlikely to have \n",
       "    the same exact count as other values. So, the common/important values get their own \n",
       "    grouping.\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q_3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Target encoding\n",
    "\n",
    "Here you'll try some supervised encodings that use the labels (the targets) to transform categorical features. The first one is target encoding. Create the target encoder from the `category_encoders` library. Then, learn the encodings from the training dataset, apply the encodings to all the datasets and retrain the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.125, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"4_TargetEncodings\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip</th>\n",
       "      <th>app</th>\n",
       "      <th>device</th>\n",
       "      <th>os</th>\n",
       "      <th>channel</th>\n",
       "      <th>click_time</th>\n",
       "      <th>attributed_time</th>\n",
       "      <th>is_attributed</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>minute</th>\n",
       "      <th>second</th>\n",
       "      <th>ip_target</th>\n",
       "      <th>app_target</th>\n",
       "      <th>device_target</th>\n",
       "      <th>os_target</th>\n",
       "      <th>channel_target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27226</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:13:23</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>23</td>\n",
       "      <td>0.073529</td>\n",
       "      <td>0.028328</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.034043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110007</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "      <td>2017-11-06 15:41:07</td>\n",
       "      <td>2017-11-07 08:17:19</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "      <td>7</td>\n",
       "      <td>0.247523</td>\n",
       "      <td>0.995841</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.950262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1047</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>157</td>\n",
       "      <td>2017-11-06 15:42:32</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>42</td>\n",
       "      <td>32</td>\n",
       "      <td>0.135593</td>\n",
       "      <td>0.009252</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.019378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>76270</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:56:17</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>56</td>\n",
       "      <td>17</td>\n",
       "      <td>0.103448</td>\n",
       "      <td>0.028328</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.034043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57862</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:57:01</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>0.028328</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.034043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840445</th>\n",
       "      <td>11718</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>26</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.071429</td>\n",
       "      <td>0.048499</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.109913</td>\n",
       "      <td>0.043930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840449</th>\n",
       "      <td>32849</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>75</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.098039</td>\n",
       "      <td>0.021686</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.157243</td>\n",
       "      <td>0.009202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840446</th>\n",
       "      <td>249422</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>103</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:53:15</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.197764</td>\n",
       "      <td>0.951479</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.157243</td>\n",
       "      <td>0.970012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840443</th>\n",
       "      <td>232256</td>\n",
       "      <td>19</td>\n",
       "      <td>6</td>\n",
       "      <td>21</td>\n",
       "      <td>59</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:50:59</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.197764</td>\n",
       "      <td>0.948415</td>\n",
       "      <td>0.953252</td>\n",
       "      <td>0.953408</td>\n",
       "      <td>0.953326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840442</th>\n",
       "      <td>227929</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>20</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:50:48</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.197764</td>\n",
       "      <td>0.839683</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.283852</td>\n",
       "      <td>0.949270</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1840449 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             ip  app  device  os  channel          click_time  \\\n",
       "0         27226    3       1  13      120 2017-11-06 15:13:23   \n",
       "1        110007   35       1  13       10 2017-11-06 15:41:07   \n",
       "2          1047    6       1  13      157 2017-11-06 15:42:32   \n",
       "3         76270    3       1  13      120 2017-11-06 15:56:17   \n",
       "4         57862    3       1  13      120 2017-11-06 15:57:01   \n",
       "...         ...  ...     ...  ..      ...                 ...   \n",
       "1840445   11718   18       1  17       26 2017-11-09 04:50:14   \n",
       "1840449   32849   15       1  19       75 2017-11-09 04:50:14   \n",
       "1840446  249422   48       1  19      103 2017-11-09 04:50:14   \n",
       "1840443  232256   19       6  21       59 2017-11-09 04:50:14   \n",
       "1840442  227929   10       1  46       20 2017-11-09 04:50:14   \n",
       "\n",
       "             attributed_time  is_attributed  day  hour  minute  second  \\\n",
       "0                       None              0    6    15      13      23   \n",
       "1        2017-11-07 08:17:19              1    6    15      41       7   \n",
       "2                       None              0    6    15      42      32   \n",
       "3                       None              0    6    15      56      17   \n",
       "4                       None              0    6    15      57       1   \n",
       "...                      ...            ...  ...   ...     ...     ...   \n",
       "1840445                 None              0    9     4      50      14   \n",
       "1840449                 None              0    9     4      50      14   \n",
       "1840446  2017-11-09 04:53:15              1    9     4      50      14   \n",
       "1840443  2017-11-09 04:50:59              1    9     4      50      14   \n",
       "1840442  2017-11-09 04:50:48              1    9     4      50      14   \n",
       "\n",
       "         ip_target  app_target  device_target  os_target  channel_target  \n",
       "0         0.073529    0.028328       0.152087   0.138712        0.034043  \n",
       "1         0.247523    0.995841       0.152087   0.138712        0.950262  \n",
       "2         0.135593    0.009252       0.152087   0.138712        0.019378  \n",
       "3         0.103448    0.028328       0.152087   0.138712        0.034043  \n",
       "4         0.096774    0.028328       0.152087   0.138712        0.034043  \n",
       "...            ...         ...            ...        ...             ...  \n",
       "1840445   0.071429    0.048499       0.152087   0.109913        0.043930  \n",
       "1840449   0.098039    0.021686       0.152087   0.157243        0.009202  \n",
       "1840446   0.197764    0.951479       0.152087   0.157243        0.970012  \n",
       "1840443   0.197764    0.948415       0.953252   0.953408        0.953326  \n",
       "1840442   0.197764    0.839683       0.152087   0.283852        0.949270  \n",
       "\n",
       "[1840449 rows x 17 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_features = ['ip', 'app', 'device', 'os', 'channel'] #Validation AUC score: 0.9540530347873288\n",
    "# cat_features = ['app', 'device', 'os', 'channel'] #Validation AUC score: 0.9627457957514338\n",
    "train, valid, test = get_data_splits(clicks)\n",
    "\n",
    "# Create the target encoder. You can find this easily by using tab completion.\n",
    "# Start typing ce. the press Tab to bring up a list of classes and functions.\n",
    "target_enc = ce.TargetEncoder(cols=cat_features)\n",
    "\n",
    "# Learn encoding from the training set. Use the 'is_attributed' column as the target.\n",
    "target_enc.fit(train[cat_features], train['is_attributed'])\n",
    "\n",
    "# Apply encoding to the train and validation sets as new columns\n",
    "# Make sure to add `_target` as a suffix to the new columns\n",
    "train_encoded = train.join(target_enc.transform(train[cat_features]).add_suffix('_target'))\n",
    "valid_encoded = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_target'))\n",
    "\n",
    "q_4.check()\n",
    "train_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment these if you need some guidance\n",
    "#q_4.hint()\n",
    "#q_4.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.9540530347873288\n"
     ]
    }
   ],
   "source": [
    "_ = train_model(train_encoded, valid_encoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5) Try removing IP encoding\n",
    "\n",
    "Try leaving `ip` out of the encoded features and retrain the model with target encoding again. You should find that the score increases and is above the baseline score! Why do you think the score is below baseline when we encode the IP address but above baseline when we don't?\n",
    "\n",
    "Uncomment the following line after you've decided your answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 4, \"questionId\": \"5_RemoveIPEncoding\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "    Target encoding attempts to measure the population mean of the target for each \n",
       "    level in a categorical feature. This means when there is less data per level, the \n",
       "    estimated mean will be further away from the \"true\" mean, there will be more variance. \n",
       "    There is little data per IP address so it's likely that the estimates are much noisier\n",
       "    than for the other features. The model will rely heavily on this feature since it is \n",
       "    extremely predictive. This causes it to make fewer splits on other features, and those\n",
       "    features are fit on just the errors left over accounting for IP address. So, the \n",
       "    model will perform very poorly when seeing new IP addresses that weren't in the \n",
       "    training data (which is likely most new data). Going forward, we'll leave out the IP feature when trying\n",
       "    different encodings.\n",
       "    "
      ],
      "text/plain": [
       "Solution: \n",
       "    Target encoding attempts to measure the population mean of the target for each \n",
       "    level in a categorical feature. This means when there is less data per level, the \n",
       "    estimated mean will be further away from the \"true\" mean, there will be more variance. \n",
       "    There is little data per IP address so it's likely that the estimates are much noisier\n",
       "    than for the other features. The model will rely heavily on this feature since it is \n",
       "    extremely predictive. This causes it to make fewer splits on other features, and those\n",
       "    features are fit on just the errors left over accounting for IP address. So, the \n",
       "    model will perform very poorly when seeing new IP addresses that weren't in the \n",
       "    training data (which is likely most new data). Going forward, we'll leave out the IP feature when trying\n",
       "    different encodings.\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q_5.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6) CatBoost Encoding\n",
    "\n",
    "The CatBoost encoder is supposed to working well with the LightGBM model. Encode the categorical features with `CatBoostEncoder` and train the model on the encoded data again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.125, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"6_CatBoostEncodings\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ip</th>\n",
       "      <th>app</th>\n",
       "      <th>device</th>\n",
       "      <th>os</th>\n",
       "      <th>channel</th>\n",
       "      <th>click_time</th>\n",
       "      <th>attributed_time</th>\n",
       "      <th>is_attributed</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>minute</th>\n",
       "      <th>second</th>\n",
       "      <th>app_cb</th>\n",
       "      <th>device_cb</th>\n",
       "      <th>os_cb</th>\n",
       "      <th>channel_cb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>27226</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:13:23</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>13</td>\n",
       "      <td>23</td>\n",
       "      <td>0.028329</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.034049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>110007</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>10</td>\n",
       "      <td>2017-11-06 15:41:07</td>\n",
       "      <td>2017-11-07 08:17:19</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>41</td>\n",
       "      <td>7</td>\n",
       "      <td>0.995828</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.950244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1047</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>157</td>\n",
       "      <td>2017-11-06 15:42:32</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>42</td>\n",
       "      <td>32</td>\n",
       "      <td>0.009261</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.019384</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>76270</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:56:17</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>56</td>\n",
       "      <td>17</td>\n",
       "      <td>0.028329</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.034049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57862</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>120</td>\n",
       "      <td>2017-11-06 15:57:01</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>0.028329</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.138712</td>\n",
       "      <td>0.034049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840445</th>\n",
       "      <td>11718</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>26</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.048500</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.109914</td>\n",
       "      <td>0.043934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840449</th>\n",
       "      <td>32849</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>75</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.021687</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.157243</td>\n",
       "      <td>0.009204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840446</th>\n",
       "      <td>249422</td>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>103</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:53:15</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.950588</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.157243</td>\n",
       "      <td>0.969908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840443</th>\n",
       "      <td>232256</td>\n",
       "      <td>19</td>\n",
       "      <td>6</td>\n",
       "      <td>21</td>\n",
       "      <td>59</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:50:59</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.948408</td>\n",
       "      <td>0.951720</td>\n",
       "      <td>0.953358</td>\n",
       "      <td>0.953319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1840442</th>\n",
       "      <td>227929</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>20</td>\n",
       "      <td>2017-11-09 04:50:14</td>\n",
       "      <td>2017-11-09 04:50:48</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>14</td>\n",
       "      <td>0.839664</td>\n",
       "      <td>0.152087</td>\n",
       "      <td>0.283847</td>\n",
       "      <td>0.949253</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1840449 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             ip  app  device  os  channel          click_time  \\\n",
       "0         27226    3       1  13      120 2017-11-06 15:13:23   \n",
       "1        110007   35       1  13       10 2017-11-06 15:41:07   \n",
       "2          1047    6       1  13      157 2017-11-06 15:42:32   \n",
       "3         76270    3       1  13      120 2017-11-06 15:56:17   \n",
       "4         57862    3       1  13      120 2017-11-06 15:57:01   \n",
       "...         ...  ...     ...  ..      ...                 ...   \n",
       "1840445   11718   18       1  17       26 2017-11-09 04:50:14   \n",
       "1840449   32849   15       1  19       75 2017-11-09 04:50:14   \n",
       "1840446  249422   48       1  19      103 2017-11-09 04:50:14   \n",
       "1840443  232256   19       6  21       59 2017-11-09 04:50:14   \n",
       "1840442  227929   10       1  46       20 2017-11-09 04:50:14   \n",
       "\n",
       "             attributed_time  is_attributed  day  hour  minute  second  \\\n",
       "0                       None              0    6    15      13      23   \n",
       "1        2017-11-07 08:17:19              1    6    15      41       7   \n",
       "2                       None              0    6    15      42      32   \n",
       "3                       None              0    6    15      56      17   \n",
       "4                       None              0    6    15      57       1   \n",
       "...                      ...            ...  ...   ...     ...     ...   \n",
       "1840445                 None              0    9     4      50      14   \n",
       "1840449                 None              0    9     4      50      14   \n",
       "1840446  2017-11-09 04:53:15              1    9     4      50      14   \n",
       "1840443  2017-11-09 04:50:59              1    9     4      50      14   \n",
       "1840442  2017-11-09 04:50:48              1    9     4      50      14   \n",
       "\n",
       "           app_cb  device_cb     os_cb  channel_cb  \n",
       "0        0.028329   0.152087  0.138712    0.034049  \n",
       "1        0.995828   0.152087  0.138712    0.950244  \n",
       "2        0.009261   0.152087  0.138712    0.019384  \n",
       "3        0.028329   0.152087  0.138712    0.034049  \n",
       "4        0.028329   0.152087  0.138712    0.034049  \n",
       "...           ...        ...       ...         ...  \n",
       "1840445  0.048500   0.152087  0.109914    0.043934  \n",
       "1840449  0.021687   0.152087  0.157243    0.009204  \n",
       "1840446  0.950588   0.152087  0.157243    0.969908  \n",
       "1840443  0.948408   0.951720  0.953358    0.953319  \n",
       "1840442  0.839664   0.152087  0.283847    0.949253  \n",
       "\n",
       "[1840449 rows x 16 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train, valid, test = get_data_splits(clicks)\n",
    "cb_features = ['app', 'device', 'os', 'channel']\n",
    "\n",
    "# Create the CatBoost encoder\n",
    "cb_enc = ce.CatBoostEncoder(cols=cb_features)\n",
    "\n",
    "# Learn encoding from the training set\n",
    "cb_enc.fit(train[cb_features], train['is_attributed'])\n",
    "\n",
    "# Apply encoding to the train and validation sets as new columns\n",
    "# Make sure to add `_cb` as a suffix to the new columns\n",
    "train_encoded = train.join(cb_enc.transform(train[cb_features]).add_suffix('_cb'))\n",
    "valid_encoded = valid.join(cb_enc.transform(valid[cb_features]).add_suffix('_cb'))\n",
    "q_6.check()\n",
    "train_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment these if you need some guidance\n",
    "#q_6.hint()\n",
    "#q_6.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.9622743228943659\n"
     ]
    }
   ],
   "source": [
    "_ = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CatBoost encodings work the best, so we'll keep those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded = cb_enc.transform(clicks[cb_features])\n",
    "for col in encoded:\n",
    "    clicks.insert(len(clicks.columns), col + '_cb', encoded[col])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical feature embeddings with SVD\n",
    "\n",
    "Now you'll create embeddings from pairs of columns using SVD to learn from a count matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "from sklearn.decomposition import TruncatedSVD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7) Learn SVD components as embeddings.\n",
    "\n",
    "Here you'll use SVD to learn embeddings for the categorical features from a matrix of counts. First, create the SVD transformer with `TruncatedSVD`. Then for each pair of features, create a count matrix and learn the SVD components. Remember you should be learning the embeddings from the train dataset to avoid leakage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.125, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"7_LearnSVDEmbeddings\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'app_device':                 0             1             2             3         4\n",
       " 0    1.300105e-10  2.901272e+02  4.654852e-07  7.549226e-06  0.015482\n",
       " 1    4.567491e+04 -5.192989e-07 -7.026271e+02  1.597962e+02  6.190715\n",
       " 2    1.726700e+05 -2.828537e-06 -1.498031e+03  4.743132e+01 -5.236503\n",
       " 3    2.832167e+05 -4.486057e-06  1.385305e+03 -1.540166e+03  3.161593\n",
       " 4    4.904132e+02  6.423896e-08 -4.725334e-01  1.331522e+00  2.302140\n",
       " ..            ...           ...           ...           ...       ...\n",
       " 366  9.992396e-01 -1.569303e-11  3.458743e-02 -1.799640e-02  0.000058\n",
       " 367  4.239255e-13  9.982279e-01 -3.308542e-09 -1.665186e-08 -0.000022\n",
       " 368  9.992396e-01 -1.569303e-11  3.458743e-02 -1.799640e-02  0.000058\n",
       " 369  4.239255e-13  9.982279e-01 -3.308542e-09 -1.665186e-08 -0.000022\n",
       " 370  9.992396e-01 -1.569303e-11  3.458743e-02 -1.799640e-02  0.000058\n",
       " \n",
       " [371 rows x 5 columns],\n",
       " 'app_os':                  0           1          2            3            4\n",
       " 0         0.200938  246.847748   1.678808    17.186464   -10.092338\n",
       " 1     16978.999843   -1.199598  39.443187  -235.789506  -163.308630\n",
       " 2     63061.997716   -5.509504  95.700139 -1133.038948 -1611.494215\n",
       " 3    102150.019578   -8.633965  51.333141    59.647089  1279.646762\n",
       " 4       144.032334    0.035235   2.572066    15.938814    15.008556\n",
       " ..             ...         ...        ...          ...          ...\n",
       " 366       0.039636    0.000002  -0.003789     0.043735    -0.009406\n",
       " 367       0.000002    0.263345  -0.000152    -0.001002    -0.000090\n",
       " 368       0.629503   -0.000055  -0.060177     0.099537     0.423755\n",
       " 369       0.000213    0.002890   0.001090     0.012755    -0.007736\n",
       " 370       0.116812   -0.000010  -0.003052    -0.006148    -0.099649\n",
       " \n",
       " [371 rows x 5 columns],\n",
       " 'app_channel':                 0            1             2             3             4\n",
       " 0    2.485156e-02    18.163591 -6.117757e-01  1.679049e-01 -2.260076e-01\n",
       " 1    9.069397e+02   184.710775  1.153590e+03  3.268639e+03  3.914558e+02\n",
       " 2    2.193817e+03  1118.051188  3.343407e+03  4.162634e+02  5.469302e+04\n",
       " 3    1.375361e+05  -116.831551 -1.473652e+03 -3.189017e+02 -8.572372e+02\n",
       " 4    2.069632e-02    15.126592 -5.094852e-01  1.398308e-01 -1.882185e-01\n",
       " ..            ...          ...           ...           ...           ...\n",
       " 366  2.827967e-04     0.000431  1.312243e-03  1.888975e-04  4.046363e-02\n",
       " 367  3.028590e-09     0.000009 -3.438305e-07 -7.627514e-09 -1.568834e-07\n",
       " 368  3.028590e-09     0.000009 -3.438305e-07 -7.627514e-09 -1.568834e-07\n",
       " 369  2.764277e-07     0.000002  1.869262e-07  1.436184e-06  6.562499e-07\n",
       " 370  2.827967e-04     0.000431  1.312243e-03  1.888975e-04  4.046363e-02\n",
       " \n",
       " [371 rows x 5 columns],\n",
       " 'device_app':                  0             1             2             3         4\n",
       " 0     1.826148e-07  72358.345321  2.804840e-05 -7.456448e-05 -0.005580\n",
       " 1     4.645228e+05     -0.000001 -2.932173e+02 -8.058511e+01  0.014834\n",
       " 2     1.806664e+04     -0.000001  7.804449e+03  1.739889e+03 -0.500978\n",
       " 3     2.073216e-11    241.937849  3.980201e-06 -1.805817e-05 -0.001236\n",
       " 4     7.298543e-09   2229.767155 -9.819500e-06  4.675903e-05  0.008393\n",
       " ...            ...           ...           ...           ...       ...\n",
       " 1713  4.607990e-14      0.000998 -2.695722e-12  2.396543e-10  0.000009\n",
       " 1714 -2.203339e-13      0.004002 -5.491057e-11  1.685891e-09  0.000060\n",
       " 1715  9.203447e-14      0.999087  1.643987e-08 -7.462989e-08 -0.000007\n",
       " 1716  9.203447e-14      0.999087  1.643987e-08 -7.462989e-08 -0.000007\n",
       " 1717  5.413056e-16      0.024662  3.831363e-10 -1.469425e-09  0.000013\n",
       " \n",
       " [1718 rows x 5 columns],\n",
       " 'device_os':                  0             1             2          3             4\n",
       " 0     3.853966e-01  37534.385284 -3.906403e-01 -29.090431  1.967880e-01\n",
       " 1     5.780741e+05     -0.050512 -2.307254e+02   0.824000  3.552621e-02\n",
       " 2     1.520865e+04      0.950676  8.769672e+03 -31.941788 -1.438815e+00\n",
       " 3     2.217953e-06     51.079669 -9.832227e-03  -1.172530  5.944824e-02\n",
       " 4     2.478705e-05    573.464537 -1.200260e-01 -15.150844 -9.736651e+01\n",
       " ...            ...           ...           ...        ...           ...\n",
       " 1713  1.320874e-08      0.304257 -5.876455e-05  -0.007024 -1.277621e-04\n",
       " 1714  1.320874e-08      0.304257 -5.876455e-05  -0.007024 -1.277621e-04\n",
       " 1715  2.305338e-12      0.000053 -1.015146e-08  -0.000001  5.822744e-08\n",
       " 1716  3.457909e-12      0.000080 -1.522701e-08  -0.000002  8.733003e-08\n",
       " 1717  4.610676e-12      0.000106 -2.030291e-08  -0.000002  1.164549e-07\n",
       " \n",
       " [1718 rows x 5 columns],\n",
       " 'device_channel':                   0             1            2            3           4\n",
       " 0       6818.514826  53641.844782   144.813178    23.609504   68.605076\n",
       " 1     250470.268282  -1431.810029  -259.771092   -87.136627    2.392175\n",
       " 2       9230.037032  -1016.634158  7294.677499  1677.617384  -40.748682\n",
       " 3         24.914387    181.783994     1.701114   -18.841657  -52.272303\n",
       " 4        272.191580   2039.109744    16.666871  -212.027558 -592.277969\n",
       " ...             ...           ...          ...          ...         ...\n",
       " 1713       0.000256      0.003540     0.000214    -0.000138    0.002170\n",
       " 1714       0.112407      0.041615     0.001351    -0.024878    0.057437\n",
       " 1715       0.122350      0.944111     0.007660    -0.097507   -0.275376\n",
       " 1716       0.122350      0.944111     0.007660    -0.097507   -0.275376\n",
       " 1717       0.244699      1.888222     0.015319    -0.195014   -0.550752\n",
       " \n",
       " [1718 rows x 5 columns],\n",
       " 'os_app':                 0             1             2             3             4\n",
       " 0    7.058477e-01  2.809519e+04 -4.291674e+00 -3.419795e+01 -1.663424e+01\n",
       " 1    5.928692e+03  1.225007e+00  3.020931e+02  1.380235e+03  1.297072e+02\n",
       " 2    1.476288e+03  4.019870e-02  4.252246e+01  9.033572e+01  4.634149e+00\n",
       " 3    7.297526e+03  7.557821e-01  2.970046e+02  9.685486e+02  9.102721e+01\n",
       " 4    1.517231e+03  1.242657e-02  2.523405e+03 -3.168543e+02 -4.877609e+01\n",
       " ..            ...           ...           ...           ...           ...\n",
       " 217  3.723098e-01 -1.118795e-04  1.110478e-02 -1.821096e-01  3.716790e-01\n",
       " 218  4.023231e-40 -4.556144e-41 -2.900266e-38 -3.264421e-37 -9.445999e-35\n",
       " 219  3.178488e+02  3.677475e+02  1.416062e+02  1.050313e+03  4.263900e+02\n",
       " 220  3.723098e-01 -1.118795e-04  1.110478e-02 -1.821096e-01  3.716790e-01\n",
       " 221  3.600202e+01  1.422952e+02  9.396599e+00  7.935889e+01  3.354093e+01\n",
       " \n",
       " [222 rows x 5 columns],\n",
       " 'os_device':                 0             1           2           3            4\n",
       " 0        0.014129  21246.440259   -1.023918 -101.499284    19.285528\n",
       " 1    19479.336393     -0.017437 -167.360410    0.723868     0.075154\n",
       " 2     4691.425369      0.009137  486.781040   -2.250048    -0.273716\n",
       " 3    24489.864259     -0.018145  -61.100359    0.223304     0.011821\n",
       " 4     8615.280195     -0.004588   49.431100   -0.247656    -0.035075\n",
       " ..            ...           ...         ...         ...          ...\n",
       " 217      0.999654     -0.000001   -0.026300    0.000119     0.000014\n",
       " 218      0.000002      2.993848   -0.000134   -0.012562     0.000227\n",
       " 219    115.436529    860.392221   16.368225   -4.264695  2596.898189\n",
       " 220      0.999654     -0.000001   -0.026300    0.000119     0.000014\n",
       " 221     51.245609    298.456013    8.678509   -1.300748     0.018784\n",
       " \n",
       " [222 rows x 5 columns],\n",
       " 'os_channel':                0             1            2            3           4\n",
       " 0    3389.345194  20787.508516    79.597064   212.336249   48.027768\n",
       " 1    3259.945179   -264.130456   245.595227  1463.309295  -81.510002\n",
       " 2     687.232521    -98.587155    31.077795   113.206998   19.504798\n",
       " 3    3865.970641   -208.429982   254.769062   990.356963 -105.740981\n",
       " 4     747.552186    -80.828503  3562.286638  -207.661142   71.136714\n",
       " ..           ...           ...          ...          ...         ...\n",
       " 217     0.083618     -0.013519    -0.001964    -0.037446    0.076082\n",
       " 218     0.003575     -0.000504     0.002570    -0.003270    0.008691\n",
       " 219   337.038843    480.922857    79.489786  1335.449953  706.519789\n",
       " 220     0.083618     -0.013519    -0.001964    -0.037446    0.076082\n",
       " 221    61.984932    174.641024     6.443311   169.238168  148.373139\n",
       " \n",
       " [222 rows x 5 columns],\n",
       " 'channel_app':                 0           1             2             3             4\n",
       " 0    3.487837e-09    0.000004 -1.495075e-07  4.411922e-08 -7.581540e-08\n",
       " 1    7.756628e+01  154.252240  4.444978e+03 -3.376420e+01 -6.217603e+02\n",
       " 2    1.957670e-05    0.000022  5.907260e-06  1.243979e-05  1.066298e-04\n",
       " 3    2.042864e-02   36.646002 -1.330541e+00 -2.759773e-02 -4.428851e-01\n",
       " 4    2.739284e+00    0.963393  6.078856e+00  1.851971e+01  2.831517e+00\n",
       " ..            ...         ...           ...           ...           ...\n",
       " 174  1.112903e+04  -11.530975 -1.306677e+02  9.572918e+02 -1.010481e+02\n",
       " 175  6.563233e-01    0.011925  8.098168e-02  6.019125e+00  3.100148e-01\n",
       " 176  2.329411e-06    0.010421 -3.859314e-04 -1.027196e-06 -2.486180e-04\n",
       " 177  5.440381e+01  110.064374  3.024836e+03 -2.803780e+01  5.342056e+01\n",
       " 178  1.332406e-03    0.035521 -9.683627e-05  3.802576e-04 -7.947793e-04\n",
       " \n",
       " [179 rows x 5 columns],\n",
       " 'channel_device':                 0           1           2           3          4\n",
       " 0       49.021624   -2.299355  -50.701229   19.134099  -1.940088\n",
       " 1     6171.981256 -184.035858 -985.306837  304.745626 -25.310580\n",
       " 2       11.512226   55.570474   -0.714008    0.086875   2.680226\n",
       " 3     3695.061922  -98.929382   98.926297  -53.373482   5.371136\n",
       " 4      394.091460  -10.695774    2.945235    3.289006  -0.447177\n",
       " ..            ...         ...         ...         ...        ...\n",
       " 174  22684.829195 -606.792313  636.512057 -337.383618  34.414493\n",
       " 175     38.254087   -1.162514   -6.280943    1.948432  -0.162989\n",
       " 176   1191.652965  -32.196505   16.486756  -11.935189   1.271352\n",
       " 177   3861.516086 -103.184974  113.996876  -59.411034   5.929955\n",
       " 178     69.925568   -1.862991    2.355572   -1.175557   0.116097\n",
       " \n",
       " [179 rows x 5 columns],\n",
       " 'channel_os':                0           1          2           3           4\n",
       " 0      32.914533   -2.368109  -0.673963    3.217676   -3.846869\n",
       " 1    2507.763501 -168.892677   0.583283  -58.735112   28.830033\n",
       " 2       7.995650   26.631595  -0.150185   -1.887786    0.021240\n",
       " 3     337.532691  -41.283451  42.735029  384.379680 -507.034739\n",
       " 4     118.404227   -8.076728   2.494637    7.005116    5.600568\n",
       " ..           ...         ...        ...         ...         ...\n",
       " 174  8038.184882 -539.469051  38.138055 -202.612938  194.425923\n",
       " 175    13.176715   -0.959892   0.351064    1.045785   -1.822076\n",
       " 176   451.860093  -30.556266 -11.839742   -5.698830   -6.462156\n",
       " 177  1406.867899  -93.610525   1.634244  -46.720049   44.665008\n",
       " 178    30.745686   -1.817762  -1.627209   -4.519905    7.015248\n",
       " \n",
       " [179 rows x 5 columns]}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train, valid, test = get_data_splits(clicks)\n",
    "cat_features = ['app', 'device', 'os', 'channel']\n",
    "\n",
    "# Create the SVD transformer\n",
    "svd = TruncatedSVD(n_components=5, random_state=7)\n",
    "\n",
    "# Learn SVD feature vectors and store in svd_components as DataFrames\n",
    "# Make sure you're only using the train set!\n",
    "svd_components = {}\n",
    "for col1, col2 in itertools.permutations(cat_features, 2):\n",
    "    # Create the count matrix\n",
    "    pair_counts = train.groupby([col1, col2])['is_attributed'].count()\n",
    "    pair_matrix = pair_counts.unstack(fill_value=0)\n",
    "    \n",
    "    # Fit the SVD and transform to get the components\n",
    "    svd_comp = pd.DataFrame(svd.fit_transform(pair_matrix))\n",
    "    \n",
    "    # Store the components in the dictionary. \n",
    "    svd_components['_'.join([col1, col2])] = svd_comp\n",
    "\n",
    "q_7.check()\n",
    "svd_components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment these if you need some guidance\n",
    "#q_7.hint()\n",
    "#q_7.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8) Encode categorical features with SVD components\n",
    "\n",
    "With the components learned from the train dataset, encode the categorical features and create a dataframe `svd_encodings`. The columns need to be named with the feature pair, `svd`, and the component index, such as `\"os_device_svd_0\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.125, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"8_ApplySVDEncoding\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_device_svd_0</th>\n",
       "      <th>app_device_svd_1</th>\n",
       "      <th>app_device_svd_2</th>\n",
       "      <th>app_device_svd_3</th>\n",
       "      <th>app_device_svd_4</th>\n",
       "      <th>app_os_svd_0</th>\n",
       "      <th>app_os_svd_1</th>\n",
       "      <th>app_os_svd_2</th>\n",
       "      <th>app_os_svd_3</th>\n",
       "      <th>app_os_svd_4</th>\n",
       "      <th>...</th>\n",
       "      <th>channel_device_svd_0</th>\n",
       "      <th>channel_device_svd_1</th>\n",
       "      <th>channel_device_svd_2</th>\n",
       "      <th>channel_device_svd_3</th>\n",
       "      <th>channel_device_svd_4</th>\n",
       "      <th>channel_os_svd_0</th>\n",
       "      <th>channel_os_svd_1</th>\n",
       "      <th>channel_os_svd_2</th>\n",
       "      <th>channel_os_svd_3</th>\n",
       "      <th>channel_os_svd_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>283216.718138</td>\n",
       "      <td>-4.486057e-06</td>\n",
       "      <td>1385.304677</td>\n",
       "      <td>-1540.165516</td>\n",
       "      <td>3.161593</td>\n",
       "      <td>102150.019578</td>\n",
       "      <td>-8.633965</td>\n",
       "      <td>51.333141</td>\n",
       "      <td>59.647089</td>\n",
       "      <td>1279.646762</td>\n",
       "      <td>...</td>\n",
       "      <td>26587.195203</td>\n",
       "      <td>-711.013555</td>\n",
       "      <td>754.972091</td>\n",
       "      <td>-397.795706</td>\n",
       "      <td>39.706117</td>\n",
       "      <td>9605.064364</td>\n",
       "      <td>-639.692285</td>\n",
       "      <td>56.155759</td>\n",
       "      <td>-338.290917</td>\n",
       "      <td>334.849976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60058.351768</td>\n",
       "      <td>-8.708913e-07</td>\n",
       "      <td>2079.688316</td>\n",
       "      <td>-1077.069410</td>\n",
       "      <td>6.152669</td>\n",
       "      <td>26427.370290</td>\n",
       "      <td>-2.390482</td>\n",
       "      <td>-1421.668517</td>\n",
       "      <td>-1635.306384</td>\n",
       "      <td>-1793.084266</td>\n",
       "      <td>...</td>\n",
       "      <td>41142.467644</td>\n",
       "      <td>-1093.126005</td>\n",
       "      <td>1331.917842</td>\n",
       "      <td>-669.993124</td>\n",
       "      <td>68.997836</td>\n",
       "      <td>19689.842015</td>\n",
       "      <td>-1232.489887</td>\n",
       "      <td>-1175.734245</td>\n",
       "      <td>-1840.713269</td>\n",
       "      <td>2246.722843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18740.410320</td>\n",
       "      <td>-2.848323e-07</td>\n",
       "      <td>-125.681549</td>\n",
       "      <td>-10.350328</td>\n",
       "      <td>0.345621</td>\n",
       "      <td>6639.894162</td>\n",
       "      <td>-0.542968</td>\n",
       "      <td>-32.530579</td>\n",
       "      <td>268.887232</td>\n",
       "      <td>550.518065</td>\n",
       "      <td>...</td>\n",
       "      <td>772.546174</td>\n",
       "      <td>-20.761715</td>\n",
       "      <td>16.562638</td>\n",
       "      <td>-9.748269</td>\n",
       "      <td>0.999530</td>\n",
       "      <td>78.679008</td>\n",
       "      <td>-6.979560</td>\n",
       "      <td>169.783860</td>\n",
       "      <td>-9.697992</td>\n",
       "      <td>3.655545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>283216.718138</td>\n",
       "      <td>-4.486057e-06</td>\n",
       "      <td>1385.304677</td>\n",
       "      <td>-1540.165516</td>\n",
       "      <td>3.161593</td>\n",
       "      <td>102150.019578</td>\n",
       "      <td>-8.633965</td>\n",
       "      <td>51.333141</td>\n",
       "      <td>59.647089</td>\n",
       "      <td>1279.646762</td>\n",
       "      <td>...</td>\n",
       "      <td>26587.195203</td>\n",
       "      <td>-711.013555</td>\n",
       "      <td>754.972091</td>\n",
       "      <td>-397.795706</td>\n",
       "      <td>39.706117</td>\n",
       "      <td>9605.064364</td>\n",
       "      <td>-639.692285</td>\n",
       "      <td>56.155759</td>\n",
       "      <td>-338.290917</td>\n",
       "      <td>334.849976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>283216.718138</td>\n",
       "      <td>-4.486057e-06</td>\n",
       "      <td>1385.304677</td>\n",
       "      <td>-1540.165516</td>\n",
       "      <td>3.161593</td>\n",
       "      <td>102150.019578</td>\n",
       "      <td>-8.633965</td>\n",
       "      <td>51.333141</td>\n",
       "      <td>59.647089</td>\n",
       "      <td>1279.646762</td>\n",
       "      <td>...</td>\n",
       "      <td>26587.195203</td>\n",
       "      <td>-711.013555</td>\n",
       "      <td>754.972091</td>\n",
       "      <td>-397.795706</td>\n",
       "      <td>39.706117</td>\n",
       "      <td>9605.064364</td>\n",
       "      <td>-639.692285</td>\n",
       "      <td>56.155759</td>\n",
       "      <td>-338.290917</td>\n",
       "      <td>334.849976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300556</th>\n",
       "      <td>172669.961330</td>\n",
       "      <td>-2.828537e-06</td>\n",
       "      <td>-1498.030919</td>\n",
       "      <td>47.431321</td>\n",
       "      <td>-5.236503</td>\n",
       "      <td>63061.997716</td>\n",
       "      <td>-5.509504</td>\n",
       "      <td>95.700139</td>\n",
       "      <td>-1133.038948</td>\n",
       "      <td>-1611.494215</td>\n",
       "      <td>...</td>\n",
       "      <td>122.497195</td>\n",
       "      <td>-3.561528</td>\n",
       "      <td>-11.890012</td>\n",
       "      <td>3.698852</td>\n",
       "      <td>0.295177</td>\n",
       "      <td>48.520465</td>\n",
       "      <td>-3.325046</td>\n",
       "      <td>0.580528</td>\n",
       "      <td>0.197144</td>\n",
       "      <td>-0.480342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300557</th>\n",
       "      <td>79928.764567</td>\n",
       "      <td>-1.205976e-06</td>\n",
       "      <td>1545.895064</td>\n",
       "      <td>-919.183624</td>\n",
       "      <td>4.333729</td>\n",
       "      <td>28752.787540</td>\n",
       "      <td>-2.443380</td>\n",
       "      <td>158.195767</td>\n",
       "      <td>-633.909891</td>\n",
       "      <td>-848.521571</td>\n",
       "      <td>...</td>\n",
       "      <td>95.230290</td>\n",
       "      <td>-2.698418</td>\n",
       "      <td>-5.307795</td>\n",
       "      <td>1.314516</td>\n",
       "      <td>-0.096705</td>\n",
       "      <td>41.842097</td>\n",
       "      <td>-2.662455</td>\n",
       "      <td>-1.770468</td>\n",
       "      <td>-3.110683</td>\n",
       "      <td>4.310531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300558</th>\n",
       "      <td>189787.724105</td>\n",
       "      <td>-2.947984e-06</td>\n",
       "      <td>-3215.582416</td>\n",
       "      <td>733.272197</td>\n",
       "      <td>-4.208618</td>\n",
       "      <td>70473.135089</td>\n",
       "      <td>-5.627409</td>\n",
       "      <td>-709.120366</td>\n",
       "      <td>250.891669</td>\n",
       "      <td>2240.970109</td>\n",
       "      <td>...</td>\n",
       "      <td>40161.115471</td>\n",
       "      <td>-1130.602497</td>\n",
       "      <td>-1848.755738</td>\n",
       "      <td>431.205539</td>\n",
       "      <td>-28.881444</td>\n",
       "      <td>16899.283067</td>\n",
       "      <td>-1150.141776</td>\n",
       "      <td>-828.147677</td>\n",
       "      <td>-32.152357</td>\n",
       "      <td>-603.239424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300559</th>\n",
       "      <td>189787.724105</td>\n",
       "      <td>-2.947984e-06</td>\n",
       "      <td>-3215.582416</td>\n",
       "      <td>733.272197</td>\n",
       "      <td>-4.208618</td>\n",
       "      <td>70473.135089</td>\n",
       "      <td>-5.627409</td>\n",
       "      <td>-709.120366</td>\n",
       "      <td>250.891669</td>\n",
       "      <td>2240.970109</td>\n",
       "      <td>...</td>\n",
       "      <td>27273.494469</td>\n",
       "      <td>-766.254400</td>\n",
       "      <td>-1173.615189</td>\n",
       "      <td>259.866933</td>\n",
       "      <td>-17.775070</td>\n",
       "      <td>10910.902308</td>\n",
       "      <td>-732.890408</td>\n",
       "      <td>-98.370534</td>\n",
       "      <td>-268.506719</td>\n",
       "      <td>56.509892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2300560</th>\n",
       "      <td>124373.544804</td>\n",
       "      <td>-1.680295e-06</td>\n",
       "      <td>-2944.553238</td>\n",
       "      <td>840.770025</td>\n",
       "      <td>9.207956</td>\n",
       "      <td>48573.938684</td>\n",
       "      <td>-3.779405</td>\n",
       "      <td>-550.046505</td>\n",
       "      <td>-1012.353257</td>\n",
       "      <td>-501.368425</td>\n",
       "      <td>...</td>\n",
       "      <td>51771.397585</td>\n",
       "      <td>-1416.499392</td>\n",
       "      <td>-220.232076</td>\n",
       "      <td>-197.507473</td>\n",
       "      <td>28.496132</td>\n",
       "      <td>19556.403365</td>\n",
       "      <td>-1314.770671</td>\n",
       "      <td>-60.549682</td>\n",
       "      <td>-464.136058</td>\n",
       "      <td>202.975826</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2300561 rows × 60 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         app_device_svd_0  app_device_svd_1  app_device_svd_2  \\\n",
       "0           283216.718138     -4.486057e-06       1385.304677   \n",
       "1            60058.351768     -8.708913e-07       2079.688316   \n",
       "2            18740.410320     -2.848323e-07       -125.681549   \n",
       "3           283216.718138     -4.486057e-06       1385.304677   \n",
       "4           283216.718138     -4.486057e-06       1385.304677   \n",
       "...                   ...               ...               ...   \n",
       "2300556     172669.961330     -2.828537e-06      -1498.030919   \n",
       "2300557      79928.764567     -1.205976e-06       1545.895064   \n",
       "2300558     189787.724105     -2.947984e-06      -3215.582416   \n",
       "2300559     189787.724105     -2.947984e-06      -3215.582416   \n",
       "2300560     124373.544804     -1.680295e-06      -2944.553238   \n",
       "\n",
       "         app_device_svd_3  app_device_svd_4   app_os_svd_0  app_os_svd_1  \\\n",
       "0            -1540.165516          3.161593  102150.019578     -8.633965   \n",
       "1            -1077.069410          6.152669   26427.370290     -2.390482   \n",
       "2              -10.350328          0.345621    6639.894162     -0.542968   \n",
       "3            -1540.165516          3.161593  102150.019578     -8.633965   \n",
       "4            -1540.165516          3.161593  102150.019578     -8.633965   \n",
       "...                   ...               ...            ...           ...   \n",
       "2300556         47.431321         -5.236503   63061.997716     -5.509504   \n",
       "2300557       -919.183624          4.333729   28752.787540     -2.443380   \n",
       "2300558        733.272197         -4.208618   70473.135089     -5.627409   \n",
       "2300559        733.272197         -4.208618   70473.135089     -5.627409   \n",
       "2300560        840.770025          9.207956   48573.938684     -3.779405   \n",
       "\n",
       "         app_os_svd_2  app_os_svd_3  app_os_svd_4  ...  channel_device_svd_0  \\\n",
       "0           51.333141     59.647089   1279.646762  ...          26587.195203   \n",
       "1        -1421.668517  -1635.306384  -1793.084266  ...          41142.467644   \n",
       "2          -32.530579    268.887232    550.518065  ...            772.546174   \n",
       "3           51.333141     59.647089   1279.646762  ...          26587.195203   \n",
       "4           51.333141     59.647089   1279.646762  ...          26587.195203   \n",
       "...               ...           ...           ...  ...                   ...   \n",
       "2300556     95.700139  -1133.038948  -1611.494215  ...            122.497195   \n",
       "2300557    158.195767   -633.909891   -848.521571  ...             95.230290   \n",
       "2300558   -709.120366    250.891669   2240.970109  ...          40161.115471   \n",
       "2300559   -709.120366    250.891669   2240.970109  ...          27273.494469   \n",
       "2300560   -550.046505  -1012.353257   -501.368425  ...          51771.397585   \n",
       "\n",
       "         channel_device_svd_1  channel_device_svd_2  channel_device_svd_3  \\\n",
       "0                 -711.013555            754.972091           -397.795706   \n",
       "1                -1093.126005           1331.917842           -669.993124   \n",
       "2                  -20.761715             16.562638             -9.748269   \n",
       "3                 -711.013555            754.972091           -397.795706   \n",
       "4                 -711.013555            754.972091           -397.795706   \n",
       "...                       ...                   ...                   ...   \n",
       "2300556             -3.561528            -11.890012              3.698852   \n",
       "2300557             -2.698418             -5.307795              1.314516   \n",
       "2300558          -1130.602497          -1848.755738            431.205539   \n",
       "2300559           -766.254400          -1173.615189            259.866933   \n",
       "2300560          -1416.499392           -220.232076           -197.507473   \n",
       "\n",
       "         channel_device_svd_4  channel_os_svd_0  channel_os_svd_1  \\\n",
       "0                   39.706117       9605.064364       -639.692285   \n",
       "1                   68.997836      19689.842015      -1232.489887   \n",
       "2                    0.999530         78.679008         -6.979560   \n",
       "3                   39.706117       9605.064364       -639.692285   \n",
       "4                   39.706117       9605.064364       -639.692285   \n",
       "...                       ...               ...               ...   \n",
       "2300556              0.295177         48.520465         -3.325046   \n",
       "2300557             -0.096705         41.842097         -2.662455   \n",
       "2300558            -28.881444      16899.283067      -1150.141776   \n",
       "2300559            -17.775070      10910.902308       -732.890408   \n",
       "2300560             28.496132      19556.403365      -1314.770671   \n",
       "\n",
       "         channel_os_svd_2  channel_os_svd_3  channel_os_svd_4  \n",
       "0               56.155759       -338.290917        334.849976  \n",
       "1            -1175.734245      -1840.713269       2246.722843  \n",
       "2              169.783860         -9.697992          3.655545  \n",
       "3               56.155759       -338.290917        334.849976  \n",
       "4               56.155759       -338.290917        334.849976  \n",
       "...                   ...               ...               ...  \n",
       "2300556          0.580528          0.197144         -0.480342  \n",
       "2300557         -1.770468         -3.110683          4.310531  \n",
       "2300558       -828.147677        -32.152357       -603.239424  \n",
       "2300559        -98.370534       -268.506719         56.509892  \n",
       "2300560        -60.549682       -464.136058        202.975826  \n",
       "\n",
       "[2300561 rows x 60 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd_encodings = pd.DataFrame(index=clicks.index)\n",
    "\n",
    "for feature in svd_components:\n",
    "    # Get the feature column the SVD components are encoding\n",
    "    col = feature.split('_')[0]\n",
    "    feature_components = svd_components[feature]\n",
    "    \n",
    "    ## Use SVD components to encode the categorical features\n",
    "    comp_cols = feature_components.reindex(clicks[col]).set_index(clicks.index)\n",
    "    \n",
    "    # Add prefix to column names\n",
    "    svd_cols = comp_cols.add_prefix(feature + '_svd_')\n",
    "\n",
    "    # Join encoded features with svd_encodings\n",
    "    svd_encodings = svd_encodings.join(svd_cols)\n",
    "\n",
    "# Fill null values with the mean\n",
    "svd_encodings = svd_encodings.fillna(svd_encodings.mean())\n",
    "\n",
    "q_8.check()\n",
    "svd_encodings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment these if you need some guidance\n",
    "#q_8.hint()\n",
    "#q_8.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the encoded data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model!\n",
      "Validation AUC score: 0.9632510172357092\n"
     ]
    }
   ],
   "source": [
    "train, valid, test = get_data_splits(clicks.join(svd_encodings))\n",
    "_ = train_model(train, valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keep Going\n",
    "\n",
    "Now you are ready to **[generating completely new features](https://www.kaggle.com/matleonard/feature-generation)** from the data itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Feature Engineering Home Page](https://www.kaggle.com/learn/feature-engineering)**\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}