data_cleaning/01-handling-missing-values.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adc3f824",
   "metadata": {
    "papermill": {
     "duration": 0.022408,
     "end_time": "2021-08-22T01:25:26.016166",
     "exception": false,
     "start_time": "2021-08-22T01:25:25.993758",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Data Cleaning](https://www.kaggle.com/learn/data-cleaning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/handling-missing-values).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9825b1c1",
   "metadata": {
    "papermill": {
     "duration": 0.021489,
     "end_time": "2021-08-22T01:25:26.059291",
     "exception": false,
     "start_time": "2021-08-22T01:25:26.037802",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you'll apply what you learned in the **Handling missing values** tutorial.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "16e4ff26",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:26.115350Z",
     "iopub.status.busy": "2021-08-22T01:25:26.114610Z",
     "iopub.status.idle": "2021-08-22T01:25:31.820994Z",
     "shell.execute_reply": "2021-08-22T01:25:31.821497Z",
     "shell.execute_reply.started": "2021-08-22T01:03:02.366228Z"
    },
    "papermill": {
     "duration": 5.742122,
     "end_time": "2021-08-22T01:25:31.821826",
     "exception": false,
     "start_time": "2021-08-22T01:25:26.079704",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3361: DtypeWarning: Columns (22,32) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  if (await self.run_code(code, result,  async_=asy)):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.data_cleaning.ex1 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e874113",
   "metadata": {
    "papermill": {
     "duration": 0.020735,
     "end_time": "2021-08-22T01:25:31.863723",
     "exception": false,
     "start_time": "2021-08-22T01:25:31.842988",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1) Take a first look at the data\n",
    "\n",
    "Run the next code cell to load in the libraries and dataset you'll use to complete the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "380dc0af",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:31.912786Z",
     "iopub.status.busy": "2021-08-22T01:25:31.911693Z",
     "iopub.status.idle": "2021-08-22T01:25:34.458096Z",
     "shell.execute_reply": "2021-08-22T01:25:34.457401Z",
     "shell.execute_reply.started": "2021-08-22T01:03:13.525675Z"
    },
    "papermill": {
     "duration": 2.573911,
     "end_time": "2021-08-22T01:25:34.458284",
     "exception": false,
     "start_time": "2021-08-22T01:25:31.884373",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3170: DtypeWarning: Columns (22,32) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# read in all our data\n",
    "sf_permits = pd.read_csv(\"../input/building-permit-applications-data/Building_Permits.csv\")\n",
    "\n",
    "# set seed for reproducibility\n",
    "np.random.seed(0) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6c5d14",
   "metadata": {
    "papermill": {
     "duration": 0.022172,
     "end_time": "2021-08-22T01:25:34.504016",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.481844",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Use the code cell below to print the first five rows of the `sf_permits` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1f1e21a5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:34.561726Z",
     "iopub.status.busy": "2021-08-22T01:25:34.560912Z",
     "iopub.status.idle": "2021-08-22T01:25:34.598582Z",
     "shell.execute_reply": "2021-08-22T01:25:34.599138Z",
     "shell.execute_reply.started": "2021-08-22T01:04:15.028701Z"
    },
    "papermill": {
     "duration": 0.073732,
     "end_time": "2021-08-22T01:25:34.599335",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.525603",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Permit Number</th>\n",
       "      <th>Permit Type</th>\n",
       "      <th>Permit Type Definition</th>\n",
       "      <th>Permit Creation Date</th>\n",
       "      <th>Block</th>\n",
       "      <th>Lot</th>\n",
       "      <th>Street Number</th>\n",
       "      <th>Street Number Suffix</th>\n",
       "      <th>Street Name</th>\n",
       "      <th>Street Suffix</th>\n",
       "      <th>...</th>\n",
       "      <th>Existing Construction Type</th>\n",
       "      <th>Existing Construction Type Description</th>\n",
       "      <th>Proposed Construction Type</th>\n",
       "      <th>Proposed Construction Type Description</th>\n",
       "      <th>Site Permit</th>\n",
       "      <th>Supervisor District</th>\n",
       "      <th>Neighborhoods - Analysis Boundaries</th>\n",
       "      <th>Zipcode</th>\n",
       "      <th>Location</th>\n",
       "      <th>Record ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201505065519</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>0326</td>\n",
       "      <td>023</td>\n",
       "      <td>140</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ellis</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.785719256680785, -122.40852313194863)</td>\n",
       "      <td>1380611233945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201604195146</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>0306</td>\n",
       "      <td>007</td>\n",
       "      <td>440</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Geary</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.78733980600732, -122.41063199757738)</td>\n",
       "      <td>1420164406718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201605278609</td>\n",
       "      <td>3</td>\n",
       "      <td>additions alterations or repairs</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>0595</td>\n",
       "      <td>203</td>\n",
       "      <td>1647</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>Av</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>constr type 1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>constr type 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Russian Hill</td>\n",
       "      <td>94109.0</td>\n",
       "      <td>(37.7946573324287, -122.42232562979227)</td>\n",
       "      <td>1424856504716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>201611072166</td>\n",
       "      <td>8</td>\n",
       "      <td>otc alterations permit</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>0156</td>\n",
       "      <td>011</td>\n",
       "      <td>1230</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>Av</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>wood frame (5)</td>\n",
       "      <td>5.0</td>\n",
       "      <td>wood frame (5)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Nob Hill</td>\n",
       "      <td>94109.0</td>\n",
       "      <td>(37.79595867909168, -122.41557405519474)</td>\n",
       "      <td>1443574295566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>201611283529</td>\n",
       "      <td>6</td>\n",
       "      <td>demolitions</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>0342</td>\n",
       "      <td>001</td>\n",
       "      <td>950</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Market</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.78315261897309, -122.40950883997789)</td>\n",
       "      <td>144548169992</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Permit Number  Permit Type            Permit Type Definition  \\\n",
       "0  201505065519            4                      sign - erect   \n",
       "1  201604195146            4                      sign - erect   \n",
       "2  201605278609            3  additions alterations or repairs   \n",
       "3  201611072166            8            otc alterations permit   \n",
       "4  201611283529            6                       demolitions   \n",
       "\n",
       "  Permit Creation Date Block  Lot  Street Number Street Number Suffix  \\\n",
       "0           05/06/2015  0326  023            140                  NaN   \n",
       "1           04/19/2016  0306  007            440                  NaN   \n",
       "2           05/27/2016  0595  203           1647                  NaN   \n",
       "3           11/07/2016  0156  011           1230                  NaN   \n",
       "4           11/28/2016  0342  001            950                  NaN   \n",
       "\n",
       "  Street Name Street Suffix  ...  Existing Construction Type  \\\n",
       "0       Ellis            St  ...                         3.0   \n",
       "1       Geary            St  ...                         3.0   \n",
       "2     Pacific            Av  ...                         1.0   \n",
       "3     Pacific            Av  ...                         5.0   \n",
       "4      Market            St  ...                         3.0   \n",
       "\n",
       "  Existing Construction Type Description Proposed Construction Type  \\\n",
       "0                          constr type 3                        NaN   \n",
       "1                          constr type 3                        NaN   \n",
       "2                          constr type 1                        1.0   \n",
       "3                         wood frame (5)                        5.0   \n",
       "4                          constr type 3                        NaN   \n",
       "\n",
       "  Proposed Construction Type Description Site Permit Supervisor District  \\\n",
       "0                                    NaN         NaN                 3.0   \n",
       "1                                    NaN         NaN                 3.0   \n",
       "2                          constr type 1         NaN                 3.0   \n",
       "3                         wood frame (5)         NaN                 3.0   \n",
       "4                                    NaN         NaN                 6.0   \n",
       "\n",
       "  Neighborhoods - Analysis Boundaries  Zipcode  \\\n",
       "0                          Tenderloin  94102.0   \n",
       "1                          Tenderloin  94102.0   \n",
       "2                        Russian Hill  94109.0   \n",
       "3                            Nob Hill  94109.0   \n",
       "4                          Tenderloin  94102.0   \n",
       "\n",
       "                                    Location      Record ID  \n",
       "0  (37.785719256680785, -122.40852313194863)  1380611233945  \n",
       "1   (37.78733980600732, -122.41063199757738)  1420164406718  \n",
       "2    (37.7946573324287, -122.42232562979227)  1424856504716  \n",
       "3   (37.79595867909168, -122.41557405519474)  1443574295566  \n",
       "4   (37.78315261897309, -122.40950883997789)   144548169992  \n",
       "\n",
       "[5 rows x 43 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO: Your code here!\n",
    "sf_permits.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4609d178",
   "metadata": {
    "papermill": {
     "duration": 0.022594,
     "end_time": "2021-08-22T01:25:34.648451",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.625857",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Does the dataset have any missing values?  Once you have an answer, run the code cell below to get credit for your work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0e5f4336",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:34.700336Z",
     "iopub.status.busy": "2021-08-22T01:25:34.698685Z",
     "iopub.status.idle": "2021-08-22T01:25:34.706820Z",
     "shell.execute_reply": "2021-08-22T01:25:34.706146Z",
     "shell.execute_reply.started": "2021-08-22T01:04:18.370261Z"
    },
    "papermill": {
     "duration": 0.036398,
     "end_time": "2021-08-22T01:25:34.706990",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.670592",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"1_TakeFirstLook\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "The first five rows of the data does show that several columns have missing values.  You can see this in the \"Street Number Suffix\", \"Proposed Construction Type\" and \"Site Permit\" columns, among others."
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "The first five rows of the data does show that several columns have missing values.  You can see this in the \"Street Number Suffix\", \"Proposed Construction Type\" and \"Site Permit\" columns, among others."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c262a33d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:34.760268Z",
     "iopub.status.busy": "2021-08-22T01:25:34.759233Z",
     "iopub.status.idle": "2021-08-22T01:25:34.763246Z",
     "shell.execute_reply": "2021-08-22T01:25:34.763777Z",
     "shell.execute_reply.started": "2021-08-22T01:04:51.274057Z"
    },
    "papermill": {
     "duration": 0.034149,
     "end_time": "2021-08-22T01:25:34.763998",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.729849",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Line below will give you a hint\n",
    "#q1.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebd25889",
   "metadata": {
    "papermill": {
     "duration": 0.022495,
     "end_time": "2021-08-22T01:25:34.809597",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.787102",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2) How many missing data points do we have?\n",
    "\n",
    "What percentage of the values in the dataset are missing?  Your answer should be a number between 0 and 100.  (If 1/4 of the values in the dataset are missing, the answer is 25.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2b35ebe1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:34.864721Z",
     "iopub.status.busy": "2021-08-22T01:25:34.863990Z",
     "iopub.status.idle": "2021-08-22T01:25:35.339280Z",
     "shell.execute_reply": "2021-08-22T01:25:35.339800Z",
     "shell.execute_reply.started": "2021-08-22T01:08:02.556155Z"
    },
    "papermill": {
     "duration": 0.505142,
     "end_time": "2021-08-22T01:25:35.339999",
     "exception": false,
     "start_time": "2021-08-22T01:25:34.834857",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"2_PercentMissingValues\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Your code here!\n",
    "# DataFrame.values return a np.array\n",
    "percent_missing = sf_permits.isnull().values.mean() * 100\n",
    "\n",
    "# Check your answer\n",
    "q2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "05af6ef2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:35.393571Z",
     "iopub.status.busy": "2021-08-22T01:25:35.392834Z",
     "iopub.status.idle": "2021-08-22T01:25:35.397988Z",
     "shell.execute_reply": "2021-08-22T01:25:35.397418Z",
     "shell.execute_reply.started": "2021-08-22T01:10:39.881194Z"
    },
    "papermill": {
     "duration": 0.033113,
     "end_time": "2021-08-22T01:25:35.398156",
     "exception": false,
     "start_time": "2021-08-22T01:25:35.365043",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q2.hint()\n",
    "#q2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acc6df06",
   "metadata": {
    "papermill": {
     "duration": 0.026423,
     "end_time": "2021-08-22T01:25:35.450985",
     "exception": false,
     "start_time": "2021-08-22T01:25:35.424562",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Figure out why the data is missing\n",
    "\n",
    "Look at the columns **\"Street Number Suffix\"** and **\"Zipcode\"** from the [San Francisco Building Permits dataset](https://www.kaggle.com/aparnashastry/building-permit-applications-data). Both of these contain missing values. \n",
    "- Which, if either, are missing because they don't exist? \n",
    "- Which, if either, are missing because they weren't recorded?  \n",
    "\n",
    "Once you have an answer, run the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "eb35b3b1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:35.506212Z",
     "iopub.status.busy": "2021-08-22T01:25:35.505276Z",
     "iopub.status.idle": "2021-08-22T01:25:35.511872Z",
     "shell.execute_reply": "2021-08-22T01:25:35.512470Z",
     "shell.execute_reply.started": "2021-08-22T01:13:28.275917Z"
    },
    "papermill": {
     "duration": 0.035382,
     "end_time": "2021-08-22T01:25:35.512657",
     "exception": false,
     "start_time": "2021-08-22T01:25:35.477275",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"3_WhyDataMissing\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "If a value in the \"Street Number Suffix\" column is missing, it is likely because it does not exist. If a value in the \"Zipcode\" column is missing, it was not recorded."
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "If a value in the \"Street Number Suffix\" column is missing, it is likely because it does not exist. If a value in the \"Zipcode\" column is missing, it was not recorded."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "362d1d0f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:35.570616Z",
     "iopub.status.busy": "2021-08-22T01:25:35.569872Z",
     "iopub.status.idle": "2021-08-22T01:25:35.573542Z",
     "shell.execute_reply": "2021-08-22T01:25:35.572844Z",
     "shell.execute_reply.started": "2021-08-22T01:14:10.811219Z"
    },
    "papermill": {
     "duration": 0.035431,
     "end_time": "2021-08-22T01:25:35.573689",
     "exception": false,
     "start_time": "2021-08-22T01:25:35.538258",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Line below will give you a hint\n",
    "#q3.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab143e7",
   "metadata": {
    "papermill": {
     "duration": 0.025775,
     "end_time": "2021-08-22T01:25:35.625171",
     "exception": false,
     "start_time": "2021-08-22T01:25:35.599396",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4) Drop missing values: rows\n",
    "\n",
    "If you removed all of the rows of `sf_permits` with missing values, how many rows are left?\n",
    "\n",
    "**Note**: Do not change the value of `sf_permits` when checking this.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9658e814",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:35.684579Z",
     "iopub.status.busy": "2021-08-22T01:25:35.683784Z",
     "iopub.status.idle": "2021-08-22T01:25:36.168501Z",
     "shell.execute_reply": "2021-08-22T01:25:36.167923Z",
     "shell.execute_reply.started": "2021-08-22T01:15:11.595217Z"
    },
    "papermill": {
     "duration": 0.515885,
     "end_time": "2021-08-22T01:25:36.168663",
     "exception": false,
     "start_time": "2021-08-22T01:25:35.652778",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO: Your code here!\n",
    "sf_permits.dropna().shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e77e503",
   "metadata": {
    "papermill": {
     "duration": 0.028596,
     "end_time": "2021-08-22T01:25:36.222386",
     "exception": false,
     "start_time": "2021-08-22T01:25:36.193790",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Once you have an answer, run the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2493a3f8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:36.283314Z",
     "iopub.status.busy": "2021-08-22T01:25:36.282250Z",
     "iopub.status.idle": "2021-08-22T01:25:36.289003Z",
     "shell.execute_reply": "2021-08-22T01:25:36.289530Z",
     "shell.execute_reply.started": "2021-08-22T01:15:15.654495Z"
    },
    "papermill": {
     "duration": 0.039227,
     "end_time": "2021-08-22T01:25:36.289727",
     "exception": false,
     "start_time": "2021-08-22T01:25:36.250500",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"4_DropMissingRows\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct:</span> \n",
       "\n",
       "There are no rows remaining in the dataset!"
      ],
      "text/plain": [
       "Correct: \n",
       "\n",
       "There are no rows remaining in the dataset!"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q4.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "61e27ad1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:36.347535Z",
     "iopub.status.busy": "2021-08-22T01:25:36.346517Z",
     "iopub.status.idle": "2021-08-22T01:25:36.350858Z",
     "shell.execute_reply": "2021-08-22T01:25:36.350210Z",
     "shell.execute_reply.started": "2021-08-22T01:16:53.942847Z"
    },
    "papermill": {
     "duration": 0.035004,
     "end_time": "2021-08-22T01:25:36.351017",
     "exception": false,
     "start_time": "2021-08-22T01:25:36.316013",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Line below will give you a hint\n",
    "#q4.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d71e859d",
   "metadata": {
    "papermill": {
     "duration": 0.02962,
     "end_time": "2021-08-22T01:25:36.409453",
     "exception": false,
     "start_time": "2021-08-22T01:25:36.379833",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 5) Drop missing values: columns\n",
    "\n",
    "Now try removing all the columns with empty values.  \n",
    "- Create a new DataFrame called `sf_permits_with_na_dropped` that has all of the columns with empty values removed.  \n",
    "- How many columns were removed from the original `sf_permits` DataFrame? Use this number to set the value of the `dropped_columns` variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "95dcb347",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:36.472830Z",
     "iopub.status.busy": "2021-08-22T01:25:36.471575Z",
     "iopub.status.idle": "2021-08-22T01:25:37.004730Z",
     "shell.execute_reply": "2021-08-22T01:25:37.004131Z",
     "shell.execute_reply.started": "2021-08-22T01:17:33.200072Z"
    },
    "papermill": {
     "duration": 0.568125,
     "end_time": "2021-08-22T01:25:37.004895",
     "exception": false,
     "start_time": "2021-08-22T01:25:36.436770",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"5_DropMissingCols\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Your code here\n",
    "sf_permits_with_na_dropped = sf_permits.dropna(axis=1)\n",
    "\n",
    "dropped_columns = sf_permits.shape[1] - sf_permits_with_na_dropped.shape[1]\n",
    "\n",
    "# Check your answer\n",
    "q5.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ceac264b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:37.072172Z",
     "iopub.status.busy": "2021-08-22T01:25:37.071415Z",
     "iopub.status.idle": "2021-08-22T01:25:37.075095Z",
     "shell.execute_reply": "2021-08-22T01:25:37.074508Z",
     "shell.execute_reply.started": "2021-08-22T01:18:07.763117Z"
    },
    "papermill": {
     "duration": 0.038211,
     "end_time": "2021-08-22T01:25:37.075257",
     "exception": false,
     "start_time": "2021-08-22T01:25:37.037046",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q5.hint()\n",
    "#q5.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ff68c81",
   "metadata": {
    "papermill": {
     "duration": 0.027246,
     "end_time": "2021-08-22T01:25:37.133346",
     "exception": false,
     "start_time": "2021-08-22T01:25:37.106100",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 6) Fill in missing values automatically\n",
    "\n",
    "Try replacing all the NaN's in the `sf_permits` data with the one that comes directly after it and then replacing any remaining NaN's with 0.  Set the result to a new DataFrame `sf_permits_with_na_imputed`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a05d2f43",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:37.199366Z",
     "iopub.status.busy": "2021-08-22T01:25:37.198536Z",
     "iopub.status.idle": "2021-08-22T01:25:38.645756Z",
     "shell.execute_reply": "2021-08-22T01:25:38.645095Z",
     "shell.execute_reply.started": "2021-08-22T01:20:06.370900Z"
    },
    "papermill": {
     "duration": 1.484321,
     "end_time": "2021-08-22T01:25:38.645954",
     "exception": false,
     "start_time": "2021-08-22T01:25:37.161633",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.16666666666666666, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"6_ImputeAutomatically\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Your code here\n",
    "sf_permits_with_na_imputed = sf_permits.fillna(method='bfill').fillna(0)\n",
    "\n",
    "# Check your answer\n",
    "q6.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bf083441",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-08-22T01:25:38.709584Z",
     "iopub.status.busy": "2021-08-22T01:25:38.708793Z",
     "iopub.status.idle": "2021-08-22T01:25:38.712420Z",
     "shell.execute_reply": "2021-08-22T01:25:38.711664Z",
     "shell.execute_reply.started": "2021-08-22T01:20:28.635085Z"
    },
    "papermill": {
     "duration": 0.037189,
     "end_time": "2021-08-22T01:25:38.712579",
     "exception": false,
     "start_time": "2021-08-22T01:25:38.675390",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q6.hint()\n",
    "#q6.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59f38dc",
   "metadata": {
    "papermill": {
     "duration": 0.030153,
     "end_time": "2021-08-22T01:25:38.774467",
     "exception": false,
     "start_time": "2021-08-22T01:25:38.744314",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# More practice\n",
    "\n",
    "If you're looking for more practice handling missing values:\n",
    "\n",
    "* Check out [this noteboook](https://www.kaggle.com/alexisbcook/missing-values) on handling missing values using scikit-learn's imputer. \n",
    "* Look back at the \"Zipcode\" column in the `sf_permits` dataset, which has some missing values. How would you go about figuring out what the actual zipcode of each address should be? (You might try using another dataset. You can search for datasets about San Fransisco on the [Datasets listing](https://www.kaggle.com/datasets).) \n",
    "\n",
    "# Keep going\n",
    "\n",
    "In the next lesson, learn how to [**apply scaling and normalization**](https://www.kaggle.com/alexisbcook/scaling-and-normalization) to transform your data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78973503",
   "metadata": {
    "papermill": {
     "duration": 0.028688,
     "end_time": "2021-08-22T01:25:38.832424",
     "exception": false,
     "start_time": "2021-08-22T01:25:38.803736",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/172650) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 23.879311,
   "end_time": "2021-08-22T01:25:40.271330",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2021-08-22T01:25:16.392019",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}