05_Linear_Regression_Recap.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data handling/modeling\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "import scipy.stats as stats\n",
    "\n",
    "# visualization\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading in the car dataset\n",
    "\n",
    "Now that we've got all of the libraries we need, lets get some data to work with.\n",
    "\n",
    "This data comes from the famous [UC Irvine Machine Learning Repository](http://archive.ics.uci.edu/ml/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cylinders         int64\n",
      "displacement    float64\n",
      "horsepower      float64\n",
      "weight          float64\n",
      "acceleration    float64\n",
      "mpg             float64\n",
      "dtype: object\n",
      "(392, 6)\n"
     ]
    }
   ],
   "source": [
    "# read data into a DataFrame\n",
    "data = pd.read_csv(\"./data/auto_mpg_data.csv\")\n",
    "print(data.dtypes)\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>horsepower</th>\n",
       "      <th>weight</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>mpg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>3504.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>165.0</td>\n",
       "      <td>3693.0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>318.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3436.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>304.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>3433.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>302.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>3449.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   cylinders  displacement  horsepower  weight  acceleration   mpg\n",
       "0          8         307.0       130.0  3504.0          12.0  18.0\n",
       "1          8         350.0       165.0  3693.0          11.5  15.0\n",
       "2          8         318.0       150.0  3436.0          11.0  18.0\n",
       "3          8         304.0       150.0  3433.0          12.0  16.0\n",
       "4          8         302.0       140.0  3449.0          10.5  17.0"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assumptions of Linear Regression\n",
    "\n",
    "Linear regression is a method of assessing whether one or more predictor variable explains a dependent variable. For the model results to be valid, these assumptions must be met:\n",
    "\n",
    "1. Linear relationship: The relationship between the predictor and dependent variables must be linear.\n",
    "2. Multivariate normality: all variables should be normally distributed\n",
    "3. No or little multicollinearity: The predictor variables cannot be correlated to each other, aka the values move similarly to each other\n",
    "4. Independence of errors: the residuals (errors) are independent of each other, no auto-correlation\n",
    "5. Homoscedasticity (constant variance): The magnitude of the residuals (errors) stays the same over the entire regression line \n",
    "\n",
    "Check out [this article](https://www.statisticssolutions.com/assumptions-of-linear-regression/) for methods of testing these assumptions: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Linear Regression with One Variable "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the simplest representation of a linear regression. Linear regression on one variable is an approach for predicting a **continuous response** using a **single feature**. Here's the mathematical way of representing a simple linear regression model:\n",
    "\n",
    "$y = \\beta_0 + \\beta_1x$\n",
    "\n",
    "- $y$ is the response\n",
    "- $x$ is the feature\n",
    "- $\\beta_0$ is the intercept\n",
    "- $\\beta_1$ is the coefficient for x\n",
    "\n",
    "$\\beta_0$ and $\\beta_1$ are called the **model coefficients**. They are the values of the model that we are going to attempt to estimate $y$.\n",
    "\n",
    "$\\beta_0$ is also called the bias (its an offset), and is equivalent to the y-intercept of the model\n",
    "\n",
    "\n",
    "So, **our model must \"learn\" the values of these coefficients, and once we've learned these coefficients, we can use the model to predict mpg.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating (\"learning\") simple linear regression model coefficients\n",
    "\n",
    "Coefficients are estimated during the model fitting process using the **least squares estimation**.\n",
    "\n",
    "We will find the line which minimizes the **sum of squared errors**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Estimating coefficients](./images/least_squares.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the following in the diagram:\n",
    "  * The black dots are the **observed values** of x and y.\n",
    "  * The thin black line is our **least squares line**.\n",
    "  * The red line is an example **residual** or **error**, which is the distance between an observed value and the least squares line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do the model coefficients relate to the least squares line?\n",
    "\n",
    "- $\\beta_0$ is the **intercept** (the value of $y$ when $x$=0)\n",
    "- $\\beta_1$ is the **slope** ($\\Delta y/\\Delta x$)\n",
    "\n",
    "Let's estimate the model coefficients for our car data where we will use the `acceleration` of the car as our single feature to predict `mpg`.\n",
    "\n",
    "We will use `scikit-learn` for the first time here and work through the process of training a scikit-learn model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The y intercept: 4.8332498048437955\n",
      "The single coefficient: [('acceleration', 1.1976241877320561)]\n",
      "R^2:  0.1792070501562546\n",
      "MSE:  49.87362732665226\n"
     ]
    }
   ],
   "source": [
    "# create X and y\n",
    "feature_col = ['acceleration']\n",
    "X = data[feature_cols]\n",
    "y = data.mpg\n",
    "\n",
    "# instantiate and fit\n",
    "linreg = LinearRegression()\n",
    "linreg.fit(X, y)\n",
    "\n",
    "# print the coefficients\n",
    "print(\"The y intercept:\", linreg.intercept_)\n",
    "print(\"The single coefficient:\", list(zip(feature_col,linreg.coef_)))\n",
    "\n",
    "# evaluate R^2\n",
    "y_pred = linreg.predict(X)\n",
    "print(\"R^2: \", metrics.r2_score(y, y_pred))\n",
    "\n",
    "# Evaluate MSE\n",
    "print(\"MSE: \", metrics.mean_squared_error(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Linear Regression\n",
    "\n",
    "Simple linear regression can easily be extended to include multiple features, which is called **multiple linear regression**:\n",
    "\n",
    "$y = \\beta_0 + \\beta_1x_1 + ... + \\beta_nx_n$\n",
    "\n",
    "Each $x$ represents a different feature, and each feature has its own coefficient:\n",
    "\n",
    "$y = \\beta_0 + \\beta_1 \\times acceleration + \\beta_2 \\times displacement + \\beta_3 \\times horsepower$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The y intercept: 46.25470749685908\n"
     ]
    }
   ],
   "source": [
    "# create X and y except now with more columns in X\n",
    "feature_list = ['acceleration', 'displacement', 'horsepower']\n",
    "X_mult = data[feature_list]\n",
    "y_mult = data.mpg\n",
    "\n",
    "# instantiate and fit like last time\n",
    "mult_linreg = LinearRegression()\n",
    "mult_linreg.fit(X_mult, y_mult)\n",
    "\n",
    "# print intercept\n",
    "print(\"The y intercept:\", mult_linreg.intercept_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('acceleration', -0.41222985131176515),\n",
       " ('displacement', -0.036659952890354765),\n",
       " ('horsepower', -0.08878252487814355)]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pair the feature names with the coefficients\n",
    "list(zip(feature_list, mult_linreg.coef_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this model we can interpret the coefficients as follows:\n",
    "\n",
    "  * For a fixed amount of acceleration and engine displacement, an increase of 1 unit in **horsepower** is associated with a **decrease in mpg of the car of ~.09**.\n",
    "  * For a fixed amount of displacement and horsepower, an increase of 1 m/s^2 in **acceleration** is associated with a **decrease in mpg of ~.41**.\n",
    "  * For a fixed amount of acceleration and horsepower, an increase of 1 in **displacement** is associated with an **decrease in mpg of ~.04**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does this model have a better r^2 value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R^2:  0.6748704313006708\n",
      "MSE:  49.87362732665226\n"
     ]
    }
   ],
   "source": [
    "# Evaluate R^2\n",
    "y_mult_pred = mult_linreg.predict(X_mult)\n",
    "print(\"R^2: \", metrics.r2_score(y_mult, y_mult_pred))\n",
    "\n",
    "# Evaluate MSE\n",
    "print(\"MSE: \", metrics.mean_squared_error(y, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation metrics for regression problems\n",
    "\n",
    "In order to evaluate how good a given regression model is, we use evaluation metrics designed for comparing **continuous values**. The 3 common evaluation metrics for regression models are here.\n",
    "\n",
    "Let's create some example numeric predictions, and calculate the three most common evaluation metrics for regression problems:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define true and predicted response values\n",
    "fake_y_true = [101, 40, 30, 20]\n",
    "fake_y_pred = [90, 50, 50, 30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mean Absolute Error** (MAE) is the mean of the absolute value of the errors/residuals:\n",
    "\n",
    "$$\\frac 1n\\sum_{i=1}^n|y_i-\\hat{y}_i|$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE for fake data: 12.75\n"
     ]
    }
   ],
   "source": [
    "print(\"MAE for fake data:\",metrics.mean_absolute_error(fake_y_true, fake_y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mean Squared Error** (MSE) is the mean of the squared errors:\n",
    "\n",
    "$$\\frac 1n\\sum_{i=1}^n(y_i-\\hat{y}_i)^2$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE for fake data: 180.25\n"
     ]
    }
   ],
   "source": [
    "print(\"MSE for fake data:\",metrics.mean_squared_error(fake_y_true,fake_y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Root Mean Squared Error** (RMSE) is the square root of the mean of the squared errors:\n",
    "\n",
    "$$\\sqrt{\\frac 1n\\sum_{i=1}^n(y_i-\\hat{y}_i)^2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE for fake data: 13.425721582097552\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE for fake data:\",np.sqrt(metrics.mean_squared_error(fake_y_true, fake_y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets compare these metrics in terms of their usefulness/interpretability:\n",
    "  * **MAE** is the easiest to understand, because it's the average error.\n",
    "  * **MSE** is more popular than MAE, because MSE \"punishes\" larger errors, which tends to be useful in the real world.\n",
    "  * **RMSE** is even more popular than MSE, because RMSE is interpretable in the \"y\" units.\n",
    "\n",
    "All of these are what are called **loss functions**, because we want to minimize the **loss** (from getting stuff wrong)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using train/test split for model evaluation\n",
    "\n",
    "How do we know that our model will perform well on new data?\n",
    "\n",
    "Sure, we may know that our model has really low RMSE on all of the data we have on hand, but can we be sure that it will be exactly the same when we try to use our model in the real world?\n",
    "\n",
    "One way we can get an estimate of how the model will perform \"in the wild\" is by building the model on a portion of our data, and then testing it on the remainder that we have.\n",
    "\n",
    "So, we **act like we have one set of data for model building, and keep a separate set of data and treat it as if it were new.** We then test our model on this \"new\" data, and, **as long as the test data was taken in an unbiased way**, we can assume that the **loss** on the test data gives us a pretty good idea of what the error \"in the wild\" will be.\n",
    "\n",
    "So, let's try to use train/test split to estimate the model's accuracy on unseen data.\n",
    "\n",
    "The basic approach would be to randomly select a fraction of the data (>50% usually) for training, and the remainder (100-training%) for testing. We will use scikit-learn's `train_test_split` function to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training data size: (274, 3)\n",
      "testing data size: (118, 3)\n"
     ]
    }
   ],
   "source": [
    "X_mult_train, X_mult_test, y_mult_train, y_mult_test = train_test_split(X_mult, y_mult, test_size=0.3, random_state=1)\n",
    "print(\"training data size:\",X_mult_train.shape)\n",
    "print(\"testing data size:\",X_mult_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we simply train on `X_mult_train` and `y_mult_train` and then generate predictions and evaluation metrics on `X_mult_test` and `y_mult_test`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set RMSE: 4.41382089426587\n",
      "Test set RMSE: 4.604331577231554\n"
     ]
    }
   ],
   "source": [
    "#train on training set\n",
    "mult_linreg2 = LinearRegression()\n",
    "mult_linreg2.fit(X_mult_train, y_mult_train)\n",
    "\n",
    "#generate predictions on training set and evaluate\n",
    "y_mult_pred_train = mult_linreg2.predict(X_mult_train)\n",
    "print(\"Training set RMSE:\",np.sqrt(metrics.mean_squared_error(y_mult_train, y_mult_pred_train)))\n",
    "\n",
    "#generate predictions on test set and evaluate\n",
    "y_mult_pred_test = mult_linreg2.predict(X_mult_test)\n",
    "print(\"Test set RMSE:\",np.sqrt(metrics.mean_squared_error(y_mult_test, y_mult_pred_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the test set error is greater than the training set error. This should always be the case (why?)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise Time\n",
    "\n",
    "  * Get MAE/MSE/RMSE training and test set predictions on the full linear regression model (using all features) with a test set of 30% of the data\n",
    "  * Get MAE/MSE/RMSE training and test set predictions on the full linear regression model (using all features) with a test set of 20% of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of linear regression and comparison with other models (you will see in the future)\n",
    "\n",
    "There are some obvious advantages to linear regression models:\n",
    "  * These kinds of models are very simple to explain\n",
    "  * They are highly interpretable\n",
    "  * Model training and prediction is very fast\n",
    "  * Features do not need to be scaled (we will talk about feature scaling later)\n",
    "  * They can perform well with a small number of observations\n",
    "\n",
    "However, linear regression also has some significant disadvantages:\n",
    "  * It assumes a linear relationship between the features and the outcome. This isn't always (almost never) the case.\n",
    "  * Performance is (generally) not competitive with the best supervised learning methods\n",
    "  * When you have lots of features, this approach can become sensitive to useless features"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}