3. Regression accuracy metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Expedition to Data Science and Machine Learning\n",
    "## Module 4: Machine Learning with Python\n",
    "### Lecture 3: Supervised Learning: Linear Regression and Regression accuracy metrics\n",
    "\n",
    "Instructor: Md Shahidullah Kawsar\n",
    "<br>Data Scientist, IDARE, Houston, TX, USA\n",
    "\n",
    "#### Objectives:\n",
    "- Supervised Learning: Linear Regression\n",
    "- Accuracy metric in Regression problem\n",
    "- Mean Absolute Error (MAE)\n",
    "- Mean Absolute Percentage Error (MAPE)\n",
    "- Mean Squared Error (MSE)\n",
    "- Root Mean Squared Error (RMSE)\n",
    "- R-squared or coefficient of determination\n",
    "- Prediction result evaluation\n",
    "\n",
    "#### References:\n",
    "[1] Accuracy metrics in sklearn: https://scikit-learn.org/stable/modules/model_evaluation.html\n",
    "<br>[2] Mean Absolute Error (MAE): https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error\n",
    "<br>[3] Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error\n",
    "<br>[4] R-squared or coefficient of determination: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score\n",
    "<br>[5] MAE, MSE, RMSE, Coefficient of Determination, Adjusted R Squared — Which Metric is Better? https://medium.com/analytics-vidhya/mae-mse-rmse-coefficient-of-determination-adjusted-r-squared-which-metric-is-better-cd0326a5697e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy metrics in Supervised Learning: Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "actual_value = [1,2,3,4,5,6,7,8,9,10]\n",
    "predicted_value = [1,3,4,5,6,5,6,5,8,9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mean absolute error** represents the average of the absolute difference between the actual and predicted values in the dataset. It measures the average of the residuals in the dataset.\n",
    "\n",
    "**Mean Squared Error** represents the average of the squared difference between the original and predicted values in the data set. It measures the variance of the residuals.\n",
    "\n",
    "**Root Mean Squared Error** is the square root of Mean Squared error. It measures the standard deviation of residuals.\n",
    "\n",
    "**Coefficient of determination or R-squared** represents the proportion of the variance in the dependent variable. It is a scale-free score i.e. irrespective of the values being small or large, the value of R square will be less than one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>actual</th>\n",
       "      <th>predicted</th>\n",
       "      <th>dif</th>\n",
       "      <th>abs_error</th>\n",
       "      <th>squared_error</th>\n",
       "      <th>actual_subtract_mean</th>\n",
       "      <th>squared_actual_subtract_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-4.5</td>\n",
       "      <td>20.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-3.5</td>\n",
       "      <td>12.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.5</td>\n",
       "      <td>6.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>2.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>2.5</td>\n",
       "      <td>6.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>12.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.5</td>\n",
       "      <td>20.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   actual  predicted  dif  abs_error  squared_error  actual_subtract_mean  \\\n",
       "0       1          1    0          0              0                  -4.5   \n",
       "1       2          3   -1          1              1                  -3.5   \n",
       "2       3          4   -1          1              1                  -2.5   \n",
       "3       4          5   -1          1              1                  -1.5   \n",
       "4       5          6   -1          1              1                  -0.5   \n",
       "5       6          5    1          1              1                   0.5   \n",
       "6       7          6    1          1              1                   1.5   \n",
       "7       8          5    3          3              9                   2.5   \n",
       "8       9          8    1          1              1                   3.5   \n",
       "9      10          9    1          1              1                   4.5   \n",
       "\n",
       "   squared_actual_subtract_mean  \n",
       "0                         20.25  \n",
       "1                         12.25  \n",
       "2                          6.25  \n",
       "3                          2.25  \n",
       "4                          0.25  \n",
       "5                          0.25  \n",
       "6                          2.25  \n",
       "7                          6.25  \n",
       "8                         12.25  \n",
       "9                         20.25  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.DataFrame({\"actual\":actual_value,\n",
    "                   \"predicted\": predicted_value})\n",
    "\n",
    "df[\"dif\"] = df[\"actual\"] - df[\"predicted\"]\n",
    "df[\"abs_error\"] = np.abs(df[\"dif\"])\n",
    "df[\"squared_error\"] = df[\"dif\"]**2\n",
    "\n",
    "df[\"actual_subtract_mean\"] = df[\"actual\"] - df[\"actual\"].mean()\n",
    "df[\"squared_actual_subtract_mean\"] = df[\"actual_subtract_mean\"]**2\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.5"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"actual\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE =  1.1\n",
      "MAPE =  21.79\n",
      "MSE =  1.7\n",
      "RMSE =  1.3\n",
      "r_squared =  0.79\n"
     ]
    }
   ],
   "source": [
    "# mean absolute error: lower is better\n",
    "MAE = df[\"abs_error\"].mean()\n",
    "print(\"MAE = \", MAE)\n",
    "\n",
    "# MAPE: Mean Absolute Percentage Error: lower is better\n",
    "MAPE = np.round(np.mean(df[\"abs_error\"]/df[\"actual\"])*100, 2)\n",
    "print(\"MAPE = \", MAPE)\n",
    "\n",
    "# mean squared error: lower is better\n",
    "MSE = df[\"squared_error\"].mean()\n",
    "print(\"MSE = \", MSE)\n",
    "\n",
    "# root mean squared error: lower is better\n",
    "RMSE = np.round(np.sqrt(MSE), 2)\n",
    "print(\"RMSE = \", RMSE)\n",
    "\n",
    "# coefficient of determination == r_squared: greater is better. Max =1, min=-\n",
    "r_squared = np.round(1- df[\"squared_error\"].sum()/df[\"squared_actual_subtract_mean\"].sum(), 2)\n",
    "print(\"r_squared = \", r_squared)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}