5. Classification: Accuracy Metrics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 5: Supervised Learning - Classification: Accuracy Metrics\n",
    "\n",
    "Instructor: Md Shahidullah Kawsar\n",
    "<br>Data Scientist, IDARE, Houston, TX, USA\n",
    "\n",
    "#### Objectives:\n",
    "- Supervised Learning - Classification: Logistic Regression\n",
    "- Confusion Matrix\n",
    "- Accuracy, Precision, Recall/Sensitivity/True Positive Rate, F1 score, False Positive Rate\n",
    "- ROC: Receiver Operating Characterisitcs and AUC: Area Under the Curve\n",
    "- Classification report\n",
    "\n",
    "#### References:\n",
    "[1] Machine Learning Fundamentals: The Confusion Matrix - https://www.youtube.com/watch?v=Kdsp6soqA7o&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[2] Machine Learning Fundamentals: Sensitivity and Specificity: https://www.youtube.com/watch?v=sunUKFXMHGk&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[3] ROC and AUC, Clearly Explained! https://www.youtube.com/watch?v=4jRBRDbJemM&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[4] Machine Learning Fundamentals: Bias and Variance: https://www.youtube.com/watch?v=EuBBz3bI-aA&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=5&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[5] StatQuest: Logistic Regression - https://www.youtube.com/watch?v=yIYKR4sgzI8&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[6] Logistic Regression Details Pt1: Coefficients: https://www.youtube.com/watch?v=vN5cNN2-HWE&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[7] Logistic Regression Details Pt 2: Maximum Likelihood: https://www.youtube.com/watch?v=BfKanl1aSG0&t=163s&ab_channel=StatQuestwithJoshStarmer\n",
    "<br>[8] Machine Learning Tutorial Python - 8: Logistic Regression (Binary Classification): https://www.youtube.com/watch?v=zM4VZR0px8E&ab_channel=codebasics\n",
    "<br>[9] Maximum Likelihood, clearly explained!!!: https://www.youtube.com/watch?v=XepXtl9YKwc&ab_channel=StatQuestwithJoshStarmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "pd.options.display.max_columns = 20\n",
    "# pd.options.display.max_rows = 100\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "# from sklearn.tree import DecisionTreeClassifier\n",
    "# from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
    "from sklearn.metrics import roc_curve, roc_auc_score, precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load data\n",
    "Dataset Source: https://learn.datacamp.com/courses/marketing-analytics-predicting-customer-churn-in-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Account_Length</th>\n",
       "      <th>Vmail_Message</th>\n",
       "      <th>Day_Mins</th>\n",
       "      <th>Eve_Mins</th>\n",
       "      <th>Night_Mins</th>\n",
       "      <th>Intl_Mins</th>\n",
       "      <th>CustServ_Calls</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Intl_Plan</th>\n",
       "      <th>Vmail_Plan</th>\n",
       "      <th>...</th>\n",
       "      <th>Day_Charge</th>\n",
       "      <th>Eve_Calls</th>\n",
       "      <th>Eve_Charge</th>\n",
       "      <th>Night_Calls</th>\n",
       "      <th>Night_Charge</th>\n",
       "      <th>Intl_Calls</th>\n",
       "      <th>Intl_Charge</th>\n",
       "      <th>State</th>\n",
       "      <th>Area_Code</th>\n",
       "      <th>Phone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>128</td>\n",
       "      <td>25</td>\n",
       "      <td>265.1</td>\n",
       "      <td>197.4</td>\n",
       "      <td>244.7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>...</td>\n",
       "      <td>45.07</td>\n",
       "      <td>99</td>\n",
       "      <td>16.78</td>\n",
       "      <td>91</td>\n",
       "      <td>11.01</td>\n",
       "      <td>3</td>\n",
       "      <td>2.70</td>\n",
       "      <td>KS</td>\n",
       "      <td>415</td>\n",
       "      <td>382-4657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "      <td>26</td>\n",
       "      <td>161.6</td>\n",
       "      <td>195.5</td>\n",
       "      <td>254.4</td>\n",
       "      <td>13.7</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>...</td>\n",
       "      <td>27.47</td>\n",
       "      <td>103</td>\n",
       "      <td>16.62</td>\n",
       "      <td>103</td>\n",
       "      <td>11.45</td>\n",
       "      <td>3</td>\n",
       "      <td>3.70</td>\n",
       "      <td>OH</td>\n",
       "      <td>415</td>\n",
       "      <td>371-7191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>137</td>\n",
       "      <td>0</td>\n",
       "      <td>243.4</td>\n",
       "      <td>121.2</td>\n",
       "      <td>162.6</td>\n",
       "      <td>12.2</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>...</td>\n",
       "      <td>41.38</td>\n",
       "      <td>110</td>\n",
       "      <td>10.30</td>\n",
       "      <td>104</td>\n",
       "      <td>7.32</td>\n",
       "      <td>5</td>\n",
       "      <td>3.29</td>\n",
       "      <td>NJ</td>\n",
       "      <td>415</td>\n",
       "      <td>358-1921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>299.4</td>\n",
       "      <td>61.9</td>\n",
       "      <td>196.9</td>\n",
       "      <td>6.6</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>...</td>\n",
       "      <td>50.90</td>\n",
       "      <td>88</td>\n",
       "      <td>5.26</td>\n",
       "      <td>89</td>\n",
       "      <td>8.86</td>\n",
       "      <td>7</td>\n",
       "      <td>1.78</td>\n",
       "      <td>OH</td>\n",
       "      <td>408</td>\n",
       "      <td>375-9999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>166.7</td>\n",
       "      <td>148.3</td>\n",
       "      <td>186.9</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>...</td>\n",
       "      <td>28.34</td>\n",
       "      <td>122</td>\n",
       "      <td>12.61</td>\n",
       "      <td>121</td>\n",
       "      <td>8.41</td>\n",
       "      <td>3</td>\n",
       "      <td>2.73</td>\n",
       "      <td>OK</td>\n",
       "      <td>415</td>\n",
       "      <td>330-6626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>118</td>\n",
       "      <td>0</td>\n",
       "      <td>223.4</td>\n",
       "      <td>220.6</td>\n",
       "      <td>203.9</td>\n",
       "      <td>6.3</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>...</td>\n",
       "      <td>37.98</td>\n",
       "      <td>101</td>\n",
       "      <td>18.75</td>\n",
       "      <td>118</td>\n",
       "      <td>9.18</td>\n",
       "      <td>6</td>\n",
       "      <td>1.70</td>\n",
       "      <td>AL</td>\n",
       "      <td>510</td>\n",
       "      <td>391-8027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>121</td>\n",
       "      <td>24</td>\n",
       "      <td>218.2</td>\n",
       "      <td>348.5</td>\n",
       "      <td>212.6</td>\n",
       "      <td>7.5</td>\n",
       "      <td>3</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>...</td>\n",
       "      <td>37.09</td>\n",
       "      <td>108</td>\n",
       "      <td>29.62</td>\n",
       "      <td>118</td>\n",
       "      <td>9.57</td>\n",
       "      <td>7</td>\n",
       "      <td>2.03</td>\n",
       "      <td>MA</td>\n",
       "      <td>510</td>\n",
       "      <td>355-9993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>147</td>\n",
       "      <td>0</td>\n",
       "      <td>157.0</td>\n",
       "      <td>103.1</td>\n",
       "      <td>211.8</td>\n",
       "      <td>7.1</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>...</td>\n",
       "      <td>26.69</td>\n",
       "      <td>94</td>\n",
       "      <td>8.76</td>\n",
       "      <td>96</td>\n",
       "      <td>9.53</td>\n",
       "      <td>6</td>\n",
       "      <td>1.92</td>\n",
       "      <td>MO</td>\n",
       "      <td>415</td>\n",
       "      <td>329-9001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>117</td>\n",
       "      <td>0</td>\n",
       "      <td>184.5</td>\n",
       "      <td>351.6</td>\n",
       "      <td>215.8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>...</td>\n",
       "      <td>31.37</td>\n",
       "      <td>80</td>\n",
       "      <td>29.89</td>\n",
       "      <td>90</td>\n",
       "      <td>9.71</td>\n",
       "      <td>4</td>\n",
       "      <td>2.35</td>\n",
       "      <td>LA</td>\n",
       "      <td>408</td>\n",
       "      <td>335-4719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>141</td>\n",
       "      <td>37</td>\n",
       "      <td>258.6</td>\n",
       "      <td>222.0</td>\n",
       "      <td>326.4</td>\n",
       "      <td>11.2</td>\n",
       "      <td>0</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>...</td>\n",
       "      <td>43.96</td>\n",
       "      <td>111</td>\n",
       "      <td>18.87</td>\n",
       "      <td>97</td>\n",
       "      <td>14.69</td>\n",
       "      <td>5</td>\n",
       "      <td>3.02</td>\n",
       "      <td>WV</td>\n",
       "      <td>415</td>\n",
       "      <td>330-8173</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
       "0             128             25     265.1     197.4       244.7       10.0   \n",
       "1             107             26     161.6     195.5       254.4       13.7   \n",
       "2             137              0     243.4     121.2       162.6       12.2   \n",
       "3              84              0     299.4      61.9       196.9        6.6   \n",
       "4              75              0     166.7     148.3       186.9       10.1   \n",
       "5             118              0     223.4     220.6       203.9        6.3   \n",
       "6             121             24     218.2     348.5       212.6        7.5   \n",
       "7             147              0     157.0     103.1       211.8        7.1   \n",
       "8             117              0     184.5     351.6       215.8        8.7   \n",
       "9             141             37     258.6     222.0       326.4       11.2   \n",
       "\n",
       "   CustServ_Calls Churn Intl_Plan Vmail_Plan  ...  Day_Charge  Eve_Calls  \\\n",
       "0               1    no        no        yes  ...       45.07         99   \n",
       "1               1    no        no        yes  ...       27.47        103   \n",
       "2               0    no        no         no  ...       41.38        110   \n",
       "3               2    no       yes         no  ...       50.90         88   \n",
       "4               3    no       yes         no  ...       28.34        122   \n",
       "5               0    no       yes         no  ...       37.98        101   \n",
       "6               3    no        no        yes  ...       37.09        108   \n",
       "7               0    no       yes         no  ...       26.69         94   \n",
       "8               1    no        no         no  ...       31.37         80   \n",
       "9               0    no       yes        yes  ...       43.96        111   \n",
       "\n",
       "   Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  State  \\\n",
       "0       16.78           91         11.01           3         2.70     KS   \n",
       "1       16.62          103         11.45           3         3.70     OH   \n",
       "2       10.30          104          7.32           5         3.29     NJ   \n",
       "3        5.26           89          8.86           7         1.78     OH   \n",
       "4       12.61          121          8.41           3         2.73     OK   \n",
       "5       18.75          118          9.18           6         1.70     AL   \n",
       "6       29.62          118          9.57           7         2.03     MA   \n",
       "7        8.76           96          9.53           6         1.92     MO   \n",
       "8       29.89           90          9.71           4         2.35     LA   \n",
       "9       18.87           97         14.69           5         3.02     WV   \n",
       "\n",
       "  Area_Code     Phone  \n",
       "0       415  382-4657  \n",
       "1       415  371-7191  \n",
       "2       415  358-1921  \n",
       "3       408  375-9999  \n",
       "4       415  330-6626  \n",
       "5       510  391-8027  \n",
       "6       510  355-9993  \n",
       "7       415  329-9001  \n",
       "8       408  335-4719  \n",
       "9       415  330-8173  \n",
       "\n",
       "[10 rows x 21 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.read_csv(\"Churn.csv\")\n",
    "\n",
    "display(df.head(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3333, 21)\n"
     ]
    }
   ],
   "source": [
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 3333 entries, 0 to 3332\n",
      "Data columns (total 21 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Account_Length  3333 non-null   int64  \n",
      " 1   Vmail_Message   3333 non-null   int64  \n",
      " 2   Day_Mins        3333 non-null   float64\n",
      " 3   Eve_Mins        3333 non-null   float64\n",
      " 4   Night_Mins      3333 non-null   float64\n",
      " 5   Intl_Mins       3333 non-null   float64\n",
      " 6   CustServ_Calls  3333 non-null   int64  \n",
      " 7   Churn           3333 non-null   object \n",
      " 8   Intl_Plan       3333 non-null   object \n",
      " 9   Vmail_Plan      3333 non-null   object \n",
      " 10  Day_Calls       3333 non-null   int64  \n",
      " 11  Day_Charge      3333 non-null   float64\n",
      " 12  Eve_Calls       3333 non-null   int64  \n",
      " 13  Eve_Charge      3333 non-null   float64\n",
      " 14  Night_Calls     3333 non-null   int64  \n",
      " 15  Night_Charge    3333 non-null   float64\n",
      " 16  Intl_Calls      3333 non-null   int64  \n",
      " 17  Intl_Charge     3333 non-null   float64\n",
      " 18  State           3333 non-null   object \n",
      " 19  Area_Code       3333 non-null   int64  \n",
      " 20  Phone           3333 non-null   object \n",
      "dtypes: float64(8), int64(8), object(5)\n",
      "memory usage: 546.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['no' 'yes']\n",
      "['no' 'yes']\n",
      "['yes' 'no']\n"
     ]
    }
   ],
   "source": [
    "print(df['Churn'].unique())\n",
    "print(df['Intl_Plan'].unique())\n",
    "print(df['Vmail_Plan'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Account_Length</th>\n",
       "      <th>Vmail_Message</th>\n",
       "      <th>Day_Mins</th>\n",
       "      <th>Eve_Mins</th>\n",
       "      <th>Night_Mins</th>\n",
       "      <th>Intl_Mins</th>\n",
       "      <th>CustServ_Calls</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Intl_Plan</th>\n",
       "      <th>Vmail_Plan</th>\n",
       "      <th>Day_Calls</th>\n",
       "      <th>Day_Charge</th>\n",
       "      <th>Eve_Calls</th>\n",
       "      <th>Eve_Charge</th>\n",
       "      <th>Night_Calls</th>\n",
       "      <th>Night_Charge</th>\n",
       "      <th>Intl_Calls</th>\n",
       "      <th>Intl_Charge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>128</td>\n",
       "      <td>25</td>\n",
       "      <td>265.1</td>\n",
       "      <td>197.4</td>\n",
       "      <td>244.7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>110</td>\n",
       "      <td>45.07</td>\n",
       "      <td>99</td>\n",
       "      <td>16.78</td>\n",
       "      <td>91</td>\n",
       "      <td>11.01</td>\n",
       "      <td>3</td>\n",
       "      <td>2.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "      <td>26</td>\n",
       "      <td>161.6</td>\n",
       "      <td>195.5</td>\n",
       "      <td>254.4</td>\n",
       "      <td>13.7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>123</td>\n",
       "      <td>27.47</td>\n",
       "      <td>103</td>\n",
       "      <td>16.62</td>\n",
       "      <td>103</td>\n",
       "      <td>11.45</td>\n",
       "      <td>3</td>\n",
       "      <td>3.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>137</td>\n",
       "      <td>0</td>\n",
       "      <td>243.4</td>\n",
       "      <td>121.2</td>\n",
       "      <td>162.6</td>\n",
       "      <td>12.2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>114</td>\n",
       "      <td>41.38</td>\n",
       "      <td>110</td>\n",
       "      <td>10.30</td>\n",
       "      <td>104</td>\n",
       "      <td>7.32</td>\n",
       "      <td>5</td>\n",
       "      <td>3.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>299.4</td>\n",
       "      <td>61.9</td>\n",
       "      <td>196.9</td>\n",
       "      <td>6.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71</td>\n",
       "      <td>50.90</td>\n",
       "      <td>88</td>\n",
       "      <td>5.26</td>\n",
       "      <td>89</td>\n",
       "      <td>8.86</td>\n",
       "      <td>7</td>\n",
       "      <td>1.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>166.7</td>\n",
       "      <td>148.3</td>\n",
       "      <td>186.9</td>\n",
       "      <td>10.1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113</td>\n",
       "      <td>28.34</td>\n",
       "      <td>122</td>\n",
       "      <td>12.61</td>\n",
       "      <td>121</td>\n",
       "      <td>8.41</td>\n",
       "      <td>3</td>\n",
       "      <td>2.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>118</td>\n",
       "      <td>0</td>\n",
       "      <td>223.4</td>\n",
       "      <td>220.6</td>\n",
       "      <td>203.9</td>\n",
       "      <td>6.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>98</td>\n",
       "      <td>37.98</td>\n",
       "      <td>101</td>\n",
       "      <td>18.75</td>\n",
       "      <td>118</td>\n",
       "      <td>9.18</td>\n",
       "      <td>6</td>\n",
       "      <td>1.70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>121</td>\n",
       "      <td>24</td>\n",
       "      <td>218.2</td>\n",
       "      <td>348.5</td>\n",
       "      <td>212.6</td>\n",
       "      <td>7.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>88</td>\n",
       "      <td>37.09</td>\n",
       "      <td>108</td>\n",
       "      <td>29.62</td>\n",
       "      <td>118</td>\n",
       "      <td>9.57</td>\n",
       "      <td>7</td>\n",
       "      <td>2.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>147</td>\n",
       "      <td>0</td>\n",
       "      <td>157.0</td>\n",
       "      <td>103.1</td>\n",
       "      <td>211.8</td>\n",
       "      <td>7.1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>79</td>\n",
       "      <td>26.69</td>\n",
       "      <td>94</td>\n",
       "      <td>8.76</td>\n",
       "      <td>96</td>\n",
       "      <td>9.53</td>\n",
       "      <td>6</td>\n",
       "      <td>1.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>117</td>\n",
       "      <td>0</td>\n",
       "      <td>184.5</td>\n",
       "      <td>351.6</td>\n",
       "      <td>215.8</td>\n",
       "      <td>8.7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>97</td>\n",
       "      <td>31.37</td>\n",
       "      <td>80</td>\n",
       "      <td>29.89</td>\n",
       "      <td>90</td>\n",
       "      <td>9.71</td>\n",
       "      <td>4</td>\n",
       "      <td>2.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>141</td>\n",
       "      <td>37</td>\n",
       "      <td>258.6</td>\n",
       "      <td>222.0</td>\n",
       "      <td>326.4</td>\n",
       "      <td>11.2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>84</td>\n",
       "      <td>43.96</td>\n",
       "      <td>111</td>\n",
       "      <td>18.87</td>\n",
       "      <td>97</td>\n",
       "      <td>14.69</td>\n",
       "      <td>5</td>\n",
       "      <td>3.02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
       "0             128             25     265.1     197.4       244.7       10.0   \n",
       "1             107             26     161.6     195.5       254.4       13.7   \n",
       "2             137              0     243.4     121.2       162.6       12.2   \n",
       "3              84              0     299.4      61.9       196.9        6.6   \n",
       "4              75              0     166.7     148.3       186.9       10.1   \n",
       "5             118              0     223.4     220.6       203.9        6.3   \n",
       "6             121             24     218.2     348.5       212.6        7.5   \n",
       "7             147              0     157.0     103.1       211.8        7.1   \n",
       "8             117              0     184.5     351.6       215.8        8.7   \n",
       "9             141             37     258.6     222.0       326.4       11.2   \n",
       "\n",
       "   CustServ_Calls  Churn  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
       "0               1      0          0           1        110       45.07   \n",
       "1               1      0          0           1        123       27.47   \n",
       "2               0      0          0           0        114       41.38   \n",
       "3               2      0          1           0         71       50.90   \n",
       "4               3      0          1           0        113       28.34   \n",
       "5               0      0          1           0         98       37.98   \n",
       "6               3      0          0           1         88       37.09   \n",
       "7               0      0          1           0         79       26.69   \n",
       "8               1      0          0           0         97       31.37   \n",
       "9               0      0          1           1         84       43.96   \n",
       "\n",
       "   Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  \n",
       "0         99       16.78           91         11.01           3         2.70  \n",
       "1        103       16.62          103         11.45           3         3.70  \n",
       "2        110       10.30          104          7.32           5         3.29  \n",
       "3         88        5.26           89          8.86           7         1.78  \n",
       "4        122       12.61          121          8.41           3         2.73  \n",
       "5        101       18.75          118          9.18           6         1.70  \n",
       "6        108       29.62          118          9.57           7         2.03  \n",
       "7         94        8.76           96          9.53           6         1.92  \n",
       "8         80       29.89           90          9.71           4         2.35  \n",
       "9        111       18.87           97         14.69           5         3.02  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3333, 18)\n"
     ]
    }
   ],
   "source": [
    "df = df.drop(['State', 'Area_Code', 'Phone'], axis=1)\n",
    "\n",
    "df['Churn'] = df['Churn'].replace(({'no':0, 'yes':1}))\n",
    "df['Intl_Plan'] = df['Intl_Plan'].replace(({'no':0, 'yes':1}))\n",
    "df['Vmail_Plan'] = df['Vmail_Plan'].replace(({'no':0, 'yes':1}))\n",
    "\n",
    "display(df.head(10))\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 3333 entries, 0 to 3332\n",
      "Data columns (total 18 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   Account_Length  3333 non-null   int64  \n",
      " 1   Vmail_Message   3333 non-null   int64  \n",
      " 2   Day_Mins        3333 non-null   float64\n",
      " 3   Eve_Mins        3333 non-null   float64\n",
      " 4   Night_Mins      3333 non-null   float64\n",
      " 5   Intl_Mins       3333 non-null   float64\n",
      " 6   CustServ_Calls  3333 non-null   int64  \n",
      " 7   Churn           3333 non-null   int64  \n",
      " 8   Intl_Plan       3333 non-null   int64  \n",
      " 9   Vmail_Plan      3333 non-null   int64  \n",
      " 10  Day_Calls       3333 non-null   int64  \n",
      " 11  Day_Charge      3333 non-null   float64\n",
      " 12  Eve_Calls       3333 non-null   int64  \n",
      " 13  Eve_Charge      3333 non-null   float64\n",
      " 14  Night_Calls     3333 non-null   int64  \n",
      " 15  Night_Charge    3333 non-null   float64\n",
      " 16  Intl_Calls      3333 non-null   int64  \n",
      " 17  Intl_Charge     3333 non-null   float64\n",
      "dtypes: float64(8), int64(10)\n",
      "memory usage: 468.8 KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Target variable "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2850\n",
       "1     483\n",
       "Name: Churn, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Churn'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature and target variable separation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3333, 17) (3333, 1)\n"
     ]
    }
   ],
   "source": [
    "X = df.drop('Churn', axis=1)\n",
    "y = df[['Churn']]\n",
    "\n",
    "print(X.shape, y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train test separation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2333, 17) (1000, 17) (2333, 1) (1000, 1)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, \n",
    "                                                    y, \n",
    "                                                    test_size=0.3, \n",
    "                                                    random_state=42, \n",
    "                                                    stratify=y)\n",
    "\n",
    "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    855\n",
       "1    145\n",
       "Name: Churn, dtype: int64"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[\"Churn\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training: Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LogisticRegression()\n",
    "model = model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "# print(y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prediction Comparison with the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Churn</th>\n",
       "      <th>Churn_probability</th>\n",
       "      <th>Churn_predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1653</th>\n",
       "      <td>1</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1716</th>\n",
       "      <td>0</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3251</th>\n",
       "      <td>0</td>\n",
       "      <td>0.16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2406</th>\n",
       "      <td>0</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Churn  Churn_probability  Churn_predicted\n",
       "68        0               0.35                0\n",
       "1653      1               0.24                0\n",
       "1716      0               0.12                0\n",
       "3251      0               0.16                0\n",
       "2406      0               0.10                0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_test[\"Churn_probability\"] = np.round(model.predict_proba(X_test)[:,1], 2)\n",
    "y_test[\"Churn_predicted\"] = y_pred\n",
    "\n",
    "display(y_test.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X_test =  (1000, 17)\n",
      "Shape of y_test =  (1000, 3)\n"
     ]
    }
   ],
   "source": [
    "print(\"Shape of X_test = \", X_test.shape)\n",
    "print(\"Shape of y_test = \", y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 20)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Account_Length</th>\n",
       "      <th>Vmail_Message</th>\n",
       "      <th>Day_Mins</th>\n",
       "      <th>Eve_Mins</th>\n",
       "      <th>Night_Mins</th>\n",
       "      <th>Intl_Mins</th>\n",
       "      <th>CustServ_Calls</th>\n",
       "      <th>Intl_Plan</th>\n",
       "      <th>Vmail_Plan</th>\n",
       "      <th>Day_Calls</th>\n",
       "      <th>Day_Charge</th>\n",
       "      <th>Eve_Calls</th>\n",
       "      <th>Eve_Charge</th>\n",
       "      <th>Night_Calls</th>\n",
       "      <th>Night_Charge</th>\n",
       "      <th>Intl_Calls</th>\n",
       "      <th>Intl_Charge</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Churn_probability</th>\n",
       "      <th>Churn_predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>126</td>\n",
       "      <td>0</td>\n",
       "      <td>211.6</td>\n",
       "      <td>216.9</td>\n",
       "      <td>153.5</td>\n",
       "      <td>7.8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>70</td>\n",
       "      <td>35.97</td>\n",
       "      <td>80</td>\n",
       "      <td>18.44</td>\n",
       "      <td>60</td>\n",
       "      <td>6.91</td>\n",
       "      <td>1</td>\n",
       "      <td>2.11</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1653</th>\n",
       "      <td>93</td>\n",
       "      <td>0</td>\n",
       "      <td>131.4</td>\n",
       "      <td>219.7</td>\n",
       "      <td>155.7</td>\n",
       "      <td>11.1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>78</td>\n",
       "      <td>22.34</td>\n",
       "      <td>106</td>\n",
       "      <td>18.67</td>\n",
       "      <td>103</td>\n",
       "      <td>7.01</td>\n",
       "      <td>2</td>\n",
       "      <td>3.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1716</th>\n",
       "      <td>36</td>\n",
       "      <td>25</td>\n",
       "      <td>152.8</td>\n",
       "      <td>242.8</td>\n",
       "      <td>147.4</td>\n",
       "      <td>9.1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>110</td>\n",
       "      <td>25.98</td>\n",
       "      <td>67</td>\n",
       "      <td>20.64</td>\n",
       "      <td>74</td>\n",
       "      <td>6.63</td>\n",
       "      <td>2</td>\n",
       "      <td>2.46</td>\n",
       "      <td>0</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3251</th>\n",
       "      <td>88</td>\n",
       "      <td>0</td>\n",
       "      <td>274.6</td>\n",
       "      <td>161.1</td>\n",
       "      <td>194.4</td>\n",
       "      <td>9.2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>105</td>\n",
       "      <td>46.68</td>\n",
       "      <td>121</td>\n",
       "      <td>13.69</td>\n",
       "      <td>123</td>\n",
       "      <td>8.75</td>\n",
       "      <td>4</td>\n",
       "      <td>2.48</td>\n",
       "      <td>0</td>\n",
       "      <td>0.16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2406</th>\n",
       "      <td>81</td>\n",
       "      <td>0</td>\n",
       "      <td>145.6</td>\n",
       "      <td>287.9</td>\n",
       "      <td>181.7</td>\n",
       "      <td>9.2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>59</td>\n",
       "      <td>24.75</td>\n",
       "      <td>131</td>\n",
       "      <td>24.47</td>\n",
       "      <td>121</td>\n",
       "      <td>8.18</td>\n",
       "      <td>4</td>\n",
       "      <td>2.48</td>\n",
       "      <td>0</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>793</th>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>112.8</td>\n",
       "      <td>218.8</td>\n",
       "      <td>240.2</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>108</td>\n",
       "      <td>19.18</td>\n",
       "      <td>120</td>\n",
       "      <td>18.60</td>\n",
       "      <td>106</td>\n",
       "      <td>10.81</td>\n",
       "      <td>3</td>\n",
       "      <td>2.43</td>\n",
       "      <td>0</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1029</th>\n",
       "      <td>116</td>\n",
       "      <td>0</td>\n",
       "      <td>201.8</td>\n",
       "      <td>231.5</td>\n",
       "      <td>226.1</td>\n",
       "      <td>16.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>82</td>\n",
       "      <td>34.31</td>\n",
       "      <td>95</td>\n",
       "      <td>19.68</td>\n",
       "      <td>130</td>\n",
       "      <td>10.17</td>\n",
       "      <td>5</td>\n",
       "      <td>4.46</td>\n",
       "      <td>0</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1943</th>\n",
       "      <td>125</td>\n",
       "      <td>0</td>\n",
       "      <td>168.6</td>\n",
       "      <td>175.6</td>\n",
       "      <td>243.3</td>\n",
       "      <td>10.9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>99</td>\n",
       "      <td>28.66</td>\n",
       "      <td>107</td>\n",
       "      <td>14.93</td>\n",
       "      <td>92</td>\n",
       "      <td>10.95</td>\n",
       "      <td>7</td>\n",
       "      <td>2.94</td>\n",
       "      <td>0</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1161</th>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>170.7</td>\n",
       "      <td>179.1</td>\n",
       "      <td>281.9</td>\n",
       "      <td>8.2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>29.02</td>\n",
       "      <td>108</td>\n",
       "      <td>15.22</td>\n",
       "      <td>89</td>\n",
       "      <td>12.69</td>\n",
       "      <td>9</td>\n",
       "      <td>2.21</td>\n",
       "      <td>0</td>\n",
       "      <td>0.41</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>203.3</td>\n",
       "      <td>228.9</td>\n",
       "      <td>222.2</td>\n",
       "      <td>14.3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>70</td>\n",
       "      <td>34.56</td>\n",
       "      <td>97</td>\n",
       "      <td>19.46</td>\n",
       "      <td>118</td>\n",
       "      <td>10.00</td>\n",
       "      <td>3</td>\n",
       "      <td>3.86</td>\n",
       "      <td>0</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  \\\n",
       "68               126              0     211.6     216.9       153.5   \n",
       "1653              93              0     131.4     219.7       155.7   \n",
       "1716              36             25     152.8     242.8       147.4   \n",
       "3251              88              0     274.6     161.1       194.4   \n",
       "2406              81              0     145.6     287.9       181.7   \n",
       "793               45              0     112.8     218.8       240.2   \n",
       "1029             116              0     201.8     231.5       226.1   \n",
       "1943             125              0     168.6     175.6       243.3   \n",
       "1161              40              0     170.7     179.1       281.9   \n",
       "2837              75              0     203.3     228.9       222.2   \n",
       "\n",
       "      Intl_Mins  CustServ_Calls  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
       "68          7.8               1          0           0         70       35.97   \n",
       "1653       11.1               1          1           0         78       22.34   \n",
       "1716        9.1               1          0           1        110       25.98   \n",
       "3251        9.2               2          0           0        105       46.68   \n",
       "2406        9.2               2          0           0         59       24.75   \n",
       "793         9.0               2          0           0        108       19.18   \n",
       "1029       16.5               0          0           0         82       34.31   \n",
       "1943       10.9               0          0           0         99       28.66   \n",
       "1161        8.2               3          1           0         55       29.02   \n",
       "2837       14.3               1          0           0         70       34.56   \n",
       "\n",
       "      Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  \\\n",
       "68           80       18.44           60          6.91           1   \n",
       "1653        106       18.67          103          7.01           2   \n",
       "1716         67       20.64           74          6.63           2   \n",
       "3251        121       13.69          123          8.75           4   \n",
       "2406        131       24.47          121          8.18           4   \n",
       "793         120       18.60          106         10.81           3   \n",
       "1029         95       19.68          130         10.17           5   \n",
       "1943        107       14.93           92         10.95           7   \n",
       "1161        108       15.22           89         12.69           9   \n",
       "2837         97       19.46          118         10.00           3   \n",
       "\n",
       "      Intl_Charge  Churn  Churn_probability  Churn_predicted  \n",
       "68           2.11      0               0.35                0  \n",
       "1653         3.00      1               0.24                0  \n",
       "1716         2.46      0               0.12                0  \n",
       "3251         2.48      0               0.16                0  \n",
       "2406         2.48      0               0.10                0  \n",
       "793          2.43      0               0.06                0  \n",
       "1029         4.46      0               0.08                0  \n",
       "1943         2.94      0               0.04                0  \n",
       "1161         2.21      0               0.41                0  \n",
       "2837         3.86      0               0.19                0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test = pd.concat([X_test, y_test], axis=1)\n",
    "\n",
    "print(test.shape)\n",
    "display(test.head(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEGCAYAAABxfL6kAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkE0lEQVR4nO3de7xd853/8df75E7ikhKCVIIEcYuKqBoq+BHaCp0iWm2oqdKU0VZdRluGZqQdrSnGtKGtKBVxDzUuTWm0LrmVkBBScckkcnMNEkl8fn+s72E7PWfvdU72yb7k/exjPfZaa6/1Xd99dn3y3d/1XZ+vIgIzM6sfDZWugJmZlZcDu5lZnXFgNzOrMw7sZmZ1xoHdzKzOdKx0BdZ36tgt1LlHpathrbDnzp+sdBWslWbMmL40IjZv6/kdNto2YvV7uY6N95bcFxHD2nqtcnBgrzB17kGXHY+tdDWsFf76+JWVroK1UrdOemltzo/VK+iy04hcx6742xWbrc21ysGB3cysFAFSpWuRmwO7mVkeqp1bkg7sZmZ5uMVuZlZPBA0dKl2J3Grnt4WZWaWIrCsmz5KnOOk7kmZJelrSjZK6Suop6QFJz6fXTQuOP0/SXElzJB1WqnwHdjOzkpR1xeRZSpUkbQ2cAQyOiF2BDsAI4FxgUkT0ByalbSQNTO/vAgwDrpJU9OeDA7uZWR5lbLGTdYN3k9QR2ABYAAwHxqX3xwFHpfXhwPiIWBkR84C5wJBihTuwm5nlkb/FvpmkaQXLKYXFRMT/AZcCLwMLgTcj4n5gi4hYmI5ZCPRKp2wNvFJQxPy0r0W+eWpmVpJa0xpfGhGDWywp6zsfDvQD3gBulnRC8Yv/g6ITaTiwm5mVIso5KuYQYF5ELAGQdBvwGWCRpN4RsVBSb2BxOn4+0Kfg/G3Ium5a5K4YM7OSVM4+9peBT0vaQJKAg4FngInAyHTMSODOtD4RGCGpi6R+QH9gSrELuMVuZpZHQ3keUIqIxyXdAswAVgN/A8YC3YEJkk4mC/7HpONnSZoAzE7Hj4qINcWu4cBuZlZK4zj2MomIC4ALmuxeSdZ6b+740cDovOU7sJuZ5eGUAmZm9aS2Ugo4sJuZ5eHsjmZmdSRnuoBq4cBuZpaHW+xmZnXGLXYzs3rSqpQCFefAbmZWSnlTCrQ7B3Yzs5LcYjczqz/uYzczqzNusZuZ1Rm32M3M6ojcx25mVnfU4MBuZlY3BMhdMWZmdUQ0P/NolXJgNzMrSW6xm5nVm1oK7LVzN8DMrIIaGhpyLaVI2lHSEwXLW5LOlNRT0gOSnk+vmxacc56kuZLmSDqsZF3X8rOamdU/tWIpISLmRMSgiBgE7AW8C9wOnAtMioj+wKS0jaSBwAhgF2AYcJWkoolrHNjNzEpQ6mPPs7TSwcDfI+IlYDgwLu0fBxyV1ocD4yNiZUTMA+YCQ4oV6j52M7McWhG0N5M0rWB7bESMbeHYEcCNaX2LiFgIEBELJfVK+7cGHis4Z37a1yIHdjOzHFoR2JdGxOAc5XUGjgTOK3VoM/ui2AnuijEzy6EdumIOB2ZExKK0vUhS73St3sDitH8+0KfgvG2ABcUKdmA3MytFoAblWlrheD7qhgGYCIxM6yOBOwv2j5DURVI/oD8wpVjB7ooxMytBZX5ASdIGwP8DvlmwewwwQdLJwMvAMQARMUvSBGA2sBoYFRFripXvwG5mlkM5A3tEvAt8osm+ZWSjZJo7fjQwOm/5DuxmZnnUzoOnDuxmZiWptlIKOLCbmeXgwG5mVkeEcuWBqRYO7GZmedROg92B3cysJPexm5nVHwd2M7M648BuZlZnWpkuoKIc2K1NTjt+KF896jMQwey5Cxh10fV87+uHccQBu/NBBEtee5tR/349ry59kwOH7MQF3z6Szp068v6q1fzo8jt4eNpzlf4I67X5r77OaRdex+Jlb9EgMfLo/Tj1+KGM/p+7uWfyTBokNu/Zg/++4AR6b75JpatbcW3MtV4xiiia/bH9LizdA3w5It6QtDwiukvqC9wdEbuWofwzyfIgv5vz+BOBwRHx7bW9dms0bNAruux47Lq85FrrvfnG/O/V3+HTx41mxcpV/OY/vs4Dj8zi7gef5O13VgBwynGfZad+vfnumPHsNmAblrz2Nq8ufZOdt+/NLZePYpfP/aDCn6LtXp96ZaWrsNZeXfomi5a+xR479eHtd1Yw9Gs/4fr/PIWtem3CRt27AfCr8Q/x7LyFXHbe8RWu7drr1knT86TSbUmXLfpH7xGX5Tr2pcu/sFbXKoeKDcyMiCMi4o12vMSZwAbtWP7HSFqvfv107NiBrl060aFDAxt07cyrS978MKgDbNitC42Nhqeem8+rS98E4Jm/L6Rr50507rRe/bmqzpabbcweO2WZYHts2JUBfbdk4ZI3PgzqAO+8t7KmWqntrZ1mUGoX7fJfl6SzgRURcbmky4A9IuIgSQcDJ0XECZJeJGshL81R3oHAhcBSYFdgOnBCREQq89L0WaYCp5FlTNsKeFDS0ogY2qS8vYFfABsCK/ko8c5Wku4Ftgduj4iz0/HLI6J7Wv8S8PmIOFHStcBrwJ7ADEmfAN4CBgNbAmdHxC2t++tVv4VL3uSK6yfx1F0Xs2Ll+zz4+LM8+PizAPzgtC8w4nNDeGv5e3zh1Mv/4dwjDxrEzOde4f1Vq9d1ta0FLy9Yxsw589lrl74AXHzVRMb/YQobde/GXb88o7KVqybVEbNzaa8W+2Rg/7Q+GOguqRPwT8DDbSxzT7JW+EBgO2A/SV2Ba4HjImI3suB+WkRcTpaIfmgzQb0zcBPwrxGxB3AI8F56exBwHLAbcJykwuT2LRkAHBIR30vbvck+5+fJ0nD+A0mnSJomaVqsfq+5Q6raxj26ccQBuzFo+AXsfPj5bNC1M8cevjcAP/6fu9j18z/k5nun8Y1jD/jYeTtttyUXnj6c7/zH+EpU25qx/N2VfO2ca7jku//8YWv9h986kll/+DHHDBvM1RMmV7iG1aOWWuztFdinA3tJ6kHWIn6ULMDvT9sD+5SImB8RHwBPAH2BHYF5EdF4J24ccEDzp39oR2BhREwFiIi3IqKx+TgpIt6MiBVkuY+3zVGvm5vkRr4jIj6IiNnAFs2dEBFjI2JwRAxWx27NHVLVDhyyEy8tWMayN5azes0H3PXgkwzZvd/Hjrnl3qkcedCgD7e36rUJv/vpKZx2we948f9K/kizdWDV6jWMPOdqjhk2mC8UfFeNvjRsbyb+6Yl1Xq9qJEFDg3It1aBdAntErAJeBE4CHiEL5kPJujieaWOxKwvW15C1ztvyVxQtzxfY3DVocnzXJue8U6SM6viWy2z+q68xeLd+dOvSCYDP7r0jc+YtYrs+m394zLADdue5F7MZvzbq3o2bLjuVi/57Io/PfKEidbaPiwhOv/gGBvTdklFf+SgF+N9fXvzh+r2TZzKgb7Ntk/VQvtZ6tbTY2/MO1mTgLODrwFPAz4HpUd5hOM8CfSXtEBFzga8Cf07vvQ30IOuXb3rOVpL2joip6VdFqf6QRZJ2BuYAR6ey11vTZ73ExEl/46Hrz2HNmg+YOWc+427/K1f/+ET6b9uLDz4IXnn1Nb57Sdbl8o1jD6Bfn835/r8M4/v/MgyAL377Spa+vrySH2O99tiTL3DTPVMYuMNW7P/lSwD44agjuf7OR3j+pcU0NIg+W/bk5+eNqHBNq0eVxOxc2jOwPwycDzwaEe9IWkHbu2GaFRErJJ0E3JxGpUwFfpneHgv8r6SFhf3sEfG+pOOAKyR1Iwvqh5S41LnA3cArwNNA93J+jlo0Zuw9jBl7z8f2jTznmmaP/dlv7uNnv7lvXVTLctp30PbNDts8dL9dKlCb2lAtrfE8KjaO3TK1OI59fVcP49jXN2s7jr1r7wHRd+QVuY6d85NhJa8laRPgGrJRfkHWszGHbGBHX7Ku7GMj4vV0/HnAyWRdxGdERNGWUu0kGDYzqxBR9punvwDujYidgD3I7j2eSzaAoz8wKW0jaSAwAtgFGAZcJalDscId2M3McihXYJe0EdnovV9D1j2cHtYcTjayj/R6VFofDoyPiJURMQ+YCwwpWtc2fD4zs/WLspuneRZgs8bnVNJySpPStgOWAL+V9DdJ10jaENgiIhYCpNde6fitye7vNZqf9rXIz3WbmZUgWnXzdGmJPvaOwKeA0yPicUm/IHW7FLl8U0VvjrrFbmZWUlnHsc8H5kfE42n7FrJAv0hSb4D0urjg+MKn4Lche7K+RQ7sZmY5tKIrpqiIeBV4RdKOadfBZE+6TwRGpn0jgTvT+kRghKQukvoB/YEpxa7hrhgzs1JSSoEyOh24IeWueoHsKf0GYIKkk4GXgWMAImKWpAlkwX81MKpJGpN/4MBuZlZCK/vYS4qIJ8jyZzV1cDP7iIjRwOi85Tuwm5nlUEMPnjqwm5nlUUspBRzYzcxyqKG47sBuZlaS3GI3M6sronom0cjDgd3MLIcaarA7sJuZ5eGuGDOzepLzqdJq4cBuZlZCuR9Qam8O7GZmOTiwm5nVGY+KMTOrJ+5jNzOrLyJ3rvWq4MBuZpZDDcV1B3YzszwaaiiyO7CbmZWg8k+00a4c2M3McqihuO45T83M8ijjZNZIelHSU5KekDQt7esp6QFJz6fXTQuOP0/SXElzJB1WqvwWW+ySrgCipfcj4oxcn8DMrA60Qxf70IhYWrB9LjApIsZIOjdtnyNpIDAC2AXYCvijpAHF5j0t1hUzrQwVNzOreSIb8tjOhgMHpvVxwEPAOWn/+IhYCcyTNBcYAjzaUkEtBvaIGFe4LWnDiHhnraptZlajytzHHsD9kgL4VUSMBbaIiIUAEbFQUq907NbAYwXnzk/7WlTy5qmkfYFfA92BT0raA/hmRHyr1R/FzKwWqVUTbWzW2G+ejE2Bu9B+EbEgBe8HJD1b7OrN7GuxmxzyjYr5L+AwYCJARDwp6YAc55mZ1QXRqnHsSyNicLEDImJBel0s6XayrpVFknqn1npvYHE6fD7Qp+D0bYAFxcrPNSomIl5psqvFTnszs3ok5VtKl6MNJfVoXAcOBZ4mazyPTIeNBO5M6xOBEZK6SOoH9AemFLtGnhb7K5I+A4SkzsAZwDM5zjMzqxtlzBWzBXB7Kq8j8PuIuFfSVGCCpJOBl4FjACJilqQJwGxgNTCq2IiYxkJLORX4BVln/f8B9wGj2vZ5zMxqT97WeB4R8QKwRzP7lwEHt3DOaGB03muUDOxpnOVX8hZoZlaPOtRQrpiSfeyStpN0l6QlkhZLulPSduuicmZm1aKcT562tzw3T38PTAB6kz31dDNwY3tWysysmmSjYvIt1SBPYFdE/C4iVqflekqMoTQzqys5W+vV0mIvliumZ1p9MOUtGE8W0I8D/rAO6mZmVjWqJGbnUuzm6XSyQN74cb5Z8F4AF7dXpczMqk21tMbzKJYrpt+6rIiZWbUS0KFaOtBzyDXRhqRdgYFA18Z9EXFde1XKzKza1E5Yz5cE7AKyVJIDgXuAw4G/AA7sZrZekGprztM8o2K+RPY01KsRcRLZE1Nd2rVWZmZVply5YtaFPF0x70XEB5JWS9qILOOYH1Ays/VKXdw8LTBN0ibA1WQjZZZTIrOYmVm9qaG4nitXTOOEGr+UdC+wUUTMbN9qmZlVD0n1MSpG0qeKvRcRM9qnSmZm1adeumJ+VuS9AA4qc13WS7sM6MPEBy6tdDXMrIRcsxJViWIPKA1dlxUxM6tWon5a7GZmltRQF7sDu5lZKVJtpRSopW4jM7OKKXc+dkkdJP1N0t1pu6ekByQ9n143LTj2PElzJc2RdFjJuua4uCSdIOlHafuTkobkr76ZWe1rhydP/xV4pmD7XGBSRPQHJqVtJA0ERgC7AMOAqyR1KFZwnhb7VcC+wPFp+23gv1tTezOzWpbNoKRcS67ypG2AzwHXFOweDoxL6+OAowr2j4+IlRExD5gLFG1c5wns+0TEKGAFQES8DnTOVXszszrRkHMBNpM0rWA5pZni/gs4G/igYN8WEbEQIL32Svu3Bl4pOG5+2teiPDdPV6VmfwBI2rxJZczM6l4rulmWRsTglsvR54HFETFd0oF5Lt3MvqLTk+YJ7JcDtwO9JI0my/b4gxznmZnVhTKnFNgPOFLSEWRzXGwk6XpgkaTeEbFQUm+yhIuQtdD7FJy/DbCg2AVKdsVExA1kPxkuARYCR0XEza3+KGZmNaxco2Ii4ryI2CYi+pLdFP1TRJwATARGpsNGAnem9YnACEldJPUD+lMiEWOeiTY+CbwL3FW4LyJeLv0RzMxqX+PN03Y2Bpgg6WTgZeAYgIiYJWkCMBtYDYyKiDXFCsrTFfMHPprUuivQD5hDNvTGzGy90B5xPSIeAh5K68vIJjVq7rjRwOi85eZJ27tb4XbK+vjNvBcwM6t5rXz4qNJanVIgImZI2rs9KmNmVq1UQ9NZ5+lj/27BZgPwKWBJu9XIzKzKCOhYQwlY8rTYexSsrybrc7+1fapjZlad6iZtb3owqXtEfH8d1cfMrOpko2IqXYv8ik2N1zEiVhebIs/MbL3Q+gRfFVWsxT6FrD/9CUkTgZuBdxrfjIjb2rluZmZVYx2MYy+bPH3sPYFlZHOcNo5nD8CB3czWCwI61MnN015pRMzTfBTQGxVNQGNmVl9EQ50Md+wAdKcNmcXMzOpJNpl1pWuRX7HAvjAiLlpnNTEzq1Z19ORpDX0MM7P2VS83T5tNRmNmtr6pm66YiHhtXVbEzKyalXGijXbX6iRgZmbrG5Fvguhq4cBuZlaK6ihXjJmZZWonrDuwm5mVtI6mxiubWuo2MjOrGOVcSpYjdZU0RdKTkmZJ+ve0v6ekByQ9n143LTjnPElzJc2RdFipaziwm5mVJBoa8i05rAQOiog9gEHAMEmfBs4FJkVEf2BS2kbSQGAE2TzTw4CrUkr1Fjmwm5mV0DgqJs9SSmSWp81OaQlgODAu7R8HHJXWhwPjI2JlRMwD5gJDil3Dgd3MLAdJuRZgM0nTCpZTmimrg6QngMXAAxHxOLBFRCwESK+90uFbA68UnD4/7WuRb56ameXQilunSyNicLEDImINMEjSJsDtknZt5aWLJmJ0i93MrBS1qsWeW0S8ATxE1ne+SFJvgPS6OB02H+hTcNo2wIJi5Tqwm5mVIKCDlGspWZa0eWqpI6kbcAjwLDARGJkOGwncmdYnAiMkdZHUD+hPNsNdi9wVY2aWQxlHsfcGxqWRLQ3AhIi4W9KjwARJJwMvA8cARMQsSROA2cBqYFTqymmRA7uZWQ7lej4pImYCezazfxktZNWNiNHA6LzXcGA3MyshG+5YO0+eOrCbmeVQQxkFHNjNzEoTcovdzKx+NI6KqRUO7GZmpchdMWZmdceB3cyszriP3cysjmQTbVS6Fvk5sJuZ5VBLMyg5sJuZ5eCuGKt75196E39+fDY9N+nOxKu/D8B/jr2Lhx6bTaeOHemz1ScYfdZxbNS9G2+89Q5nXnQdT815haMPHcwPTv9ihWtv8199ndMuvI7Fy96iQWLk0ftx6vFDP3z/it/9kR9dfgdzHxjDJzbpXsGaVoda64qpm+yOku4pyJi2PL32lfR0K8p4SFLRPMqWOfrQwYz9j298bN9nPjWAO68+izvGfo++W2/G1TdOAqBzp46cfuIwvn/K5ytRVWtGx44N/PjML/L4zT/k/t+exTW3TObZFxYCWdB/aMqzbLPlpiVKWZ8o9/+qQd0E9og4IuU2rphS8xDWk8G7b8/GPTb42L79Bu9Ixw7Zn2CPnbfl1aVvArBBty7stWs/unTutM7rac3bcrON2WOnLMV3jw27MqDvlixc8gYA5192KxeeflSrc4vXtTSOPc9SDWoisEs6W9IZaf0ySX9K6wdLuj6tvyhps1aW+VSaKXxMwVvHpBnEn5O0fzr2RElXFpx7t6QD0/pySRdJehzYN22PTuU+JmmLtf38tei2+6aw/947VboalsPLC5Yxc8589tqlL/f8eSa9N9+E3QZsU+lqVR3lXKpBTQR2YDKwf1ofDHSX1An4J+Dh1hYm6XCyiWL3STOF/7Tg7Y4RMQQ4E7ggR3EbAk9HxD4R8Ze0/VgqdzLwjaYnSDqlcT7E15YtaW31q94vb/gjHTp04AsHf6rSVbESlr+7kq+dcw2XfPef6dixAz//7X2cd+rnKl2tqlPOiTbWhVoJ7NOBvST1AFYCj5IF+P1pQ2Anm7HktxHxLkBEvFbw3m0F1+ybo6w1wK0F2+8DdxcrIyLGRsTgiBjc8xObt67mVe6O+6fy58ef4afnftk/5avcqtVrGHnO1RwzbDBfOGgQ8+Yv4aUFy9j/y5ew+5E/YsHiN/jsCT9h0dK3Kl3V6lBDTfaaGBUTEaskvQicBDwCzASGAtsDz7ShSNHyZLAr0+saPvr7rObj/wh2LVhf0WQ2k1UR0Vh2YRl17+Gpz3LNTQ9y3c++RbeunStdHSsiIjj94hsY0HdLRn0lm9thlx225vn7P+qV3P3IH/HgdWd7VExSLTdG86iloDMZOAv4OvAU8HNgekEQbY37gR9J+n1EvCupZ5NWe1MvAt+S1ABsDQxpwzXrylmjr2fKzL/zxpvvMPT4i/n21w5l7Pg/sWrVak4+ZywAe+z8SS4880sAHHLCaJa/u4JVq9Yw6ZFZXD3mG+yw7ZaV/AjrtceefIGb7pnCwB22Yv8vXwLAD0cdyaH77VLhmlWvWvoBWkuB/WHgfODRiHhH0gra1g1DRNwraRAwTdL7wD3AvxU55a/APLJ/UJ4GZrTluvXk0vNP+Id9/3z4Pi0e/8frz2/P6lgr7Ttoe16femXRY2ZOvGgd1aY2lCuuS+oDXAdsCXwAjI2IX0jqCdxE1n37InBsRLyezjkPOJmsF+CMiLiv6DXa1uC1ctlt0F4x8Y9/rXQ1rBV6b9K19EFWVbp10vSIaPMzKjvvtmdcN/GhXMcO2W6ToteS1BvoHREz0n3D6WSDOU4EXouIMZLOBTaNiHMkDQRuJOsp2Ar4IzCg2ITWtXLz1MysYqQsV0yepZSIWBgRM9L622T3CbcGhgPj0mHjyII9af/4iFgZEfOAuZToDnZgNzPLoRWDYjZrHM6cllNaLFPqC+wJPA5sERELIQv+QK902NbAKwWnzU/7WlRLfexmZpWTv5N9aZ5uH0ndyYZKnxkRbxUZHtzcG0X70N1iNzMrqby5YtIDlrcCN0RE47Mzi1L/e2M//OK0fz7Qp+D0bYAFxcp3YDczy6FcuWKUNc1/DTwTET8veGsiMDKtjwTuLNg/QlIXSf2A/sCUYtdwV4yZWQmirOPY9wO+Cjwl6Ym079+AMcAESScDLwPHAETELEkTgNlkD0uOKjYiBhzYzcxyKdeTpymnVEuFHdzCOaOB0Xmv4cBuZpaDnzw1M6szNRTXHdjNzEqqosyNeTiwm5nl4OyOZmZ1pNYms3ZgNzPLw4HdzKy+uCvGzKzOeLijmVmdqaG47sBuZpZLDUV2B3YzsxIaJ9qoFQ7sZmY51E5Yd2A3M8unhiK7A7uZWUn5J9GoBg7sZmY51FAXuwO7mVkpZZ5oo905sJuZ5eCuGDOzOlNLLXZPZm1mloNyLiXLkX4jabGkpwv29ZT0gKTn0+umBe+dJ2mupDmSDstTVwd2M7NSlLXY8yw5XAsMa7LvXGBSRPQHJqVtJA0ERgC7pHOuktSh1AUc2M3McilPmz0iJgOvNdk9HBiX1scBRxXsHx8RKyNiHjAXGFLqGu5jNzMroZUTbWwmaVrB9tiIGFvinC0iYiFARCyU1Cvt3xp4rOC4+WlfUQ7sZmY5tOLm6dKIGFyuyzazL0qd5K4YM7MclPN/bbRIUm+A9Lo47Z8P9Ck4bhtgQanCHNjNzPIo17CY5k0ERqb1kcCdBftHSOoiqR/QH5hSqjB3xZiZ5VCuYeySbgQOJOuLnw9cAIwBJkg6GXgZOAYgImZJmgDMBlYDoyJiTalrOLCbmZXQiqGMJUXE8S28dXALx48GRrfmGg7sZmY5qIYePXVgNzPLoXbCugO7mVkuNdRgd2A3MyvNE22YmdUV52M3M6tDDuxmZnXGXTFmZvWkjOPY1wUHdjOzEtYuW8C658BuZpZHDUV2B3Yzsxzcx25mVmdaMdFGxTmwm5nl4cBuZlZf3BVjZlZHau3JU0WUnD7P2pGkJcBLla5HO9gMWFrpSlir1PN3tm1EbN7WkyXdS/b3yWNpRAxr67XKwYHd2oWkaWWc0NfWAX9n9cNznpqZ1RkHdjOzOuPAbu1lbKUrYK3m76xOuI/dzKzOuMVuZlZnHNjNzOqMA/t6SNI9kjZJ68vTa19JT5ep/DMlbdCK40+UdGU5rr2+K8d3K+khSR72WMMc2NdDEXFERLzRjpc4E8gd2NeWJD9BnayD77YkSR0qeX1zYK87ks6WdEZav0zSn9L6wZKuT+svSsr1FJ2kA1ML7hZJz0q6Qcoerk5l/k3SU5J+I6lLuvZWwIOSHmymvL0lPSLpSUlTJPVIb20l6V5Jz0v6acHxywvWvyTp2rR+raSfp2v8JG1fnsp+QdKX2vDnq2rl/m4LynwqfR9jCt46Jn0/z0naPx37sV9Wku6WdGBaXy7pIkmPA/um7dGp3MckbbG2n9/yc2CvP5OB/dP6YKC7pE7APwEPt7HMPcla4QOB7YD9JHUFrgWOi4jdyPIOnRYRlwMLgKERMbSwEEmdgZuAf42IPYBDgPfS24OA44DdgOMk9clRrwHAIRHxvbTdm+xzfh4Y0+JZtaus362kw4GjgH3S9/HTgrc7RsQQsu/9ghzFbQg8HRH7RMRf0vZjqdzJwDdaWz9rOwf2+jMd2Cu1hFcCj5IFgf1pe2CfEhHzI+ID4AmgL7AjMC8inkvHjAMOKFHOjsDCiJgKEBFvRcTq9N6kiHgzIlYAs4Ftc9Tr5ohYU7B9R0R8EBGzgXpsIZb7uz0E+G1EvAsQEa8VvHdbwTX75ihrDXBrwfb7wN2tLMPKxH2TdSYiVkl6ETgJeASYCQwFtgeeaWOxKwvW15D9/6Ytue4EtPTgRHPXoMnxXZuc806RMmooF18+7fDd5vk+Cr+L1Xy8MVj4faxo8o/sqvjoIZnCMmwdcIu9Pk0GzkqvDwOnAk8U/IdWDs8CfSXtkLa/Cvw5rb8N9GjhnK0k7Q0gqUeOG5+LJO0sqQE4ugz1rnXl/G7vB77eOIJJUs8Sx78IDJLUkLrKhrThmrYOOLDXp4fJ+psfjYhFwAra3g3TrNRlchJws6SngA+AX6a3xwL/2/TmaUS8T9aPfoWkJ4EH+MdWeFPnkv2k/xOwsHyfoGaV7buNiHuBicA0SU+Q/YNRzF+BecBTwKXAjLZc19qfUwqYmdUZt9jNzOqMA7uZWZ1xYDczqzMO7GZmdcaB3cyszjiwW9WTtEbSE5KelnRzazJHNlPWtY15ZCRdI2lgkWMPlPSZNlyj2XwtefK4FObGyXmtCyWVGqZo6xkHdqsF70XEoIjYlexR9VML32xrNsGI+JeUfqAlBwKtDuxmlebAbrXmYWCH1Jp+UNLvgackdZD0n5KmSpop6ZsAylwpabakPwC9GgtSQd5xScMkzUjZCCdJ6kv2D8h30q+F/SVtLunWdI2pkvZL535C0v3KMl3+ihzpDCTdIWm6pFmSTmny3s9SXSZJ2jzt215Z9svpkh6WtFNZ/ppWl5y/wWpGSj9wOHBv2jUE2DUi5qXg+GZE7C2pC/BXSfeTZabckSxr5BZkCcZ+06TczYGrgQNSWT0j4jVJvwSWR8Sl6bjfA5dFxF8kfRK4D9iZLPvhXyLiIkmfAz4WqFvw9XSNbsBUSbdGxDKyrIgzIuJ7kn6Uyv422dO8p0bE85L2Aa4CDmrDn9HWAw7sVgu6pUfeIWux/5qsi2RKRMxL+w8FdtdHedg3BvqTZZy8MSWoWqCUw7yJTwOTG8tqkuWw0CHAQOnDBvlGKdPiAcAX07l/kPR6js90hqTG3Dd9Ul2XkaVmuCntvx64TVL39HlvLrh2lxzXsPWUA7vVgvciYlDhjhTgCrM7Cjg9Iu5rctwRtJzBsPDcPLk1GoB9I+K9wp2pLrlzc6TJKQ5JZb0r6SFazpkT6bpvNP0bmLXEfexWL+4DTksTTyBpgKQNybIgjkh98L3J0tw29SjwWUn90rmNWQ6bZqm8n6xbhHTcoLQ6GfhK2nc4sGmJum4MvJ6C+k5kvxgaNQCNvzq+TNbF8xYwT9Ix6RqStEeJa9h6zIHd6sU1ZP3nM5RN3Pwrsl+ktwPPk2Uk/B8+Si38oYhYQtYvflvKOtnYFXIXcHTjzVPgDGBwujk7m49G5/w7cICkGWRdQi+XqOu9QEdJM4GLgccK3nsH2EXSdLI+9IvS/q8AJ6f6zQKG5/ib2HrK2R3NzOqMW+xmZnXGgd3MrM44sJuZ1RkHdjOzOuPAbmZWZxzYzczqjAO7mVmd+f+zJEOQB7swrAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(model,\n",
    "                      X_test,\n",
    "                      y_test[\"Churn\"],\n",
    "                      display_labels = [\"will not churn\", \"will churn\"],\n",
    "                      cmap = \"Blues\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[832  23]\n",
      " [121  24]]\n"
     ]
    }
   ],
   "source": [
    "confusion_matrix_ = confusion_matrix(y_test[\"Churn\"],\n",
    "                                     y_test[\"Churn_predicted\"])\n",
    "\n",
    "print(confusion_matrix_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True Negatives =  832\n",
      "False Negatives =  121\n",
      "False Positives =  23\n",
      "True Positives =  24\n"
     ]
    }
   ],
   "source": [
    "TN = confusion_matrix_[0,0]\n",
    "print(\"True Negatives = \", TN)\n",
    "\n",
    "FN = confusion_matrix_[1,0]\n",
    "print(\"False Negatives = \", FN)\n",
    "\n",
    "FP = confusion_matrix_[0,1]\n",
    "print(\"False Positives = \", FP)\n",
    "\n",
    "TP = confusion_matrix_[1,1]\n",
    "print(\"True Positives = \", TP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.856\n"
     ]
    }
   ],
   "source": [
    "accuracy = (TN+TP)/(TN+TP+FP+FN)\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Precision, Recall/Sensitivity/True Positive Rate, F1 score\n",
    "\n",
    "**precision:**\n",
    "Precision can be seen as a measure of a classifier’s exactness. For each class, it is defined as the ratio of true positives to the sum of true and false positives. Said another way, “for all instances classified positive, what percent was correct?”\n",
    "\n",
    "**recall:**\n",
    "Recall is a measure of the classifier’s completeness; the ability of a classifier to correctly find all positive instances. For each class, it is defined as the ratio of true positives to the sum of true positives and false negatives. Said another way, “for all instances that were actually positive, what percent was classified correctly?”\n",
    "\n",
    "**f1 score:**\n",
    "The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally speaking, F1 scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of F1 should be used to compare classifier models, not global accuracy.\n",
    "\n",
    "**support:**\n",
    "Support is the number of actual occurrences of the class in the specified dataset. Imbalanced support in the training data may indicate structural weaknesses in the reported scores of the classifier and could indicate the need for stratified sampling or rebalancing. Support doesn’t change between models but instead diagnoses the evaluation process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculated __________________________\n",
      "Precision =  0.51\n",
      "Recall =  0.17\n",
      "f1_score =  0.26\n",
      "\n",
      "Scikit-Learn ________________________\n",
      "Precision =  0.51\n",
      "Recall =  0.17\n",
      "f1_score =  0.25\n"
     ]
    }
   ],
   "source": [
    "print(\"Calculated __________________________\")\n",
    "precision = np.round(TP/(TP+FP), 2) #predicted class 1\n",
    "recall = np.round(TP/(TP+FN), 2) #actual class 1\n",
    "f1_score_ = np.round((2*precision*recall)/(precision+recall), 2)\n",
    "\n",
    "print(\"Precision = \", precision)\n",
    "print(\"Recall = \", recall)\n",
    "print(\"f1_score = \", f1_score_)\n",
    "\n",
    "print(\"\\nScikit-Learn ________________________\")\n",
    "precision_ = np.round(precision_score(y_test['Churn'], \n",
    "                                      y_test['Churn_predicted']), 2)\n",
    "recall_ = np.round(recall_score(y_test['Churn'], \n",
    "                                y_test['Churn_predicted']), 2)\n",
    "\n",
    "f1_score__ = np.round(f1_score(y_test['Churn'], \n",
    "                               y_test['Churn_predicted']), 2)\n",
    "\n",
    "print(\"Precision = \", precision_)\n",
    "print(\"Recall = \", recall_)\n",
    "print(\"f1_score = \", f1_score__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.97      0.92       855\n",
      "           1       0.51      0.17      0.25       145\n",
      "\n",
      "    accuracy                           0.86      1000\n",
      "   macro avg       0.69      0.57      0.59      1000\n",
      "weighted avg       0.82      0.86      0.82      1000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "classification_report_ = classification_report(y_test[\"Churn\"],\n",
    "                                               y_test[\"Churn_predicted\"])\n",
    "print(classification_report_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}