Session_8/(5) Evaluating Metrics Notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Evaluating Metrics",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Evaluating a model is a core part of building an effective machine learning model**\n",
        "\n",
        "Simply building a predictive model is not your motive. It’s about creating and selecting a model which gives high accuracy on out of sample data. Hence, it is crucial to check the accuracy of your model prior to computing predicted values."
      ],
      "metadata": {
        "id": "7FU_C7jaCsV0"
      }
    },
    {
      "metadata": {
        "id": "YjJmGxnbY9m9"
      },
      "cell_type": "markdown",
      "source": [
        "Lets start with importing libraries we will need in this notebook."
      ]
    },
    {
      "metadata": {
        "id": "eXSU_fk7RlEM"
      },
      "cell_type": "markdown",
      "source": [
        "### Importing libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "`sklearn` Simple and effective tools for predictive data analysis.\n",
        "\n",
        "`seaborn` Python data visualization library based on matplotlib.\n",
        "\n",
        "`itertool` It is a module in Python, to interate over data structures."
      ],
      "metadata": {
        "id": "AtlUiusVCcNM"
      }
    },
    {
      "metadata": {
        "id": "Sx4KZX7cF8Yd"
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.datasets import load_breast_cancer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn import svm\n",
        "from sklearn import metrics\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import itertools\n",
        "\n",
        "np.random.seed(42)  # for reproducibility of code\n",
        "sns.set(rc={\"figure.figsize\": (8, 8)})\n",
        "sns.set_style(\"ticks\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0pddJrEbRtbr"
      },
      "cell_type": "markdown",
      "source": [
        "### Dataset\n",
        "For comparing different metrics, we will use the [Breast Cancer Dataset](https://goo.gl/U2Uwz2). This dataset is used for binary classification between two types of cancer."
      ]
    },
    {
      "metadata": {
        "id": "glPEglrYGV6f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84e17fb1-7b4a-4eba-ab47-e05017c487a8"
      },
      "cell_type": "code",
      "source": [
        "data = load_breast_cancer()\n",
        "print(data.DESCR[:760])  # gives a short description of dataset"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ".. _breast_cancer_dataset:\n",
            "\n",
            "Breast cancer wisconsin (diagnostic) dataset\n",
            "--------------------------------------------\n",
            "\n",
            "**Data Set Characteristics:**\n",
            "\n",
            "    :Number of Instances: 569\n",
            "\n",
            "    :Number of Attributes: 30 numeric, predictive attributes and the class\n",
            "\n",
            "    :Attribute Information:\n",
            "        - radius (mean of distances from center to points on the perimeter)\n",
            "        - texture (standard deviation of gray-scale values)\n",
            "        - perimeter\n",
            "        - area\n",
            "        - smoothness (local variation in radius lengths)\n",
            "        - compactness (perimeter^2 / area - 1.0)\n",
            "        - concavity (severity of concave portions of the contour)\n",
            "        - concave points (number of concave portions of the contour)\n",
            "        - symmetry\n",
            "        - fractal dimension (\"coastline appr\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "XQyaWAlhSTLL"
      },
      "cell_type": "markdown",
      "source": [
        "Let's see what are the targets classes i.e. types of cancer."
      ]
    },
    {
      "metadata": {
        "id": "bStQWeSOSSp2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc661365-db7b-4b5a-b261-6044a9709d3d"
      },
      "cell_type": "code",
      "source": [
        "print(f\"Types of cancer (targets) are {data.target_names}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Types of cancer (targets) are ['malignant' 'benign']\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "JizLx4ZfSZ-y"
      },
      "cell_type": "markdown",
      "source": [
        "What is the dimension of data? \n",
        "\n",
        " We see there are **569 examples** and each example has **30 features**.  \n",
        "The **target** variable is binary (**0** and **1** for `malignant` and `benign`).   "
      ]
    },
    {
      "metadata": {
        "id": "xG-br8XcHXHC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0114d726-34a5-4a03-c29d-271d92b4c81e"
      },
      "cell_type": "code",
      "source": [
        "X = data.data  # features\n",
        "y = data.target  # labels of the target\n",
        "print(f\"Shape of features is {X.shape}, and shape of target is {y.shape}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of features is (569, 30), and shape of target is (569,)\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "zpdfCVt8TW5K"
      },
      "cell_type": "markdown",
      "source": [
        "### Split the data\n",
        "Since we shouldn't train and test our model with the same dataset, we split the dataset into training and testing data, with 369 examples for training, and 200 examples for testing."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "`stratify` ensures that the dataset is split in such a way that the same proportions of examples is preserved in each class."
      ],
      "metadata": {
        "id": "6oUrsP2Tf81G"
      }
    },
    {
      "metadata": {
        "id": "QsgrNr2RINsN"
      },
      "cell_type": "code",
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=200, random_state=42, stratify=y)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YAU7JsyNJhc-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a003eaf3-414e-4b4f-fd9a-34c5428c6b91"
      },
      "cell_type": "code",
      "source": [
        "y_train[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 1, 1, 0, 1, 1, 1, 0, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "metadata": {
        "id": "DVCi5RMsJE-O"
      },
      "cell_type": "markdown",
      "source": [
        "### Training and predicting data"
      ]
    },
    {
      "metadata": {
        "id": "4iRqwJU-TmIt"
      },
      "cell_type": "markdown",
      "source": [
        "In this example, we will use scikit's Support Vector Machines classifier to predict whether its a benign cancer.  \n",
        "`SVC` classifier is used from `sklearn.svm`. \n",
        "So basically we are using the SVM Linear classfier to predict whether it is benign or malignant cancer.\n",
        "`Suggestion` Try using other classifiers and compare the accuracies."
      ]
    },
    {
      "metadata": {
        "id": "4pKL5VoJKh8A"
      },
      "cell_type": "code",
      "source": [
        "classifier = svm.SVC(kernel='linear', probability=True, verbose=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6DwkhGMyT2BJ"
      },
      "cell_type": "markdown",
      "source": [
        "Next we fit/train the model on our _training dataset_. It trains quite fast since we are working with relatively small dataset."
      ]
    },
    {
      "metadata": {
        "id": "1rj0E-XnLWaH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6bb66453-df2b-4fa6-d798-da997bb65cd1"
      },
      "cell_type": "code",
      "source": [
        "classifier.fit(X_train, y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[LibSVM]"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SVC(kernel='linear', probability=True, verbose=True)"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Once we have chosen and fit a Machine Learning Model, we can use it to make predictions on new data instances."
      ],
      "metadata": {
        "id": "omdMm3nvY1Fd"
      }
    },
    {
      "metadata": {
        "id": "hZvqwrrAT6UV"
      },
      "cell_type": "markdown",
      "source": [
        "Now save the prediction results both as probability and as classes.  \n",
        "**`y_preds`** is a **1D vector** of one of `{0, 1}` values, denoting predictions as malignant and benign, respectively.  \n",
        "**`y_proba`** is a **2D vector**, where for each example, it contains a vector of length 2,  \n",
        "`[prob. of malignant, prob. of benign]`"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can predict the class for new data instances using our finalized classification model using the predict() function. This gives us the class prediction.\n",
        "\n",
        "You can make these types of probability predictions by calling the predict_proba() function"
      ],
      "metadata": {
        "id": "wWBD3DdYZXkQ"
      }
    },
    {
      "metadata": {
        "id": "1PdjKHbfT6AX"
      },
      "cell_type": "code",
      "source": [
        "y_preds = classifier.predict(X_test)\n",
        "y_proba = classifier.predict_proba(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "b-9Dsn3yGXx7"
      },
      "cell_type": "markdown",
      "source": [
        "We need to reshape `y_proba` to a 1D vector denoting the probability of having  `benign` cancer."
      ]
    },
    {
      "metadata": {
        "id": "To9oQknC19dh"
      },
      "cell_type": "code",
      "source": [
        "y_proba = y_proba[:,1].reshape((y_proba.shape[0],))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "J_VDguyUL7fu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b63cc69d-4608-4562-ca47-8a1aaaf915fd"
      },
      "cell_type": "code",
      "source": [
        "y_proba[:5], y_preds[:5], y_test[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([0.99697263, 0.081367  , 0.99999168, 0.96809411, 0.99999907]),\n",
              " array([1, 0, 1, 1, 1]),\n",
              " array([1, 0, 1, 1, 1]))"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "metadata": {
        "id": "QM2WA4_FJfgT"
      },
      "cell_type": "markdown",
      "source": [
        "### Confusion Matrix"
      ]
    },
    {
      "metadata": {
        "id": "HEtVeYiBUDvH"
      },
      "cell_type": "markdown",
      "source": [
        "Let us calculate confusion matrix of the predictions. It is implemented in Scikit-learn's [`sklearn.metrics.confusion_matrix`](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html)."
      ]
    },
    {
      "metadata": {
        "id": "gXSq_kfcN8U8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "01b8cf42-b54b-4227-9cdf-111d5c79c5cd"
      },
      "cell_type": "code",
      "source": [
        "conf = metrics.confusion_matrix(y_test, y_preds)\n",
        "conf"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 68,   7],\n",
              "       [  2, 123]])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "metadata": {
        "id": "FISHOdmxqPXl"
      },
      "cell_type": "markdown",
      "source": [
        "We can also implement our own confusion matrix.  "
      ]
    },
    {
      "metadata": {
        "id": "e1yDrh9AqZjB"
      },
      "cell_type": "code",
      "source": [
        "def get_confusion_matrix(y_true, y_pred):\n",
        "    n_classes = len(np.unique(y_true))\n",
        "    conf = np.zeros((n_classes, n_classes))\n",
        "    for actual, pred in zip(y_true, y_pred):\n",
        "        conf[int(actual)][int(pred)] += 1\n",
        "    return conf.astype('int')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "txQRtyAZrnbG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "80fe7c35-8cf0-4577-ff11-7d55c58ce9f2"
      },
      "cell_type": "code",
      "source": [
        "conf = get_confusion_matrix(y_test, y_preds)\n",
        "conf"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 68,   7],\n",
              "       [  2, 123]])"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "metadata": {
        "id": "PiSZNOpnOeYu",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        },
        "outputId": "7f9d6d59-74b7-48e5-d6d0-19477727d83a"
      },
      "cell_type": "code",
      "source": [
        "classes = [0, 1]\n",
        "# plot confusion matrix\n",
        "plt.imshow(conf, interpolation='nearest', cmap=plt.cm.Greens)\n",
        "plt.title(\"Confusion Matrix\")\n",
        "plt.colorbar()\n",
        "tick_marks = np.arange(len(classes))\n",
        "plt.xticks(tick_marks, classes)\n",
        "plt.yticks(tick_marks, classes)\n",
        "\n",
        "fmt = 'd'\n",
        "thresh = conf.max() / 2.\n",
        "for i, j in itertools.product(range(conf.shape[0]), range(conf.shape[1])):\n",
        "    plt.text(j, i, format(conf[i, j], fmt),\n",
        "             horizontalalignment=\"center\",\n",
        "             color=\"white\" if conf[i, j] > thresh else \"black\")\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.ylabel('True label')\n",
        "plt.xlabel('Predicted label')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 12.5, 'Predicted label')"
            ]
          },
          "metadata": {},
          "execution_count": 17
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVUAAAEpCAYAAAAj7LGUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwU9ZnH8U/1oICKF4gKiBh1Hi+8rxhRk2h0s7rRmOii4pVowBPNYTSuaGIMMW48AglsEhMjipFo3CTebDzwiBoE48UjKiCXcjkqyDkz+0fVQNPMTFf3VHdXT3/fvPo101XVVc8MLx6eqt+vngqam5sREZFkZCodgIhIZ6KkKiKSICVVEZEEKamKiCRISVVEJEFKqiIiCVJSFcysu5n91cw+MrMJHdjP6Wb2WJKxVYKZPWxmZ1U6DqlOgeapVg8zOw24HNgN+ASYCvzY3Z/p4H6HABcDh7n7mg4HmjAzOwp4AnjA3U/KWr4P4e/gKXc/KsZ+rgV2cfczShOpiCrVqmFmlwO3ADcA2wL9gV8CX0lg9zsCb6UxoWZZCHzWzHpmLTsLeCupA5hZYGb6NyEdokq1CpjZFsBc4Bx3b/X03My6Aj8FTokW3Qtc4e4ro0pvHHAzcAXQCFzl7r8zs+uAK4EAWAlcCuxAVkVnZgOAGcBG7r7GzM4GrgG2ARYBV7v7XdHyb7r74dHnDgNuBeoJk9+l7v5ctO5JYBLwBWBv4HngNHdf1MrP1hL/34BX3X20mdUB7wFjgS+0VKpmdivwVWALYDow3N0nmdlxwF+yfs533H2fKI5ngaOA/YGBwG+Ace7+GzP7FdDb3U+O9v9T4EDgaHfXPx7ZgP5Xrg6fBboBf25nmx8AhwL7AvsABwNXZ63fjjDR9AW+AYw2s63cfQRh9ftHd9/M3X/bXiBmtilwG/Bv7t4DOIzwFDx3u62BB6NtewI/Bx7MqTRPA84BegMbA99p79jAH4Azo++PBV4D5uVs8xLh72Br4G5ggpl1c/dHcn7OfbI+MwQ4H+gBzMrZ37eBgWZ2tpkNIvzdnaWEKm1RUq0OPYFFeU7PTwd+6O4L3H0hcB1hsmixOlq/2t0fApYCVmQ8TcBeZtbd3ee7++utbPPvwHR3v9Pd17j7eGAacELWNr9z97fcfTlhZb1veweNqtytzcwIk+sfWtlmnLsvjo7530BX8v+cv3f316PPrM7Z36eEv8efE1bLF7v7nDz7kxqmpFodFgO9zKxLO9v0Yf0qa1a0bO0+cpLyp8BmhQbi7suAU4GhwHwze9DMdosRT0tMfbPev19EPHcCFwGfp5XK3cy+Y2ZvRjMZGgir81559jm7vZXu/gLwLuGlg3tjxCg1TEm1OjxPeB3wxHa2mUc44NSiPxueGse1DNgk6/122Svd/VF3PwbYnrD6/HWMeFpimltkTC3uBC4AHoqqyLWi0/PvEV5X3srdtwQ+IkyGAG2dsrd7Km9mFxJWvPOi/Yu0qb3KR1LC3T8ys2sIr4OuAR4jPJ0/Gvi8u38PGA9cbWYvESaJawhPV4sxFbjCzPoTJqUrW1aY2baE124nAssJLyM0tbKPh4BfRNPA7gVOBvYgHGwqmrvPMLMjCSvHXD2ANYQzBbqY2feBzbPWfwAcY2YZd28t5g2YWT1wPeFA1qfAi2b2sLtvcB1ZBFSpVo3o+uDlhINPCwlPWS8CHog2uR74J/Av4FXg5WhZMcd6HPhjtK/JrJ8IM1Ec84AlwJHAsFb2sRg4nnCgZzFhhXd8a6P7RcT3jLu3VoU/CjxCONNgFrCC9U/tW2ZOLDazl/MdJ7rcMg74qbu/4u7TgauAO6PZFiIb0JQqEZEEqVIVEUmQkqqISIKUVEVEEpT60f9oQOAgYD7h7ZUikg51hNPqXnL3lUnsMLoTb/O8G67zsbsvSeLYSUl9UiVMqJMqHYSItGkQ0KFOaRAm1MZM8+K6piD/xut8aGa7pCmxVkNSnQ+w1/CD6LpVt0rHIh10zSGXVToEScgH7y/g/LOHQvRvNAGb1zUFvNd7OWu65J+V1GVNQP8F3bcirGyVVAvQCNB1q25069m90rFIB/Xp2yf/RlJtEr0st2YjWBMnMxVU0JZPNSRVEaklAfGG0AtIqmZ2E+FdfQOAge7+WtQx7U5gZ2AVYavIb0UNiTCzQwlbS3YHZgJnuPuCfMfS6L+IpEsQxH/F9wBwBOs3+WkGbnR3c/eBwDvASICoWfk44EJ3rweeblmXjypVEUmXTBCv3MusTar9wm6Q62lw94aWNy2PHMreLhrcejLrM/9g3S3XBwArsh5VNIawWj03b1gxQhcRKZ9MAa/QJMInU2S/hhdyyKgyHUb4dAgIO6qtrWqjnhWZaMpXu1Spiki6BMQ7tV+3ySAgt3F4A4X5BWHHtVEFfm4DSqoikj6FjezPcfeZxR4qGsTaFTghqyXke2T1AzazXkBTnPmwOv0XkXTJBPFfHWRmNxBePz0x566wyUB3Mzs8ej+Uda0j26VKVUTSJSBepVrYlKrbCJ+yux0w0cwWEz4h4krC/rvPRYNYM9z9JHdvMrMhwFgz60Y0pSrOsZRURSRdgiDxpOrulwCXFLKX6EGTA+MfJaSkKiLpUoJKtZyUVEUkXWLPUy15JEVRUhWRdAliJlVVqiIiMahSFRFJkK6piogkqASj/+WkpCoi6aJKVUQkQRqoEhFJkCpVEZEExe5Slf85VpWgpCoi6bJ+r9T2t0shJVURSZe4HagyED4RJV2UVEUkXeI+fyoAJVURkXxK8DTVclJSFZF0KahSTR8lVRFJF02pEhFJUEEDVemjpCoi6aLTfxGRBOn0X0QkOQEBQYxKNaU5VUlVRNIlk4FMjGuqmUz65qiCkqqIpEwmCMjEqFTjbFMJSqoikipBEPP0X0lVRCSGmEk11gyBClBSFZFUUaUqIpKg2NNU05lTlVRFJF3iT6lKZ1ZVUhWRVNHpv4hIgoLoT5zt0khJVURSJZOB5liT/+Pv08xuAk4GBgAD3f21aHk9cAfQE1gMnOnu0/Otazeu+GGJiJReEE3+z/cq8PT/AeAIYFbO8jHAaHevB0YDY2Oua5MqVRFJlSKuqfYzs9zVDe7e0PLG3Z8ByN7OzHoD+wPHRIvGA6PMbBvC1gKtrnP3he3FpUpVRFKlJanGeUUmATNyXsNjHGoHYK67NwJEX+dFy9tb1y5VqiKSKkXMUx0EzMlZ3UCFKKmKSKoUMU91jrvPLOJQs4G+Zlbn7o1mVgf0iZYH7axrl07/RSRVijj9L4q7LwCmAoOjRYOBKe6+sL11+farSlVEUqUUk//N7Dbgq8B2wEQzW+zuewJDgTvM7BrgQ+DMrI+1t65NSqoiki4l6FLl7pcAl7SyfBpwSBufaXNde5RURSRVgpjPqErpXapKqpXSvUt3ztljMH0360NzczO/e+MuVjWu5szdT2Wjuo1oam7izjfvZcbHuXOVJc2m+3TOOeMba9/PnDGTq665kgsuGVbBqKpLJhPQHON2qSDTVIZoCqekWiGn2cm8uvhNfvmv26kL6ti4bmOG7X0Of3n3EV5d/AYDe+3B13f9CjdOvq3SoUoBdrVdeealpwFobGxkt5325PivHF/hqKpLJghoruKGKhr9r4DuXbpRv9UuTJr7PACNzY0sX7McmqFbl24AbNKlOw0rP6pkmNJBT/79KXb6zAD675h3vrhkCVg3V7XdV6UDbYMq1Qro1a0nn6xayrl7nsEOm/Vh1iezuXvafYx/6z4u3+8CTq0/kYCAG176eaVDlQ64f8L9fO2UkysdRtUJYs7+T2ulWrakWmzHl86oLpNhxx79uHvaBN79eBaD7WT+fadj6N6lG/e8dT+TF7zCQdvuxzl7nM5NL4+qdLhShFWrVvHQ3x5hxI+uqXQoVShe67+01qrlPP0vquNLZ7RkRQMfrmzg3WgQ6p8fTKV/jx04bPtDmLzgFQBe+mAKO23Rv5JhSgc8/shE9tl3b3pv27vSoVSdck3+L5WyVKrtdYPJvkPBzLYEtsz5eL9yxFhOH6/6hCUrGthuk968/+kC9ti6nnnL5rNN957YVrvgH77N7lvX88GneW/ekJT607338bVTdepfjLin/2mdU1Wu0/8NOr6YWUvHl+zMMRwYUaaYKuquaRM4f+BZ1AV1LFy+mNtfH8fUha8y2E6mLqhjddNq7njjnkqHKUVYtmwZT/zfk9wy+uZKh1KV4s5TTenZf+oGqm4Bfp+zrB9ha69OZfbSufzwhZ+tt2x6w7sbLJPqs+mmmzJz/juVDqNqqVKNp71uMGtFTWXXa9nVSvNZEenEMpkg3rNSYjxypRLKMlDVkY4vIlJbNE81vqI6vohIjYk5sh/nrqtKKFtSLbbji4jUltjTpYKA5tKHU7C0DVSJSI1TUhURSVDcwf+0XlRVUhWRVCmkUk0jJVURSRUlVRGRBCmpiogkKAiiGwDyaEpnTlVSFZGU0W2qIiLJKcUjqstJSVVEUqXKC1UlVRFJF1WqIiIJUlIVEUmQkqqISIJaWv/F2S6NlFRFJF00+V9EJDmZTEAmRuf/5pR2/ldSFZFUKdU1VTM7HvgR0RUG4Dp3v9/M6oE7gJ7AYuBMd59eaNwtyvI4FRGRuGI9SiVue8CImQXAncAQd98XGEL4JJIMMAYY7e71wGhgbEfiV6UqIqlSRKXar5UHhDZEDxLN1gRsEX2/JTAf6AXsDxwTLR8PjDKzbYp9hp4qVRFJlZakGucVmQTMyHkNz96nuzcDpwD/a2azgAcIn5O3AzDX3Ruj7RqBedHyoqhSFZFUCYhZqa6bVDUImJOzOvdR912AK4GvuPuzZvY54F7CywCJUlIVkXQp/HEqc9x9Zp6t9wX6uPuzAFFiXQasAPqaWZ27N5pZHdAHmF1U7Oj0X0RSpojT/zjmEF57NQAz2x3YFpgOTAUGR9sNBqYUez0VlFRFJG1KMPzv7u8Dw4A/mdkrwD3Aue6+BBgKXGxmbwEXR++LptN/EUmVukxAXYyJ/YVO/nf3u4C7Wlk+DTikoJ21Q0lVRFIlLELjTKkqQzBFaDOpmtmdQHO+Hbj7mYlGJCI1LRMEZGJkzDjbVEJ7lerbZYtCRGStuINQVZZU3f26cgYiIgLh6HmcEfS0jrLHvqZqZscA/wn0dvcTzOxAYHN3/3vJohORmlPtp/+xkr2ZXQz8inBO1xHR4uXA9SWKS0RqVInmqZZN3Ap6OHC0u48kbEoAMA3YoIuBiEhHtFSqcV5pFPf0vwfrbttqmRGwEbAq8YhEpLZVeef/uJXq08D3c5ZdAjyRbDgiUusyBbzSKG6lejHwVzM7D+hhZg58AhxfsshEpCaFd1TlT5lx7rqqhFhJ1d3nm9lBwEHAjoSXAl5096b2PykiUphqH/0v5DbVDOF1VIA60jrzVkSqXjUnl1hJ1cz2JuyU3RWYC/QDVpjZSe7+SgnjE5EakyFmpZrS1Bv3Wu/thA/E6ufuBwN9gVHRchGRxFT7lKq4SbUeuCV6zkvL815uBXYtVWAiUptqZfL/Q8B/5Cw7AXgw2XBEpNZVe6Uat/VfHXCPmU0mHPnfATgA+N+SRygiNSed6TKeQlr/vZb1/RvAo8mHIyK1rtNOqVLrPxGphJqY/A9gZhsTNlDpRVZ1rtZ/IpKkTlupZjOzw4EJhPNUNwc+Zl2Tlc+ULDoRqUnpTJfxxB39vxm40d23Bj6Jvv4I+GXJIhORmtQy+T/vK6Wpt5B5qrfmLBsJXJZsOCJS6zrtlKocHxGe9jcA881sD2AxsFmpAhOR2hR3Yn+1T/6/H/hy9P3thH1UJwN/KkVQIlK7aqKfqrsPz/r+JjN7gbBK1VxVEUlWlXf+L6T131ruPinpQEREoBNPqTKzSay7TbVN7n5Evm2S8KPDvk/fvn3KcSgpoe7H1Vc6BElIlzUBn2GTxPfbmSf//6ZsUYiIRDLEmy6V1ilV7d2mekc5AxERAUp2TdXMuhHOuT8aWAE87+7nm1k9cAfQk3BW05nuPr3AqNdK6wCaiNSoEk7+v5Ewmda7+0Dgv6LlY4DR7l5P2Ix/bMfiFxFJkaCAP3GZ2WbAmcB/ZTXb/8DMegP7A+OjTccD+5vZNsXGX9Tov4hIqRQx+b+fmeWubnD3hqz3OxOe2o8ws88DS4GrgeXAXHdvBHD3RjObR9gzemEx8atSFZFUKeI21UnAjJzX8Jzd1hE2f5ri7gcCVxDe1JT4XaFxu1R1Ba4BBgM93X0LM/sS4bWJUUkHJSK1Kzyxz1/vZZ3+DwLm5KxuyHn/HrCG6DTf3V8ws0WElWpfM6uLqtQ6oA9hB76ixD39v5nwCaqnAw9Hy16PliupikhigpiT/7NO/+e4+8z2tnX3RWb2BHAM8Fg04t8beAuYSlgwjou+TnH3ok79If7p/0nAae7+PNAUBTmXMNGKiCQmE2Rivwo0FLjKzF4F7gGGRNddhwIXm9lbwMXR+6LFrVRX5W4bjY4t7sjBRURyleo2VXd/FziqleXTgEMK2lk74qb6CcAdZrYTgJltT3jaf09SgYiIwLrR/zivNIqbVK8iHFF7FdgSmA7MA/RwQBFJVEAm1p84g1mVELf13yrCLv+XRaf9i1om0IqIJCkI4jWgTmmhGntKVe7D/Xq0TLaNrlOIiCQiIObk/2prqJLjbcI2gNk/RUulWpdoRCJS0zptl6ps7r7exQsz2w4YQXgng4hIYmrlGVXrcff3CW8D+0my4YhIrauVp6m2xqAEbb9FpKZlgjrqgvxXFTMxtqmEuANVuY9W2QTYE/hhKYISkdpVE6P/bPholWXAKx3pji0i0pq4vVKrdvQ/6tryBeB8d19Z+pBEpJZV+9NU8w5URc1bv0TUSEVEpJRK0fm/nOKO/t8MXGdmG5UyGBGRIObIf1qnVLV7+m9mg919PGE7rO2Ay81sIVmDVu7ev7QhikgtCYIMQYy2fnG2qYR811THEnbKPqMMsYiIdPqBqgDA3Z8qQywiImSINwiVzjo1f1Kti5482OZP6O5/TzYkEall1X6bar6k2hX4LW0n1WbCJxSKiCQiE2Soi3G9tIjHqZRFvqS6zN2VNEWkbDr7QJWISFl19tZ/6YxaRDqtTn1N1d17lCsQEREIK7l4U6rSSaf/IpIqnbpSFREptyDmNdVqnfwvIlJWGv0XEUlQZ79NVUSkrDJBJtbE/mqd/C8iUlYBMR+nUvpQiqKkKiKpooEqEZEEaUqViEiCguhG1TjbFcPMRgDXAgPd/TUzO5Swd3R3YCZwhrsvKGrnpLcloYjUqJZKNc6rUGa2P3AoMCt6nwHGARe6ez3wNDCyI/GrUhWRVCliSlU/M8td3eDuDdkLzKwrMBoYDDwZLT4AWOHuz0TvxxBWq+cWETqgSlVEUibOQ/9yHmM9CZiR8xreyq5/CIxz95lZy/oTVa0A7r4IyJjZ1sXGr0pVRFKliIYqg4A5Oatzq9TPAgcC3+9ofPkoqYpIqgRBHZmgLtZ2kTk51WdrjgR2B2ZElwr6AY8CtwE7tmxkZr2AJndfUnDgESVVEUmVUkypcveRZA1AmdlM4HjgDeB8Mzs8uq46FJhQWMTr0zVVEUmVzNpJVflfHeXuTcAQ4FdmNp2wou3QJQJVqiKSKuWY/O/uA7K+fw4YWPTOciipikiqVHuXKp3+V9js2XM49ov/xn4DD2D/vQ9k1G2jKx2SxPDbb9/EB/dO5dX/mbh22Y3nXc2bv32SV8Y+zv0jfsMWm24OwEG2L1PGPMqUMY8ydcxjnPi54yoVdlUIgrg3AFQ60tYpqVZYly51jPzZDUx5dTJPPfsEY3/1P7z5xpuVDkvy+P1jEzjuqjPWW/b4y0+z13lfZJ9vHcNbc9/lysEXAfDazGkceMGX2W/osRx31RmMvXQkdZn8o9u1K94V1bSmr3RGVUO233579tt/PwB69OjBbrsZ8+bOq3BUks+kV19gySfrTYXk8clP09jUCMA/3nyZfr22B2D5yhVrl3fbuCvNNJc32CqTIebk/5Se/uuaaorMmjmLqVNf4aBDDqp0KNJB5x57Kn986q9r3x+8237c/u2b2HHbfgz56aVrk6xsqNqvqZYlqZrZTcDJwACizjDlOG41Wbp0KYNPOY2f/fxGNt9880qHIx1w1WkXs6axkbv+7/61y16cNoW9zvsiu/XfhTu+ewsPv/gEK1evrGCU6VXtnf/LFdUDwBFk3WMr66xevZrBXz+NUwefyoknfaXS4UgHnPWlr3P8IUdz+siLWl0/7b23Wbp8GXvttEEDEImUsktVOZSlUm3pANNKJ5ma19zczNDzhmG7G5dedkmlw5EOOPbAo/jeKcM48ttfY/nKFWuXD9huB2YvmEdjUyP9e/dlt/47M/P92RWMNN3Czv9x+qnWcFKNy8y2BLbMWdyvErGUy3PPPs/d48az18A9OeSAQwG47kfXctyXNe0mze6+ahRH7f1Zem2xNbPvfokRf/hvrvzPi+i60cY8/tPxQDhYNezWKzl8r4P5/qkXsLpxDU1NTVxw2w9Y/PGHFf4J0kud/5M1HBhR6SDK6XOHH8byNcsqHYYU6LQbNjy9v/2Re1rddtzE+xg38b5Sh9RpaKAqWbcAv89Z1o+wX6KI1ABVqgmKOnXn9kGsUDQiUgnVXqmWZfTfzG4zszmEVedEM3u9HMcVkeoTFPAnjco1+n8JoKFtEckvvPk/3nYplKrTfxGR8DbUOJP/lVRFRGKIe2qvpCoikle1D1QpqYpIqmhKlYhIglSpiogkSElVRCRBATFP/5VURUTyC4iXMNOZUpVURSRlNFAlIpIozVMVEUlMtT9ORUlVRFJFo/8iIglSUhURSZAGqkREEpfOhBmHkqqIpEopKlUz6wncCewMrAKmA99y94VmdigwFugOzATOcPcFhUceSufwmYjUrBJ1/m8GbnR3c/eBwDvASDPLAOOAC929HngaGNmR+FWpikjKFDxPtV8rz7JriJ55B4C7LwGezFr/D2AYcACwwt2fiZaPIaxWzy047IgqVRFJlZbT/zivyCRgRs5reFv7j6rTYcBfgP7ArJZ17r4IyJjZ1sXGr0pVRFIlE/2Js11kEDAnZ3UDbfsFsBQYBZxURIjtUlIVkVQpYqBqjrvPjLNvM7sJ2BU4wd2bzOw9YMes9b2ApuhyQVF0+i8iqdLSpSrpB1Sb2Q2E11BPdPeV0eLJQHczOzx6PxSY0JH4VamKSKqUaErVnsCVwFvAc9HA1gx3P8nMhgBjzawb0ZSqIsJeS0lVRFKlFLepuvvrtHFHgbs/BwyMvbM8lFRFJGUC4t1Rlc67rpRURSR10pku41FSFZFU0TOqREQSpdN/EZHEqJ+qiEiCqr2fqib/i4gkSJWqiKRKyx1VcbZLIyVVEUmVar+mqtN/EZEEqVIVkVTRQJWIiKylSlVEUqXar6kqqYpIyhT8jKpUUVIVkVSp9muqSqoikjK6919EJDHVnVKVVEUkdao7rSqpikiqBEG866UpvaSqeaoiIklSpSoiqaJ5qiIiidI1VRGRxGSCgEyMC6ZxtqkEJVURSRlVqiIiianulKqkKiKpU91pVUlVRNIl5r3/aZ2oWg1JtQ7gg/c/qHQckoAua9L5D0EKl/V3WZfkfhfE/Lced7tyq4akuj3AOUPOrXQckoDPsEmlQ5DkbQ+8k8B+PgY+PGfIN7Yq4DMfRp9LjWpIqi8Bg4D5QGOFYymVfsAkwp9zToVjkY6rlb/POsKE+lISO3P3JWa2C7B5AR/72N2XJHH8pATNzc2VjqHmmdkAYAawk7vPrGw00lH6+6xtuvdfRCRBSqoiIglSUhURSZCSajo0ANdFX6X66e+zhmmgSkQkQapURUQSpKQqIpIgJVURkQRVwx1VnZ6Z1QN3AD2BxcCZ7j69slFJMczsJuBkYAAw0N1fq2xEUm6qVNNhDDDa3euB0cDYCscjxXsAOAKYVelApDKUVCvMzHoD+wPjo0Xjgf3NbJvKRSXFcvdn3H12peOQylFSrbwdgLnu3ggQfZ0XLReRKqOkKiKSICXVypsN9DWzOoDoa59ouYhUGSXVCnP3BcBUYHC0aDAwxd0XVi4qESmWblNNATPbjXBK1VaEnczPdHevbFRSDDO7DfgqsB2wCFjs7ntWNiopJyVVEZEE6fRfRCRBSqoiIglSUhURSZCSqohIgpRURUQSpKQqbTKz35vZ9dH3g8ysLNO8zKw5ev57a+ueNLNvxtzPTDM7usgYiv6s1Da1/qtyZjYT2BZoBJYBDwMXufvSJI/j7pMAixHP2cA33f3wJI8vUi1UqXYOJ7j7ZoTdrg4Ers7dwMz0H6hIGegfWifi7nPN7GFgLwhPo4GLgOGEf9c7mdnxwPWETZTfAIa6+7+i7fcDfgvsCjwErL0zxMyOAsa5e7/o/Q7ArcAgwv+cxxP2gh0DbGRmS4E17r6lmXUFfgycAnQF/gxc5u7Lo319F7g8Ot4G/yG0xcx2Bn4N7BN99lHgQnfPforpQdFdTtsT9jod5u4ros+3+bsQKZYq1U4kSnRfBqZkLT4ROATYI0qatwPfInzKwFjgL2bW1cw2Jkw6dwJbAxMIO9i3dpw64G+EjZgHAH2Be9z9TWAo8Ly7b+buW0YfGQnUA/sCu0TbXxPt6zjgO8AxhMm8kOuYAfATwgY0uxO2S7w2Z5vTgWOBnaMYro6O2+bvooDji2xAlWrn8ICZrQE+Ah4Ebsha9xN3XwJgZucDY939hWjdHWZ2FXAoYaW3EXCLuzcDfzKzy9s43sGEiey77r4mWvZMaxuaWQCcD+ydFccNwN3AlYTV6+9aHjtiZteyrrlMu9z9beDt6O1CM/s5MCJns1EtTaPN7MfALwgTa3u/i6fiHF+kNUqqncOJ7j6xjXXZLQR3BM4ys4uzlm1MmCCbCZtlZzeDaOuRIDsAs7ISanu2ATYBJputHecKgLro+z7A5BjH3ICZbcu6SxA9CM+8PszZLPvnnxUdD9r/XYgUTUm188tOkrOBH7v7j3M3MrMjCfu6BlmJtT/wTiv7nA30N7MurSTW3A49i4DlwJ7uPreVfc1n/acc9G/7R9nADdHxBrr7EjM7ERiVs03uvudl/Qyt/oi6mLIAAAEgSURBVC5EOkJJtbb8GvizmU0EXiSsII8CngaeB9YAl5jZL4ETCE/zn2hlPy8SJsORZjaCcDrXAe7+LPAB0M/MNnb3Ve7eZGa/Bm42s4vcfYGZ9QX2cvdHgXuB35nZH4CZbHj63p4ehJc8Por2+d1WtrnQzP4GfAr8APhjvt+Fu39SQAwi69FAVQ1x938C5xFWcx8SXo88O1q3irAP6NnAEuBU4P429tNImHR3Ad4D5kTbA/wdeB1438wWRcuuiI71DzP7GJhINOfV3R8Gbok+93b0Na7rCKeRtVxLbi3eu4HHgHcJq+7r8/0uRDpC/VRFRBKkSlVEJEFKqiIiCVJSFRFJkJKqiEiClFRFRBKkpCoikiAlVRGRBCmpiogk6P8Blb0pyFRuAiYAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "v81eYMJ0UM29"
      },
      "cell_type": "markdown",
      "source": [
        "From the confusion matrix, we can see the number of examples predicted correct by our classifier, for both classes seperately.  \n",
        "We can get the numbers of `True Positives, True Negatives, False Positives, and False Negatives` from this confusion matrix.   \n",
        "Lets store these terms in some variables."
      ]
    },
    {
      "metadata": {
        "id": "gjTIQRccOr-A"
      },
      "cell_type": "code",
      "source": [
        "# from the confusion matrix\n",
        "TP = true_pos = 123\n",
        "TN = true_neg = 68\n",
        "FP = false_pos = 7\n",
        "FN = false_neg = 2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "USDonfT5Jpr2"
      },
      "cell_type": "markdown",
      "source": [
        "### Some basic metrics"
      ]
    },
    {
      "metadata": {
        "id": "X5dcb4jEPpDR"
      },
      "cell_type": "markdown",
      "source": [
        "Now, we will calculate some basic metrics from these four values.  \n",
        "We will need a dictionary to store these metrics. Lets create a dictionary `results`."
      ]
    },
    {
      "metadata": {
        "id": "wFE2EKmxV_Xx"
      },
      "cell_type": "code",
      "source": [
        "results = {}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "U6O_iUIHZqgj"
      },
      "cell_type": "markdown",
      "source": [
        "#### Accuracy \n",
        "number of examples correctly predicted / total number of examples  \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/c72ec21ef2505c2d376e96197637fc64f75e5891)"
      ]
    },
    {
      "metadata": {
        "id": "4y3hcfI0WFfv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ef84aaf1-9593-4fc4-ee64-befa8402d69c"
      },
      "cell_type": "code",
      "source": [
        "metric = \"ACC\"\n",
        "results[metric] = (TP + TN) / (TP + TN + FP + FN)\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ACC is  0.955\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "vjmFM1v_Z_a8"
      },
      "cell_type": "markdown",
      "source": [
        "#### True Positive Rate\n",
        "number of samples actually and predicted as  `Positive` / total number of samples actually `Positive`  \n",
        "Also called **Sensitivity or Recall**.  \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/64d5540cbadeb83f864d7a731b7ab43cccd0f353)\n"
      ]
    },
    {
      "metadata": {
        "id": "q8JoI-S6WKMS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "50bfbdb4-d393-4144-f32a-aba69654eaf0"
      },
      "cell_type": "code",
      "source": [
        "# Sensitivity or Recall\n",
        "metric = \"TPR\"\n",
        "results[metric] = TP / (TP + FN)\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TPR is  0.984\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "f6y1c1SDa0Lg"
      },
      "cell_type": "markdown",
      "source": [
        "#### True Negative Rate\n",
        "number of samples actually and predicted as  `Negative` / total number of samples actually  `Negative`  \n",
        "Also called **Specificity**.  \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/aac65b6586cb61c1268403895012c2a8f19b0c5e)\n"
      ]
    },
    {
      "metadata": {
        "id": "rkRmDKjSYLYE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5788aab1-8c35-49c4-8ac0-a646844587de"
      },
      "cell_type": "code",
      "source": [
        "# Specificity\n",
        "metric = \"TNR\"\n",
        "results[metric] = TN / (TN + FP)\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TNR is  0.907\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "qK7WCIlpbDis"
      },
      "cell_type": "markdown",
      "source": [
        "#### Positive Predictive Value\n",
        "number of samples actually and predicted as  `Positive` / total number of samples predicted as `Positive`  \n",
        "Also called **Precision**.  \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/699fcdb880b7f6a92742bc0845b8b60b59806a98)\n"
      ]
    },
    {
      "metadata": {
        "id": "lNgS7UwfYce9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bf983800-c761-439e-cdfe-bdcbd3a96971"
      },
      "cell_type": "code",
      "source": [
        "# Precision\n",
        "metric = \"PPV\"\n",
        "results[metric] = TP / (TP + FP)\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PPV is  0.946\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "VBPShM1JdQpO"
      },
      "cell_type": "markdown",
      "source": [
        "#### Negative Predictive Value\n",
        "number of samples actually and predicted as  `Negative` / total number of samples predicted as `Negative`    \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/b6da0f59faffbc1704974e97847ab4fa599f5521)\n"
      ]
    },
    {
      "metadata": {
        "id": "qAAhCMaOdsSn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "081151d5-e62b-42fa-ca96-58af0e9d5471"
      },
      "cell_type": "code",
      "source": [
        "metric = \"NPV\"\n",
        "results[metric] = TN / (TN + FN)\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NPV is  0.971\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "D6mxQtZyfIte"
      },
      "cell_type": "markdown",
      "source": [
        "#### F1 score\n",
        "Harmonic Mean of Precision and Recall.  \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/5663ca95d471868169c4e4ea57c936f1b6f4a588)"
      ]
    },
    {
      "metadata": {
        "id": "8pNnIhEbdzFc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df9935be-a545-4a1e-c0c4-658295e8df48"
      },
      "cell_type": "code",
      "source": [
        "metric = \"F1\"\n",
        "results[metric] = 2 / (1 / results[\"PPV\"] + 1 / results[\"TPR\"])\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "F1 is  0.965\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "ZF98MS34edwA"
      },
      "cell_type": "markdown",
      "source": [
        "#### Matthew's correlation coefficient\n",
        "Matthew's coefficient range between `[-1, 1]`. `0` usually means totally random predictions. `1` means a perfect classifier, while a negative value (`[-1, 0)`) suggests a negative correlation between predictions and actual values. A negative value basically indicates that there exists a disagreement between the prediction and observation.  \n",
        "Here' the formual for MCC  \n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/5caa90fc15105b74b59a30bbc9cc2e5bd43a13b7)"
      ]
    },
    {
      "metadata": {
        "id": "IxtxoxxmgZxI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a30bbb41-ab0e-43e2-f08b-91ec963590c2"
      },
      "cell_type": "code",
      "source": [
        "metric = \"MCC\"\n",
        "num = TP * TN - FP * FN\n",
        "den = ((TP + FP) * (TP + FN) * (TN + FP) * (TN + FN)) ** 0.5\n",
        "results[metric] = num / den\n",
        "print(f\"{metric} is {results[metric]: .3f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "MCC is  0.904\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "7UI7NWzcpu8p"
      },
      "cell_type": "markdown",
      "source": [
        "### ROC curve (Receiver Operating Characteristic curve)\n",
        " A receiver operating characteristic curve, i.e. **ROC curve**, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied.  \n",
        " The ROC curve is created by plotting the **True Positive Rate (TPR)** against the **False Positive Rate (FPR)** at various threshold settings.  \n",
        " Let's plot the ROC curve for the Breast Cancer Dataset.  \n",
        " "
      ]
    },
    {
      "metadata": {
        "id": "1-gXSQ_HrdnI"
      },
      "cell_type": "code",
      "source": [
        "def get_roc_curve(y_test, y_proba, delta=0.1):\n",
        "    \"\"\"\n",
        "    Return the True Positive Rates (TPRs), False Positive Rates (FPRs), \n",
        "    and the threshold values, seperated by delta.\n",
        "    \"\"\"\n",
        "    thresh = list(np.arange(0, 1, delta)) + [1]\n",
        "    TPRs = []\n",
        "    FPRs = []\n",
        "    y_pred = np.empty(y_proba.shape)\n",
        "    for th in thresh:\n",
        "        y_pred[y_proba < th] = 0\n",
        "        y_pred[y_proba >= th] = 1\n",
        "        \n",
        "        # confusion matrix from the function we defined\n",
        "        (TN, FP), (FN, TP) = get_confusion_matrix(y_test, y_pred)\n",
        "        \n",
        "        TPR = TP / (TP + FN)  # sensitivity\n",
        "        FPR = FP / (FP + TN)  # 1 - specificity\n",
        "        TPRs.append(TPR)\n",
        "        FPRs.append(FPR)\n",
        "    return FPRs, TPRs, thresh"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "W3iQtSMKJ63N"
      },
      "cell_type": "markdown",
      "source": [
        "Here, `delta` is the difference between each of the thresholds at which ROC curve is calculated."
      ]
    },
    {
      "metadata": {
        "id": "2zkIsIz31HGY"
      },
      "cell_type": "code",
      "source": [
        "delta = 0.001\n",
        "FPRs, TPRs, _ = get_roc_curve(y_test, y_proba, delta)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Igzm5LZDzsNZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "outputId": "49d0fd27-0489-4d90-f8cf-72e3e0b0a935"
      },
      "cell_type": "code",
      "source": [
        "# Plot the ROC curve\n",
        "plt.plot(FPRs, TPRs, color='red',\n",
        "         lw=2, label='ROC curve')\n",
        "plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label=\"Mean\")\n",
        "plt.xlim([-0.05, 1.0])\n",
        "plt.ylim([0.0, 1.05])\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.title(f'ROC curve (threshold delta = {delta})')\n",
        "plt.legend(loc=\"lower right\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEcCAYAAAAGD4lRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3gU1dfA8e+m0HsJLVSRI4KAVAuoVAFBQcUC/iyAUVB4FSxgxQZYUFFEQZoNG6I0FVQUxIKIIkU9iNI7SCCU9H3/mEkMIWUTsrsp5/M8ebJ7c3fmzCaZs/femXs9Xq8XY4wxJjdCgh2AMcaYgsuSiDHGmFyzJGKMMSbXLIkYY4zJNUsixhhjcs2SiDHGmFwLC3YAxuQFETkbeBNoo6qnXLcuIvWAzUC4qib6ORYvcKaqbsrh6+qRRYwiMgZoqKo35GUcInIJ8LaqRuYk3sJIRIoDvwEdVHV/sOMpCCyJFFAisgWoBiQBR4HPgTtV9WiaOhcATwJtgGRgOXC/qv6epk454HHgSqASsBdYADypqgcCcSx55AnguZQE4r4/g1X1y2AGVRAF670TkUrAdKAbcAAYraqzM6nrAcYDg92iacCoNL//Fu62GgN/AINUdY37s47AI0BL4JCq1kvZrqrGicgMYBQwMq+PsTCy7qyCrbeqlgFaAOcCo1N+ICLnA0uAeUBNoD7OJ6zvRKSBW6cY8BXQBOgOlAPOBw4Cbf0VtIjk6YcXEakBdAQ+ycNt2geswHsFiMf5cDQAeFVEmmRSNwroAzQHmgG9gdsg9e96HvA2UBF4A5jnlgMcA2YA92ay7dnATW6rxGTD/lEKAVXdIyKLcZJJimeAN1V1Ypqyh0SkFTAGuNH9qgN0TNOC2YfzqT5D7j/1i0ArIAGYqKpjRWQWsENVH3LrXUKaLhL30+2rOCcHEZFHcLqerk6z7YmAR1WHi0h54HmgJ04raibwqKomZRBWV+AXVY11t/OWe1wLRCQJp6X1gVt3gIg8AZQCXlDVp9zXjAGaArHA5cAIEfkwsxhEpCHOJ90W7vvwlapemyamLiLyGVAVeAenlegVkRDgAeBWoCROC3KYqh7O4L2uD8zC+cT8I6AZHHva+vcCIwAv8FC6nxUHngKuAYoDHwN3q+qJdPVOee9U9Rn3vejgxvwbMERVN2QVT06ISGngKqCp+7e4QkTmA//DaRWkdxMwQVV3uK+fgPOevgZcgnNue9FtmbwkIvcAnYDPVfUn4CcR6ZJRLKq6Q0QOAecBy/LqGAsra4kUAiISCfQANrnPSwEXAB9mUP0DnJMuQBecf6qjGdTLaD9lgS9xTnw1gYY4LRlfXQ9cBlQA3gN6uttEREJxTnAp3RezgER3H+fidHEMJmPnkOYEq6r/A7bhttRU9Zk0ddsDAnQGHhGRxml+dgUwx43vnWxieAKnpVcRiAReThdTL5xuxGbucV3qlt/sfnUEGgBlgEmZHNdsYDVQxd3fTZnUQ0S6A/fg/G7PxPndpjUeaIST9BoCtXC6dE6SxXv3mbvdCOAXnPcns1gmi0h0Jl9rM3lZIyBRVTemKfsNp5WckSbuzzOq2wRYm25sbG0W28rIHzitHJMNa4kUbJ+4g6dlgKXAo255JZwPCLszeM1unJMSQGWck5SvegF7VHWC+zwWWJmD17+kqtvdx1tF5BegL86AeCfguKr+KCLVcD79V3A/KR8TkRdwujCmZLDdCjhdcL54zN3mbyLyG86J4g/3Zz+o6ieQOlaUVQwJQF2gpvtpeEW6/YxX1WggWkS+xjl5f47TEnteVf9x9zMaWC8it6R9sYjUwUlCXVQ1DlguIguyOK5rgJmqut59/RicpJ0yfhAFNFPVf92ysThJanSGW0tHVWekiW0McEhEymfUglLVocBQX7abRhngSLqyw0DZLOofTle3jHus6X+W3bYyEoPzd2WyYUmkYOujql+KyMU4J4QqQDRwCKf7pQbwZ7rX1MAZtATnxFsjB/urDfx9GvFuT/d8Ns6J7k2gP/+1QuoC4cBuEUmpG5LB61McwvcTxJ40j4/jnHAyii+7GO7DaR385HZ9TEh7os1iPzWBrWl+thXn/7Baujhr4gz6HktXt3Ymx1WTkz8QpN1HVZzuu9VpjsUDhGayrZO4rcSngH7utpLdH1Xh1JN1bh3FGZNLqxzOydyX+uWAo26XYU63lZGyOP9LJhuWRAoBVV3mjkk8h5NYjonIDzj/9F+nq34N/3VBfQk8KSKl052sMrMduC6Tnx3DOVGlqJ5BnfSX3n4ITHC74/riDOqn7CcOqOLj5bhrObWrJzfTU6d9TZYxqOoenD54RKQ98KWILPfhst5dOAkqRR2cLrO9ON1iKXYDFdP9buqQ+XHt5uQEUyfN4wPACaCJqu7MJj4y2Ed/nK6+LsAWoDxO4vZk9GIReQ3I7DLkraqaUbfSRiBMRM5U1b/csuZAZuMuG9yf/5RB3Q3ASBHxpOnSaoYzcO+rxsCEbGsZSyKFyIvAFhFprqq/4QxGLhaRP3EGhMNwLlk8H6ebBOAtnCtaPhKRu3D+kSu6ZWtU9dN0+1gIPO/WfRUoBpytqiuBNTj/uE+65XdlF7Cq7heRb9z4NqvqH275bhFZgpNgHsb51FkfiFTVjAY6vwAmikiJlMF1nJNyg+xiyCK2LGMQkX443V87cE6oXv77hJ6Vd4H73UH3/cBY4H1VTUzTSkBVt4rIz8BjIvIAztVyvYH5mWz3A2CmiLyJc6JP6dpEVZNF5HXgBRG5U1X3iUgtnEHsxRlsK/17VxYnoR7E+aAwNqsDVNXbgduzqpPBa46JyFzgcREZjNP9dwXO2F5G3sS5+OFTnPd+JP+NS32Dc+n7cDeh3eqWLwVwL24ohtPS9IhICSBZVePdn9fC6RL+MSfHUFTZwHoh4d4Y9SbuYKmqrsAZzL0S51PqVpzB4fYpn/TcvvYuOF1eX+D0Sf+E001xyliHqsbgDNz2xumu+QtngBichPQbzglsCfC+j6HPdmNIfz/AjTj/6L/jnKTnkEnXm6ruxTlBXJGmeBzO1WjR7pU5uZFVDG2AlW7XyXzg/1LGObIxA+e9Wo5zY2EsMCyTuv2BdsC/OEnhzcw2qqqf4XyQWIpzgcXSdFXud8t/FJEjOK1QIWPp37s3cf5+duK8F/46uQ7FufprH06yTb0CTEQ6uO91iik49zOtA9YDi9wy3GTQB+f3Fw0MxGmhx7uvvQinZfYpTovtBM7fbIr+wBvu/4fJhscWpTKFgTh3rL8BtM3ojnVjfCH/3bF+karuC3Y8BYElEWOMMblm3VnGGGNyzZKIMcaYXLMkYowxJtcKzSW+7oBYG5wrkTKaX8kYY8ypQnGuOlyVmyvSCk0SwUkg3wY7CGOMKaA6cOr0PdkqTElkN8A777xD9eoZ3SxtjDEmvT179jBgwADIeK69bBWmJJIEUL16dSIji/wCbcYYk1O5GgawgXVjjDG5ZknEGGNMrlkSMcYYk2uWRIwxxuRaQAbWReQ5nPWT6wHnpKy+lq5OKPAS0B1naufxqjotEPEZY4zJnUC1RD7BmX55axZ1BuCs/XwmzpoXY0Sknv9DM8YYk1sBaYm4a1uQdtGdDFwLvK6qycB+EfkEZ2W+Z9NXFJEKnLr+ceG8rnfBAhg/HhJ9WeDPGGNyKOn0JvjIT/eJ1OHklso2Ml9P+i7SrNxWqCQkwI4d/z1//nn4/vvgxWOMKXSS8DCNltzAWgjzQoNcLwKar5JITrwIzEpXFklhmPbkwgth1apTy198Ec47L/DxGGMKlQ3/xDDoqfWs3HCYP6+7k5EDqsDIkbneXn5KItuAukDKGTR9yySVqkbjLHuZKpuusuDYsgU++CBnzcXVq53vdeuCx+M8rlULbroJKqTvwTPGGN/ExycxfvwKnnzyBxISkqlVqyyd+p8PLUqf1nbzUxL5ELhVROYClXHWSO4Q3JBO0z33wEcf5fx1xYvDpk0Qlp9+PcaYgmrVqp0MGjSfdeucFX+jolryzDNdKV++BDvSdp/nQqAu8X0JuBKoDnwpIgdVtYmIfAo8oqo/A28B7YC/3Jc9rqqbAxGf3xw54ny/5pqc9TleeKElEGNMntiwYR/nnTed5GQvZ5xRkddf703HjvXzbPuBujprODA8g/KeaR4nAUMCEY/fxMbC5jR57+hR5/ugQdCtW3BiMsYUaU2aRNCv39lERpbj8cc7UqpUeJ5u3z7u5qV27WDt2lPLU8Y2jDHGz44ciWPUqC+JimpFixbOshizZ19FSIh/zkOWRPLSH38430X+SxyRkU5yMcYYP1u0aCO33baQnTtj+OWX3fzwwyA8Ho/fEghYEvGPtWuhWLFgR2GMKSL27z/GXXctZvbsdQC0bVuLadMuxxOAXhBLIsYYU0B5vV7ee289w4d/zoEDxylZMoynnurE8OHtCA0NzKxWlkROx759sGvXf8+Tk4MXizGmyNm//zi33baQmJh4Onasx+uv9+aMMyoFNAZLIrm1cyfUr+9MU2KMMQHi9XrxeiEkxENERGkmTuxOUpKXQYPODUj3VXqWRHx16BC8+up/937s2OEkkJIloVGj/+p16WLjIcYYv9i06V+iohbQt+9ZDBvmXLBzyy3nBjUmSyK+euMNePDBU8svvhg++yzw8RhjiozExGQmTvyRhx/+mhMnEtmyJZrbb29NeHhosEOzJOKzY8ec75dcApde6jwOCYErrwxaSMaYwm/dur0MGjSfVauc8dcbbmjGCy9cmi8SCFgSybkLLoBRo4IdhTGmkEtISOLJJ5czduwKEhOTiYwsx5QpvejZ88xgh3YSSyLGGJMPeTwePvtsE4mJyQwZ0prx47tQrlzxYId1CksixhiTTxw7Fk9sbCKVK5ciLCyEmTOv4ODBE1x0Ud1gh5apQK2xbowxJgtLl26mWbPXGDx4AV6vF3AmT8zPCQSsJWKMMUEVHR3LvfcuYdq0XwEoU6YYhw/HUaFCiSBH5htLIsYYEyTz5v3JkCGL2L37KMWKhfLIIxdx330X5psrr3xhScQYYwLM6/UyYMBc3n13PQDnnx/J9OmX07hx1SBHlnOWRIwxJsA8Hg81a5alVKlwxo3rzB13tAnYhIl5zZKIMcYEwPbth9m5M4bzzosE4PHHO3LHHW2oX79ikCM7PQUz9RljTAGRnOzl1VdX0aTJZPr1+5AjR+IAKFUqvMAnELCWSOZOnIDvvoOkJOf5X38FNx5jTIGzceNBbr11AcuXbwWgS5cGxMcnBTmqvGVJJDNDhjiTLqYXZm+ZMSZriYnJPP/8Dzz66DfExiZSrVppXnmlJ1dddXawQ8tzdkbMzM6dzveWLaFKFedxmTIwYEDwYjLGFAj9+n3IJ5/8CcCNNzbnhRcupVKlkkGOyj8siWRn/Hjo2jXYURhjCpBbb23JL7/sZsqUXnTv3jDY4fiVJRFjjDlNP/ywnZUrd3LXXecB0LPnmajeSYkShf8UW/iP0Bhj/OTo0XgeemgpL720Eo/HQ/v2dWjduiZAkUggYEnEGGNy5Ysv/iYqaiFbtkQTGurhvvsupGnTiGCHFXCWRIwxJgcOHTrBPfcsYcaMNQC0aFGd6dMvp2XLGkGOLDgsiRhjTA7ce+8XzJixhuLFQ3n00Yu5554LCtSEiXnNkogxxuTAE090ZM+eozz3XDfOOqtKsMMJOpv2xBhjMuH1ennzzd/o3v1tEhOTAahRoywLF/a3BOKylogxxmRg69ZobrttIYsX/w3AnDm/c911TYMcVf5jScQYY9JImTBx1KivOHo0nooVS/DCC5dy7bVNgh1avhSwJCIijYA3gMrAQeBGVf0rXZ0IYCZQGwgHvgaGq2pioOI0xhRdqgcYPHgBK1ZsA+Dqq8/m5Zd7UL16mSBHln8FckzkNeAVVW0EvAJMyaDOA8AfqtoMaAa0Aq4MXIjGmKLsiy/+YcWKbVSvXoaPPrqGDz/sZwkkGwFpibgtjJZAyiRU7wKTRKSqqu5PU9ULlBWREKA4UAzYGYgYjTFF0+HDsZQvXwKAoUPbEBMTx+23t6ZixcI5YWJeC1R3Vm1gp6omAahqkojscsvTJpEngI+A3UBpYJKqfpd+YyJSAaiQrjjSH4EbYwqn2NhEHn98Ga+99jNr1txOnTrlCQnxMHp0h2CHVqDkt0t8+wFrgRpALeAiEbk6g3p3AZvTfX0bqCCNMQXbd99to0WL1xg3bgXR0bEsWfJ3sEMqsAKVRLYDtUQkFMD9XtMtT2sY8I6qJqvqYWAe0DGD7b0I1E/3ZR8fjDFZiomJY9iwT+nQYSaqB2ncuArffTeQwYNbBju0Aisg3Vmquk9E1gDXA2+7339NNx4CTouiO/CTiBQDugBzM9heNBCdtkxE/BG6MaaQWLFiGwMGzGXbtsOEhYUwatSFPPTQRRQvbnc6nI5Avnu3A2+IyCPAIeBGABH5FHhEVX/G6aZ6TUTWAaE4l/i+HsAYjTGFVJkyxdi1K4ZWrWowffrlNG9ePdghFQoBSyKq+ifQLoPynmke/81/V3AZY8xp+f777VxwQW3AmW136dIbOf/82oSF5bfh4ILL53dSRLqKyHQRWeA+by0infwXmjHG5M7u3TFcddUHXHjhDObO/SO1vEOHupZA8phP76aIDANeBf4CLnKLTwBP+ikuY4zJMa/Xy8yZv3L22ZOZO/cPypYtxrFj8cEOq1DztTvrLqCzqm4Rkfvdsj8BG802xuQLW7ZEExW1gC+++AeAHj0aMmVKL2rXLh/kyAo3X5NIWf67HNfrfg8HLMUbY4Ju+fKt9Oz5DseOJVC5ckkmTuxO//7n4PF4gh1aoedrElkOjAKeSlM2HOfqKWOMCapWrWoQEVGaNm1q8fLLPYiIKB3skIoMX5PIMGCBiNyKM7eVAjFAL79FZowxmUhISGLSpJ8YPLglZcsWp3TpYqxadSuVK5cKdmhFjk9JRFV3i0gboA1QF6dr6ydVTfZncMYYk97q1bsYNGg+v/22l7//PsSkSc5dApZAgsOnJCIi81T1CuAn9yulfK6q2lTtxhi/O3EigcceW8Zzz31PUpKXBg0qcuWVjYMdVpHna3dWRvNXAVySR3EYY0ymli/fyuDB8/nrr38JCfEwYsR5PP54R0qXLhbs0Iq8LJOIiDzuPiyW5nGKBsBWv0RljDGu9ev3cfHFswA4++yqzJhxOe3a2coP+UV2LZHa7veQNI/Bucx3OzDGDzEZY0yqpk0juPHG5tSvX4HRo9vbhIn5TJa/DVW9BUBEvldVmwjRGON3Bw4cZ+TIJQwb1pbWrWsCMGvWFXbPRz7l69VZrwOISFmgCuBJ87N//BOaMaYo8Xq9fPjh79x556fs33+cjRsP8v33A/F4PJZA8jFfr85qDMwGmuN0ZXn47871UP+EZowpKnbtimHo0EXMm6cAXHJJPV5/vbcljwLA187FV3HuTu+Is3BUPWAc8L1/wjLGFAVer5cZM35l5MglHD4cR7lyxXn22a4MHtySkBBLIAWBr3MiNwfud1cU9LhL194LPOG3yIwxhd6+fccYMcJJIL16NWLDhqFERbWyBFKA+NoSicWZcDEBOCAidXBWJ6zsr8CMMYVTUpIz0UVoaAjVqpVh8uSehIR4uO66ptZ9VQD52hL5FrjGfTwH+AxYBiz1R1DGmMJpw4Z9XHjhDF56aWVq2YABzbj+eptxt6Dy9eqsa9I8fQDYAJQB3vBHUMaYwiU+Ponx41fw5JPLSUhI5tChWO68sy3h4XZdTkGX47t23EkX3xKRYsCtwCt5HpUxptBYtWongwbNZ926fQDcdlsrnn66iyWQQiLbJCIinYEWwCZVnSciYcBQ4H7gXyyJGGMyEBeXyEMPLeX5538kOdnLGWdUZNq0y7nkknrBDs3koezmzrofeBin+6qJiEzGmXQxDohS1UV+j9AYUyCFhYWwYoWzIOo995zPY491pFSp8CBHZfJadi2R24CLVXW1iJwHfAeMVNUX/R+aMaagOXw4lri4JCIiShMaGsLMmVcQExNHmza1gh2a8ZPsrs6qoqqrAVT1R5wWyES/R2WMKXAWLtxIkyaTGTRoPl6vM6HFWWdVsQRSyPkyJuLBmebEg3O/CCKSmnxsdUNjirb9+4/xf//3Oe++ux6AyMhyHDkSR/nyJYIcmQmE7JJIGSAxzXNPmucp82fZJRbGFEFer5f33lvP8OGfc+DAcUqVCueppzoxbFhbQkN9vQXNFHTZJZH6AYnCGFOgJCd7ufrqD/j44z8B6Ny5PlOn9qZBg4pBjswEWnbridjKhcaYU4SEeBCpTPnyxZkwoRsDB55rd5wXUbZEmDHGJ5s2/cuuXTFcdFFdAB555GKGDWtHzZplgxyZCSbruDTGZCkxMZkJE76nWbNXue66OURHxwJQsmS4JRBjLRFjTObWrdvLoEHzWbVqFwBXX90g9fJdYyCHSUREagO13HtGjDGFVFxcImPHfsvYsStITEymdu1yTJnSix49zgx2aCaf8XV53DrAuzhzaHmBMiJyNdBdVQf7uI1GOLP+VgYOAjeq6l8Z1LsGZ6qVlEuIu6jqXl/2YYzJG1dd9QGLFjn/nkOHtmbcuC6UK1c8yFGZ/MjXMZEpwCKgLM7CVABfAF1zsK/XgFdUtRHOpI1T0lcQkdbAGKCrqjYF2gOHc7APY0weGD68HY0aVWbZspt55ZXLLIGYTPmaRNoC4927070A7hK55X15sYhEAC1xWjO431uKSNV0Ve8GnlPVPSn7UNVYH2M0xuTS0qWbGT9+Rerzbt3OYP36IalXYhmTGV/HRPYCDYGNKQUicjawzcfX1wZ2qmoSgKomicgut3x/mnpnA5tFZDnO3fJzgadU9aSRPBGpAFRIt49IH2Mxxriio2O5994lTJv2Kx6Pc9NgylxXtt6H8YWvSeQ5YKGIjAPCROR6nBUOx+dxPKFAM5xusmLA5ziJ6s109e4CHs3jfRtTpMyb9ydDhixi9+6jFCsWyiOPXESLFtWDHZYpYHzqzlLVGcC9QD9gO3Aj8LCqvuPjfrYDtUQkFMD9XtMtT2sbMEdV41Q1BpiH05WW3os4U7Kk/ergYyzGFGn79h3juuvm0KfP++zefZTzz49kzZrbePDBi6z1YXLM16uzQlV1Hs5JPcdUdZ+IrAGuB952v/+qqvvTVZ0N9BSRt9zYOgNzMtheNBCdLsbchGZMkfPAA1/x/vsbKF06nHHjOjN0aBubMNHkmq/dWXtE5EPgHVX9Lpf7uh14Q0QeAQ7htGYQkU+BR1T1Z+A9oDXwO5AMLAam53J/xhiX1+tNndvqqac6ceRIHM8805V69dIPLRqTM74mkW44rYd3RSQJ52Q/W1XX+bojVf0TaJdBec80j5OBEe6XMeY0JSd7mTLlZ957bwNffvk/wsNDqVatDB980C/YoZlCwtcxkV9V9T5VrQPcDFQElorIWn8GZ4zJvY0bD3LJJbMYOvRTli/fyrx5GuyQTCGUm47QP4E/cAbB6+VpNMaY05aYmMwzz3xH8+av8e2326hWrTRz5vTj6qvPDnZophDydWC9AnAV0B84D1gCPA3M919oxpic+u23PQwcOJ9fftkNwE03Nef55y+lUqWSQY7MFFa+jonsAr7HuXrqKvfqKGNMPvPDDzv45Zfd1KlTnqlTe3HppQ2DHZIp5HxNImeo6m6/RmKMyZUDB45TpUopAKKiWhEbm8igQedStqzNd2X8L9MkIiIXqepy92ljEWmcUT1VXeqXyIwxWTp6NJ4HH/yKmTPXsHbtEOrVq0BIiIe77jov2KGZIiSrlshkoKn7OLN7NbxAgzyNyBiTrS+++JuoqIVs2RJNaKiH5cu32j0fJigyTSLuVOwpj+sHJhxjTFYOHTrByJFLmDlzDQDnnludGTOusDmvTND4dImviGQ43YmIzM3bcIwxmVm6dDNnnz2ZmTPXULx4KOPGdWblysGWQExQ+Tqw3jGT8kvyKA5jTDYiIkpz8OBx2revw7RpvRGpEuyQjMk6iYjI4+7DYmkep2gAbPVLVMYYvF4vX321mc6d6+PxeGjaNILvvhtIq1Y1CQnxBDs8Y4Dsu7Nqu18haR7XxlkAajvO1PDGmDy2dWs0PXq8Q9eub/Hhh7+nlrdpU8sSiMlXsmyJqOotACLyvaq+HpiQjCm6kpO9TJ68ilGjvuTYsQQqVSqJx3KGyceyuk+knqpucZ9+JSIZXsqrqv/4IzBjihrVAwwaNJ/vvnPWauvX72xefrkH1aqVCXJkxmQuq5bIOqCs+3gTzj0h6T8TeXGWtDXGnIavv95Mjx7vEBeXRPXqZZg8uSd9+2Z4f68x+UpW94mUTfPYlj0zxo/atYukTp3ydOhQh+ee60bFijZhoikYfL3E9yRu11Zymu4uY0wOxMYmMmHC99x5Z1vKly9BqVLhrF4dZfNdmQLH15sN3xWRC9zHtwAbgA0iMsifwRlTGK1YsY3mzV/joYe+5r77vkgttwRiCiJfu6k6Az+7j0cAXYC2wCh/BGVMYRQTE8edd35Khw4z2bjxII0bV+Hmm1sEOyxjTouv3VnFVDVeRGoBlVT1OwARqea/0IwpPBYv3kRU1EK2bTtMWFgIo0e358EHO1C8eK56lI3JN3z9C14jIqOBusAiADehHPFXYMYUFmvX7qV793cAaNWqBtOnX07z5jbflSkcfE0ig4AngATgPrfsfOAdfwRlTGHSrFk1oqJa0rBhJe6++3zCwuxiR1N4+JREVPVvnPXV05bNAeb4IyhjCrLdu2MYNuwz7rnnAs47LxKAKVN6BzkqY/zD5w5Z96qs/wG1gJ3AW6o601+BGVPQeL1eZs1aw4gRS4iOjmXXrhi++24gHpu3xBRiPiUREXkQuBGYgDNzb13gPhGpqapP+TE+YwqEzZsPERW1kC+/dGYB6tnzTF577TJLIKbQ87UlMhi4RFVTp34XkcXAcsCSiCmykpKSmTTpJx54YCnHjydQuXJJJk7sTv/+51gCMUWCr0mkNLA/XdlBwOZmMEXawYMnGDNmGcePJ3DddU2ZOLE7ERGlgx2WMQHjaxL5HHhHREYB23C6s54CFvsrMGPyq4SEJDweD2FhITq7mfcAAB92SURBVERElGbq1F4ULx7G5ZdLsEMzJuB8vdbwTiAGWAscBdYAx4BhforLmHxp9epdtG79Os8//0NqWb9+TSyBmCIr25aIiJQHzgDuAG4GqgAHVDXZv6EZk3+cOJHAmDHf8NxzP5Cc7CUpKZkRI+yeD2OyW2P9MuADnLGPGKCPqn4diMCMyS+WL9/K4MHz+euvfwkJ8TBixHk88UQnSyDGkH1L5AngfmAGcCvOOMgF/g7KmPwgNjaRESMW8+qrztyjTZpUZfr0y2nXLjLIkRmTf2T3UaqBqk5S1ePAK0DDAMRkTL5QrFgo69btIzw8hDFjLuaXX26zBGJMOtm1RFKTjKomikiupxwVkUbAG0BlnMuDb1TVvzKpK8CvwGRVvSe3+zQmpw4cOE58fBI1a5YlJMTDzJlXEBubSNOmEcEOzZh8KbukUEpElqd5Xjbdc1T1Ih/39Rrwiqq+LSI3AFOATukriUio+7NPfNyuMafN6/XywQcbGDbsM1q2rMFnnw3A4/HQsGGlYIdmTL6WXRJJv3Lh9NzsREQigJZAV7foXWCSiFRV1fQ3MY4CFgJl3K+MtlcBqJCu2PoZTK7s2hXDkCGLmD9fAYiLSyImJp5y5WylQWOyk2USUdU38mg/tYGdqprkbjdJRHa55alJRESaA5cCHYGHs9jeXcCjeRSbKaK8Xi/Tp//KPfcs4fDhOMqVK85zz3Vl8OCWNmWJMT7KN8uqiUg4MBW4xU0yWVV/EZiVriwS+NY/0ZnCJjnZy2WXzebzzzcB0Lt3I1599TJq1SoX5MiMKVgClUS2A7VEJNRNEKFATbc8RQ2cmxo/dRNIBcAjIuVUNSrtxlQ1GohOW5ZN0jHmJCEhHtq0qcnPP+/i5Zd7cO21Taz1YUwuBCSJqOo+EVkDXA+87X7/Ne14iKpuw7kbHgARGQOUCcrVWb//7nwBhIYGfPfGP9av38fevUfp3LkBAA8+2IFhw9pStapNmGhMbgXyltvbgWEishFnzq3bAUTkUxFpHcA4svbBB9C2LezaBc2awYUXBjsic5ri45N47LFvaNlyCv37z+Xff08AULx4mCUQY06Tr4tSFQcewWlBVFbV8iLSDWikqpN82Yaq/gm0y6C8Zyb1x/iy3TyTkAD33w8vvOA8798fpk6F4naFTkG2atVOBg6cz/r1+wC48sqzbLoSY/KQr/9NLwBNgQGA1y3bAAzxR1BBccMNTgIJC4OXX4a334bS9im1oDp+PIF77lnCeedNZ/36fTRsWIlvvrmJV1/tZZfuGpOHfB0T6Qs0VNVjIpIMoKo7RaSW/0ILsLlzne9Ll0KHDsGNxZy2vn3fZ8mSvwkJ8XDvvRcwZswllCoVHuywjCl0fE0i8enrikhVnOlLCpfzzgt2BCYP3H//hezZc5Tp0y+ndeuawQ7HmELL1+6sD4E3RKQ+gIjUACYB7/krMGNyYsECZcyYb1Kfd+pUn19/vc0SiDF+5msSeQDYDKzDuX/jL2AX8Jif4jLGJ/v3H6N//4+4/PL3eOyxZaxatTP1ZyEhdt+HMf7mU3eWqsYDdwN3u91YB1TVm83LjPEbr9fLu++uZ/jwzzh48ASlSoUzdmwnWrasEezQjClSfL3Et0G6orIpd4ir6j95HZQxWdm+/TBDhixi0SJnJYHOneszdWpvGjSoGOTIjCl6fB1Y34RzaW/a/oGUlojd0m0C6rHHlrFo0V+UL1+c55+/lFtuaWFTlhgTJL52Z500diIi1XFm0bUJD01AJCd7U8c4xo3rTGJiMmPHdqZmzbJBjsyYoi1Xt+6q6h6c6djH5W04xpwsMTGZ5577ngsumE58fBIAVauWZtasPpZAjMkHTmcCRgFK5VUgxqS3du1eBg2az88/7wJg0aKN9O3bOMhRGWPS8nVg/Vv+GwMBJ3k0AR73R1CmaIuLS+Spp75l3LgVJCYmU7t2OaZM6UWPHmcGOzRjTDq+tkSmpXt+DPhNVf/K43hMEbdy5Q4GDpzP7787qwTccUcbxo3rTNmyNt+VMflRtknEXUCqExClqnH+D8kUZevW7eP33/fTqFFlpk3rTYcOdYMdkjEmC9kmEXclwm5AcgDiMUXQzp1HUpelHTToXJKTvfzvf80oWdImTDQmv8vJVPCPueugG5MnoqNjGTRoHo0aTWLTpn8B8Hg8REW1sgRiTAGRZUtERK5X1XdxViKsDowQkf2kGWRX1Tr+DdEURvPm/cmQIYvYvfsoxYqFsmrVTho2rBTssIwxOZRdd9YU4F3ghgDEYoqAvXuPMnz453zwwQYALrigNtOnX85ZZ1UJcmTGmNzILol4AFR1WQBiMYXc4sWbUtc4L106nHHjOnPHHW1ttl1jCrDskkioiHTk5DmzTqKqS/M2JFNY1alTnqNH4+natQFTp/amXr0KwQ7JGHOasksixYHpZJ5EvED6GX6NAZz5rhYu3Ejv3o3weDw0blyVn3++laZNI2zCRGMKieySyDFVtSRhcmzjxoMMHjyfb7/dxttv92XAgGYAnHNOtSBHZozJS6czd5Yxp0hMTGbChO959NFviItLolq10pQrZ3ebG1NY+TSwbowvfvttDwMHzueXX3YDcPPNLZgwoRuVKpUMcmTGGH/JMomoqs21bXzyxRd/07PnbBITk6lbtzxTp/amW7czgh2WMcbPrDvL5IkOHepy5pmV6NKlAWPHdqZMmWLBDskYEwCWREyuHD0az9ix33LvvRdQsWJJSpQIY/XqKJuuxJgixpKIybElS/4mKmoBW7ceZv/+Y7z++uUAlkCMKYIsiRif/fvvCUaOXMKsWWsAOPfc6gwd2ibIURljgsmSiPHJRx/9zh13fMrevccoXjyUMWMuYeTI8wkPDw12aMaYILIkYrK1Zs0err76QwDat6/DtGm9EbEJE40xlkSMD1q0qM7w4W0RqcLtt7e2CRONMakClkREpBHwBlAZOAjcmH6NdhF5GLgOSAISgAdUdXGgYjSOLVuiGTJkEQ891IELL3SWi5k4sUeQozLG5Ee+rmyYF14DXlHVRsArOGuVpPcT0EZVmwEDgfdFxG53DpDkZC8vv7ySpk0n8/nnm7jvvi+DHZIxJp8LSBIRkQigJc4CV7jfW4pI1bT1VHWxqh53n67FmXalciBiLOr++GM/HTrMZPjwzzl2LIF+/c5m7txrgh2WMSafC1R3Vm1gp6omAahqkojscsv3Z/KaG4G/VXVH+h+ISAUg/WIUkXkYb5GRkJDEs89+z2OPLSM+Ponq1csweXJP+vZtHOzQjDEFQL4cWBeRi4EngK6ZVLkLeDRwERVe0dGxTJjwA/HxSQwadC7PPtuVihWtB9EY45tAJZHtQC0RCXVbIaFATbf8JCJyPvA2cIWqaibbexGYla4sEvg270IuvE6cSCAsLITw8FCqVi3NjBmXU7p0Mbp0saVjjDE5E5Akoqr7RGQNcD1Ogrge+FVVT+rKEpE2wPvA1ar6Sxbbiwai0702z+MujL79diuDBy/gppua88ADHQC44oqzghyVKeySk5M5cOAA0dHRJCUlBTucIqlEiRJERkYSHp630xMFsjvrduANEXkEOIQz5oGIfAo8oqo/A5OBksCUNEnhf6q6LoBxFkoxMXGMGvUlkyf/DMCcOb9z330XEhYWyAv0TFG1Y8cOPB4P9erVIzw83JZHDjCv18vBgwfZsWMH9evXz9NtByyJqOqfQLsMynumeWwTMfnB559v4rbbFrJt22HCwkIYPbo9Dz7YwRKICZhjx44hIoSE2N9cMHg8HipXrsz+/Zldx5R7+XJg3eSN48cTGDJkEW+++RsArVrVYMaMK2jWzNY5N4FnCSS4/NX6syRSiJUoEca2bYcpUSKMxx+/hLvvPt9aH8a4OnXqRLFixShWrBgJCQkMHDiQfv36pf5848aNPP3002zbto3k5GSaNGnC6NGjqVGjRmqd+fPnM2PGDGJjYylVqhR169bl3nvvpWbNmsE4pKCwJFLI7N4dQ2JiMrVrlyckxMOMGZeTmJjMmWfaPZvGpPfSSy/RqFEjNm7cyJVXXslFF11EtWrVOHz4MLfccgsPPfQQPXo4U/7MmjWLgQMHMn/+fMLDw/nwww+ZOXMmkydPpl69egCsXLmSAwcO5HkSSUxMJCwsf56u82dUJse8Xi8zZ65hxIjFtG5dky+++B8ej4f69SsGOzRj8r1GjRpRrlw59u7dS7Vq1Xjrrbdo27ZtagIBuPnmm1m0aBGLFi2iT58+TJo0iSeffDI1gQC0a3fKsC8AMTExjB07lvXr1+PxeGjdujWPPPIIo0aNomnTptxwww0AJz0fNWoUoaGhbN68mWPHjnHppZcSHR3NAw88AMChQ4fo3r07X3/9NWFhYbzwwgusWrWK+Ph4RIQxY8ZQunRp/71pLuvbKAQ2bz5Et25vM2jQfA4fjqNYsVCOHo0PdljGZO6yy8Dj8c/XZZflOJzVq1dTsWJFzjrLudx948aNNG/e/JR6zZs3R1U5ePAge/bsybBORsaOHUupUqWYN28e8+fP58477/TpdX/88QfTpk1j3rx59OnTh08//ZTExEQAFi5cSKdOnShVqhTTpk2jbNmyzJkzh/nz5xMREcHUqVN9PPrTYy2RAiwpKZlJk37igQeWcvx4ApUrl2TixO7073+OXUJpjA+GDx+O1+tl27ZtTJw4kWLFigFOyz4vff3118ydOzf14oJKlSr59Lru3btTqlQpAGrWrEnDhg1ZtmwZnTt35uOPP2b06NEALF26lKNHj7J4sTPpeXx8fGpC9DdLIgVUcrKXzp3fZNmyrQBcd11TJk7sTkSE/5uvxpy2RYuCHQHw35jIZ599xujRo2nZsiVVqlRBRPjtt99Oqb927Vr69+9P5cqVqVatGmvXrqV9+/a53n9oaCjJycmpz+Pi4k76eUoCSdG3b18++eQTIiMjiYmJoXXr1oCT9B599FHOP//8XMeSW9adVUCFhHjo1Kk+NWuWZd6863j33assgRiTSz169ODCCy9kyhRnhYobbriBlStX8tlnn6XWmTVrFkeOHOEyt7ts6NChjB8/nm3btqXWWbVqFWvXrj1l+x07dmT69OmpLZx///0XgLp167JunXMv9b59+1i5cmWWcXbr1o1Vq1Yxc+ZM+vbtm9rj0KlTJ2bNmkVsbCwAR48e5e+//87Ve5FT1hIpQH7+eRf79x+jR48zARg1qj3Dh7ejQoUSQY7MmIJv5MiRXHnlldx6661EREQwY8YMnnnmGSZMmIDX66Vx48ZMnz49ddqQ6667jhIlSjB8+HBiY2MJCQnhrLPO4t577z1l26NHj2bs2LH06tWL0NBQ2rZty0MPPUS/fv0YPnw4PXv2pF69ejRr1izLGEuWLEnnzp2ZO3cuX331VWp5VFQUkyZN4uqrr8bj8eDxeLjzzjs544wz8vZNyoAnr/v+gkVE6gGbv/rqKyIjczErfHg4JCZCfLzzOB85cSKBRx/9hgkTfqBSpZL88ccdVKlSKvsXGpNP/PHHHzRubMsLBFtGv4cdO3bQuXNngPqquiWn27SWSD63bNkWBg9ewKZN/xIS4uHGG5tRsqT92owx+YOdjfKpI0fiuP/+L3jttdUANGlSlenTL6ddO1t7yxiTf1gSyaf69n2fpUs3Ex4ewoMPdmD06A4UKxYa7LCMMeYklkTyqUcfvZjjxxN4/fXeNG0aEexwjDEmQ3aJbz7g9Xp57731jB79ZWrZRRfV5fvvB1oCMcbka9YSCbKdO48wZMgiFizYCMCVVzamTZtagP+mbjbGmLxiSSRIvF4v06b9wj33fMGRI3GUK1ec557rSuvWRWcKaWNMwWfdWUHw99//0rnzm0RFLeTIkTh6927E778P5dZbW1nrw5gA6dSpE+3btz9pzfe5c+ciIrz99ttBjKxgsSQSBOPHr+Drr7dQtWop3nvvKubNu45atcoFOyxjipyIiAhWrFiR+vzjjz+mSZMmQYyo4LHurABJTExOXVXw6ae7Eh4eyuOPd7Q7z40Jor59+zJ37lwuvvhitm/fzvHjx2nUqBHgzISb2RodCxYs4M033yQhIQGA+++/P3Xyw06dOnHFFVfw/fffs3//fgYOHJi6XkhhZEnEz+Ljkxg79lvmzVN+/HEQxYuHUalSSSZPzvmaB8YUJh7PY5n+bMqUXkRFtQJg6tTV3Hbbwkzrer2Ppj5u1Woqq1dH+RxD27ZtmT17NocPH+bjjz+mT58+bNiwAeCkNToAnn32WaZOncrdd99N+/bt6dWrFx6Ph3/++Yebb76Z5cuXp243NjaW999/nx07dtC7d2/69u0bkAWigsGSiB/99NNOBg6cx4YN+wFYsuRveveWIEdljEnh8Xjo0aNH6oqF7733XmoSyWqNju3btzNy5Ej27t1LWFgYBw4cYP/+/VStWhWAnj17AhAZGUm5cuXYs2dPQCZDDAZLIn5w/HgCDz+8lBdfXElyspeGDSsxbVpvLr64XrBDMybfSNuCyEpUVKvUVkl2ctIKSdG3b1/69etHmzZtqFjxv+Wks1qjY8SIEYwaNYouXbqQnJxM8+bNT1oLpHjx4qmPQ0NDTxq8L2wsieSxZcu2MHDgfP755xAhIR7uvfcCxoy5hFKl8tfMwMYYR+3atbn77rtPmYY9ZY2Oc889lxIlSnD06FH27t3LGWecQUxMTOps4R999BHx8UV3OWpLInls69bD/PPPIc45J4IZM66w+z6MKQCuvfbaU8qyWqNj9OjRDB06lPLly9OhQwcqVKgQhKjzB1tPJMVprCfyzz+HaNDAaQZ7vV7eeWcd11zTxCZMNMZl64nkD/5YT8TuEzkN+/cf4/rrP+Lss19B9QDgDNTdcEMzSyDGmCLBkkgueL1eZs9eR+PGr/Dee+sJDQ1h3bp9wQ7LGGMCzsZEcmj79sMMGbKIRYv+AqBz5/pMndo7tTvLGGOKEksiObBggTJgwFxiYuIpX744zz9/Kbfc0sLmuzLGB8nJyYSEWOdHsPhr/NuSSA6IVCEhIZk+fc7ilVd6UrNm2WCHZEyBULp0aXbu3Em1atUIDw+3D14B5vV6OXjwICVKlMjzbVsSyUJiYjJz5vzOtdc2wePx0KhRZdauvZ2GDSvZP4ExORAZGcmBAwfYunUriYmJwQ6nSCpRokTurlzNhiWRTKxdu5dBg+bz88+7iItL5KabWgBw5pmVgxyZMQVPSEgIERERRETYSp2FTcCSiIg0At4AKgMHgRtV9a90dUKBl4DugBcYr6rTAhUjQFxcIk89sYJx41aQmJhM7drlqFHDuq2MMSYjgRzleg14RVUbAa8AUzKoMwBoCJwJnA+McW8iDIgfiOTcNtN54onlJCYmc8cdbdiwYSjduhXOidOMMeZ0BaQlIiIRQEugq1v0LjBJRKqq6v40Va8FXlfVZGC/iHwC9AOeTbe9CkD6eQbqAuzZsydXMS4LacAtYb3xbtrKmWdW4Omnu9CmTS0OH97P4cO52qQxxuR7ac6ZubpDOlDdWbWBnaqaBKCqSSKyyy1Pm0TqAFvTPN/m1knvLiDDKUAHDBiQuwjrQX0WpD4dNerN3G3HGGMKpjOBv3P6ooI6sP4iMCtdWTGgAfAXkJt5lyOBb4EOwI7TCa6AsuMvusdflI8d7PjrAMuAf3Lz4kAlke1ALREJdVshoUBNtzytbTjdUqvc5+lbJgCoajQQncF+NuY2QJHUxaJ25GYSsoLOjr/oHn9RPnaw409z/Lmazz4gA+uqug9YA1zvFl0P/JpuPATgQ+BWEQkRkapAH2BOIGI0xhiTc4G8Out2YJiIbASGuc8RkU9FpLVb5y2cJtVfwI/A46q6OYAxGmOMyYGAjYmo6p9AuwzKe6Z5nAQMCVRMxhhjTo/NhvafaOAxMh5rKQrs+Ivu8RflYwc7/tM6/kKzsqExxpjAs5aIMcaYXLMkYowxJtcK6s2GuVZQJoL0Fx+P/2HgOpybNhOAB1R1caBj9Qdfjj9NXQF+BSar6j2Bi9I/fD12EbkGeBjw4Pz9d1HVvYGM1R98/NuPAGbizJQRDnwNDFfVAj1/vYg8B1wF1APOUdX1GdTJ1XmvKLZE8v1EkH7my/H/BLRR1WbAQOB9ESkZwBj9yZfjT/mHmgJ8EsDY/C3bY3cvtx8DdFXVpkB7oLDMHufL7/4B4A/3b78Z0Aq4MnAh+s0nwEVkcPN2Grk67xWpJJJmIsh33aJ3gZbujY1ppU4E6d4QmTIRZIHm6/Gr6mJVPe4+XYvzibTAL6SSg98/wChgIacxC0J+koNjvxt4TlX3AKjqYVWNDVyk/pGD4/cCZUUkBCiOM53SzoAF6iequkJV088Qkl6uzntFKomQwUSQQMpEkGn5OhFkQePr8ad1I/C3qhaGOYV8On4RaQ5cCrwQ8Aj9x9ff/dlAAxFZLiK/iMhDIlIYlvH09fifABoBu4E9wGJV/S6QgQZRrs57RS2JmBwQkYtx/qmuz65uYSEi4cBU4PaUE04RE4rTjdMVuBjoAfwvqBEFVj+c1ncNoBZwkYhcHdyQ8reilkRSJ4KE1H7vrCaCTFEngzoFka/Hj4icD7wN9FFVDWiU/uPL8dcAzgA+FZEtOMsO3CoiUwMbap7Lyd/+HFWNU9UYYB7QNqCR+oevxz8MeMft0jmMc/wdAxpp8OTqvFekkkhRnwjS1+MXkTbA+8DVqvpLYKP0H1+OX1W3qWoVVa2nqvVwlh14XVWjAh5wHsrB3/5soJuIeNxWWWfgt8BF6h85OP7NOFcnISLFgC7AKVcyFVK5Ou8VqSTiKuoTQfpy/JOBksAUEVnjfp0TnHDznC/HX1j5cuzvAfuA33FOuhuA6UGI1R98Of67gA4isg7n+DcCrwcj2LwkIi+JyA6ctVO+FJENbvlpn/ds2hNjjDG5VhRbIsYYY/KIJRFjjDG5ZknEGGNMrlkSMcYYk2uWRIwxxuSaJRFToInINyIyONhxZEVEBojIkix+3kFECssNnaaIsUt8Tb7h3iFeDWcK+hSNVHVXFq/5Bng7L6fqd7d5HpAIxALLgTtUdXcebd8LnKmqm/Jie1nsZwzwIBCHcyy/AyNV9QcfXx+QOE3BZi0Rk9/0VtUyab4yTSB+dqeqlsGZjK8CBXcyxvfd46iCszbGh0GOxxQyRW5RKlOwiEhFnDtp2+H8vX6HMzniKbMKi0hDnLurW+AspvWVql7r/uws4GWc9SH2Aw+r6gfZ7V9V/xWRj4Ah7nYuACbiJJeNwP+p6vfuz24GHgGqAgeAh1T1Hbd8sKq2F5Hl7qZ/cz/pDwL24rSmIkXkfpy1XFIn/RORiYBHVYeLSHngeaAnkIyzgNKj2U0WqaqJIvIO8ICIVFXV/SLS1j2WxsAJ4CNghKrGZxSnqr4vIr2AJ3EWN/od53exNrv30RRe1hIx+V0IzomyLs6EcCeASZnUfQJYAlTEmd7hZQARKQ18gTMvVATOqo2TReTs7HYuIlVwVoT7VUQqAYtwVn+rjHMyXyQild19vAT0UNWywAU402acRFUvch82d1ta76er8h7QU0TKuvsPBa5xYweYhdM11RA4F+gGZDsm5M4DdSPOin6H3OIknPVDquAsQtQZGJpZnCJyLjADuM09/inAfBEpnt3+TeFlLRGT33wiIilLkX6jqn1wPiEDICJP4XTLZCQBJ9nUdFsqK9zyXsAWVZ3pPv/VbV30Ax7LZFsvuUuKHgO+AUYAlwF/qepbbp13RWQ40BunmygZaCoi29zxkxyPoajqVhH5BegLvAl0Ao6r6o8iUg2nBVJBVU8Ax0TkBSCKTFZoBK5xWw9lgWjgqpSlXlV1dZp6W0RkCs707y9msq0oYIqqrnSfvyEiD+CMHy3L6bGawsGSiMlv+qjqlylPRKQUznhEd5wWBjgrz4Vm0IVzH05r5CcROQRMUNUZOImlnYhEp6kbhtNNlpnh6QfrRaQmpy4vuhWoparHRORa4B5guoh8hzOI/acPx5zebJxZZt8E+vNfK6Quzrrfu0UkpW4IWU/X/YGq3uC2qD7C6c77xj2eRjitqdZAKZz3ZHUm20nZ/00iMixNWTGcKdVNEWVJxOR3IwEB2qnqHhFpAfyKs2TvSdwlXW8FEJH2OLOVLsc5yS5T1a6nGcsuTl5vAZwuts/d/S8GFouzHv2TOLO/dsjFfj4EJohIJE6L5Hy3fDvOlVZVUloTvlLVAyISBfwsIrPdltKrOO/l9aoaIyJ3AVktwLQdeEpVn8rh8ZhCzJKIye/K4oyDRLtjEo9mVlFE+gE/uF1Zh3DWy07GWSt9vIj8D2fMAZzB96Oq+kcOYvkUeFlE+gMf4IyVnA0sdLuazgO+dOM96u47I3uBBkCGl866g97f4IwFbU6JUVV3u/ebTBCRh9191AciVTXb7iRVVRFZjNNiuxvnvT0CHHUvPBiCc9FBZnG+DnwsIl8CP+G0Xi4BlrsLWJkiyAbWTX73Is7aJgdw1jj4PIu6bYCVInIUmI9z5dQ/7gmuG86A+i6ctbOfBnI0IKyqB3HGV0biDFDfB/RS1QM4/0sj3O3/izO2MCSTTY3BGU+IFpFrMqkzG2dBpNnpym/E6UL6HSdRzsFZjdFXzwJRIhKB0/XWH4jBSRDpB/lPilNVf8Zp6U1y970JuDkH+zaFkN1saIwxJtesJWKMMSbXLIkYY4zJNUsixhhjcs2SiDHGmFyzJGKMMSbXLIkYY4zJNUsixhhjcs2SiDHGmFyzJGKMMSbX/h8RgbWX0dFAuAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "q6B3omej42Rk"
      },
      "cell_type": "markdown",
      "source": [
        "The peak towards left-most corner means near perfect classifier, while random prediction will have the curve as the blue dotted straight line.  \n",
        "This ROC curve tells us that our model is nearly perfect classifier, with high accuracy :)"
      ]
    },
    {
      "metadata": {
        "id": "SW550-BH5nRb"
      },
      "cell_type": "markdown",
      "source": [
        "Let's calculate an ROC curve with random predictions, and plot it to see the difference."
      ]
    },
    {
      "metadata": {
        "id": "o6EvFVw655bo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b6dc4292-3ae5-4b28-a777-a6a1a0754d1c"
      },
      "cell_type": "code",
      "source": [
        "# create random predictions\n",
        "rand_proba = np.random.random(size=(y_proba.shape))\n",
        "rand_proba[:5]  # 0.5 probability of being 0 or 1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.79654299, 0.18343479, 0.779691  , 0.59685016, 0.44583275])"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "metadata": {
        "id": "ZNA8dltO6itt"
      },
      "cell_type": "code",
      "source": [
        "FPRs, TPRs, _ = get_roc_curve(y_test, rand_proba, delta)  # passing random preds"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a-MFHXue2g4C",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "outputId": "3f27f327-9385-4b16-b1ef-5e9c968398a3"
      },
      "cell_type": "code",
      "source": [
        "# Plot the ROC curve\n",
        "plt.plot(FPRs, TPRs, color='red',\n",
        "         lw=2, label='ROC curve')\n",
        "plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--', label=\"Mean\")\n",
        "plt.xlim([-0.05, 1.0])\n",
        "plt.ylim([0.0, 1.05])\n",
        "plt.xlabel('False Positive Rate')\n",
        "plt.ylabel('True Positive Rate')\n",
        "plt.title(f'Random Predictions ROC curve (threshold delta = {delta})')\n",
        "plt.legend(loc=\"lower right\")\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEcCAYAAAAGD4lRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZyN5fvA8c+ZMWPs+5A95EqUspW+lZgoolBaKN8yUhTfKIUKLWhTiZQ1rVpERItKhfgJJVvdsu8MGcYy6zm/P+5nZo4xy5kxc84s1/v1mtec85z7ec41Z84513Mvz327PB4PSimlVE4EBToApZRSBZcmEaWUUjmmSUQppVSOaRJRSimVY5pElFJK5ZgmEaWUUjmmSSQbRGS0iHwY6DjOl4h4RKSBc/sdEXkmh8c5KSL1cjc6lR0i8quIXJHJ4z+LSF8/xDFLRF7I4b4ZxigidZ33a7HcjkNEdorIDdmJtbASkfEi0j8n+/r0j8nPRGQnUBVIAk4C3wKPGGNOBjCs8yIi1wNLgNOAB9gPvGiMeTe3n8sY85CPMf0MfGiMme61b+ncjieLGEYDTwFxQCKwGXjMGLPSq0x5YBzQDSgLbANeS/vaiUhPYAhwMRADrAPGGGOW5/1fkjtEpAsQY4z5w7k/GmhgjLknoIEVQIF87URkMPAkUBKYA/Q3xsRlUDYCeAuoDawC7jPG7HIeKw68DdyO/e542RjzmvNYKPAx0AKoA7Q1xvzsdehXgd9EZIYxJj478ReWmkgX5wvtcuAKYHiA48kN+52/qSz2DTZNRC5JW8jXM7RC5FPndakM/AR8nvyA80H5AfshaQ2UA4YCL4rIEK9yQ4A3gLHYE5DawGTg1rwMPA/+Vw8BH+TWwYrgeyngRORGYBgQgX3f1gOezaBsZWAu8AxQEVgDfOpVZDRwkXOctsATInKT1+PLgXuAg2mPbYw5APwN3JLdv6FQvWmMMQdF5DtsMgFARIYBDwDhwB7gKWPMPOex+4C+wP8BkUA0MMAY843z+IXALKCZU8Z4P5+I3II9662BPZPtb4z5y3lsJ/aM4V6gPvAJMMI53jXYs4gexphjWfxNHuBLETkGXCIirZy/5zegN/C2iDwPjAHuAIoD84DBxpgzTixDsWfdHuDpNH/DLGCvMeZp5/6t2DdxPSAKeBi41vm5SkTeAGYZYx4REQ9wkTFmq4iUAyYCHbFnQdOAscYYtw+v833ASKAKcAR42hjzURavS6KIfASMEJEqxpgo57WuDbQxxpxyin4rIoOAGSIyHXABzwH3G2Pmeh3yK+fnHCJSAngBe4ZXHtgAtAeuxNbOanqV3Qn0Ncb84JzdNgFisR/OV0VkOFDDGPOvU/4K4HvgAmNMgoj0wSa+atj/cb/kM800MYUC7YAHnfs3Yd9fLhHpCmwzxjR1itcRkV+By4CVQE9jzBERqQvswP5vRgE7gesyikFEXMBrQC8gDNgF3G2M2eg8TwURWQRch60l9jTGbHPiuxqYADQEtgD/M8asSOfvCgZeAu4DTgDj0/ufeJW/ApiB/fL8Gvse9368M/Z/V9eJ6SFjzPo0ZdJ97UTkfuAJoCb2s/CSMWZKZvHkwH+BGcaYTU4szwMfYRNLWt2BTcaYz52yo4EjInKxMeZv51j3Od8px0RkGvZ1/NapXbzh7JeUQSw/Azdja0M+Kyw1EQBEpCb2S2yr1+Zt2C/Actgvxw9F5AKvx6/EJofKwMvYLxuX89jHwFrnseex/6Tk52oIzAYexX75fQ185Xy4k92G/bJpCHQBvsG+WatgX/tBPvxNQSLSjdQvr+SYt2PPoscALzrPcTnQAJvURjr73wQ87sRxEZBhG7CToN7HfoGUx34Z7DTGPAUswzYTljbGPJLO7hOxr3E9oA02wd3v9Xi6r7OIlALeBDoaY8oAV2MTclavS6jzHEeB5ETcHvjGK4Ek+wL7pdfa+QnDJlpfvQo0d2KriP1icfu4763YD2V54BXsl/htXo/3BOY4CeRW7PujO/Y9sgz7HkvPRYDbGLMXwBjzLbZm9anzP2rqVbYn9n8RDoRi3w/e2gCNgBuziKED9j3REPu/vgP7+ie7C/sZq4D9DI4BEJGKwCLs/7kSNhEtEpFK6fxdDwCdsS0KLbCJO13Oe+BLbG2sIrZWepvX41cAM7GJthIwBVjgNPukyOS1O+zEUhb7+r0uIs0yiOUaEYnO5OeaDP6MxsCfXvf/BKpm8NqcVdZ5n28DGotIBeCCdI7VOIPnTc9fQNMsS6VRWGoiXzpnxaWxfQmjkh9IztqOT50zwVbAfGfbLmPMNAAReQ/brFHVeYO2BG5w2ieXioj3meqdwCJjzPfOvq8C/8N+0fzslJlojDnkPL4MOOzVfj0PW4XNSHURicZ+We0G7jXGGBFpjW3qmugcJwnoB1zmdXY7FpsAh2M/6O8mny06Zy93Z/CckcDM5L8J2JdJfCmcs8e7gMuNMTFAjIiMx9YMZjjF0n2dsf0RbqCJiOx2qtUHMnm6O5yzyzLYGs1txphE57HK2Cr+WZxayxHncYAjXvtk9bcFAX2Aq4wxya/HCucxXw6x0hjzpXP7jIh8jP1Sn+acrNyFPbMH2zw1zqs2OxZb06qTTm2kPPa188W7xpgtzjE/49wmi9HJiVdEMowBSMC+7hcDvyWX8TLPGPObs99H2GQB9uz2H2NMctPbbKd22AVbM/d2B/CGMWaPc5xxwPUZ/F1XASFOeQ8wx7vZEvu5mGKMWeXcf09ERjj7/ZLBMVMYYxZ53f1FRBZjT0h/T6fscuz/JLtKA8e97iffLsPZCTq5bFSabcedsqW97qd9zFcx5OBvKCxJpKvTfNAG++VZGfsFg4j0xjbl1HXKJrenJ0tpHzTGnHa+GJLLHEtzVrsLqOXcru7cT97XLSJ7sLWAZIe8bp9J535mHdP7vZtJ0tjjdbsKtkNurdeXmgsI9opzbZq/ISO1sDWq7KqM/TB7H3sXZ78W6b7OThPkndiz4xlOs8tjTvU8PZ8ZY+5x2oe/wNYQfnYeO4I9GzuL09Zf2XncBVQWkWI+JpLK2JrLNh/KpmdPmvtfABOd2nBDbAJd5jxWB5jgJOBkLuzrmPb/dgzfvyC828BPc+77zjvGDGMwxiwRkUnYZto6IjIXeNwYcyKL5znrs+JI+/7Aq+yeNOUyUh3Y5ySQ9MrXAf4rIgO9toU6+2VJRDpiT0gbYlsOSpLaGpBbTmJrOsmSb6d3gpC2bHL5GOex5PuxaR7zVfKJWbYUliQCgDHmF6eN/1Wgq3P2NA17xr/SGJMkIuuwH4qsHMC28ZbySiS1SW1z3Q9cmlzYOaushY9n7+fJ+0NzBJuQGnudKXs7QGriA/s3ZGQPtv8mq+dM6wj2LLUOtt05+Xl8ei2MMd8B33n1PUzDnvFlts8REekHrBGRj50azA/A2DT/M7BNHHHYPhmc213xre33CPZDWZ+zmwoATmG/WICUGlmVNGXOet2MMcecM9o7sU1In3h9Ce7BjhDLtD/IsRXbhl/D6/+e0ym5vffLNAZjzJvAmyISDnyGbfrMaoj4fux7w1tt7EjKtLLzfj0A1BARl9drWJvUhJ/8t4zJIj44ty+lODbh9wbmO82NX5LBd4eIXIttrs5IR2PMsnS2b8I2IX3m3G8KHDLGpK2FJJf1blIvhX1fbnLeVwec/ZNbEpo6+/iqEee+x7NUqJKI4w1gp4g0xX6xeXCqgE5HWRNfDuJ0JK4BnnWqwK2w1e8FTpHPgGFih9wtxTZlxeE0dfiLUwOahm2vfcQYc1hEagBNnC/nz4B3ReR9bMfpqEwONwNYLCILsSOfLgDKOLWCQ9j+jvRiSHKaScY4Nb+K2Nrfq1nFLyJVsc0LP2CT4Ul87G9wmve+w/ZRDMa2jQ8APheRAdgk1g7bFj/aGHPcec6RwFsikggsxr5PbsAOe3wizXO4RWQm8JqI3Ou8Dq2wTRpbgDARudk5zgjswIasfIwdcVfHiS/ZO8DzIrLOGLNJ7GCFDmmaZJPjiheRH7D9GR87mw8B7UUkyBjja59NWhnGICItsWfkv2MTaCy+/a++xta+emLfj7cBlwAL0yn7GTDIeQ+eIv0O5mQrsUO9B4nIZOznsxX2vQv2ZGSe8zr9hk341wNLnWZXb2lfu1Ds/zIKSHRqJR2AjaTDSRA5GfL+PjDLaf7bjx34MiuDsvOAV0TkNmwf00hgvVet/X3gaed7qyq2fymlX9JJjMlJMFREwoA4rwTcBkgZwu+rQtWxDuCM0nkfGGmM2Ywd3bES+ya5FPg1G4frie0Q/hf75fu+1/MY7HC5idiz1S7YocbZGmOdS57Enpn+n4icwH4hixPnN9jEusQpsySjgzjt2fcDr2PbU38h9QxyAnC7iBwTkTfT2X0g9kO/HTuU8GNsp2ZWgrAJZz/2dW4DZOeip1eAfiIS7vRd3YA9A12FHd3zGnZE3itef+d45zmfxn5J7AEewXbSpudxbDPGaifGl4AgJykNwH7w9mH//r0+xLwA2zF+0Bjj3VE6zzn2J87/cSN2oEhGpmD7nZIlJ5ujInJOu70vsoihLPaL+Ri22ego9vXP6phHsR3Ujzn7PAF0NsYcSaf4NOA77Bnx79ghrRkdNx47AOA+7P/lTu/yxpg12C/SSU7MW52y6TnrtXOSzCBsUjuG/S5YkMG+OeZ06r+MTXy7sa9ryomeiGwSkV5O2ShsAh7jxHQltk8t2ShsLWwX9rP7inP8lKfDnqjVwL7GZ3A+307z6iVk/BnIkEsXpVKq4HL6kB5JHrChVE44fWDbjDGTs7uvJhGllFI5Vuias5RSSvmPJhGllFI5pklEKaVUjhWaIb7O8LWW2LHjGc0No5RS6mzB2OH8q00GswdnptAkEWwCSe9iHqWUUlm7Fjs8P1sKUxI5APDRRx9RrVq1QMeilFIFwsGDB+nVqxdkPmddhgpTEkkCqFatGjVrZjTllFJKqQzkqBtAO9aVUkrlmCYRpZRSOaZJRCmlVI5pElFKKZVjfulYd1b9uw27MNSlJnVNZu8ywdgpu2/CTt/+ojEm29MSK6WU8h9/1US+xK7NnNkqZb2w64NfhF0He7SI1M370JRSSuWUX2oizvrDWa1JfScwzVkQJspZRawH6axXICLlOXctYB3Xq5RS2eHxwIgR53WI/NQnUpuzayq7OXuZTG+PAjvS/OjV6kop5YOkJDdTpqzhlNkOs2ef17EK6sWGb3DuEpI10USilFKZ2rTpMJGRC1i1ah9/92nEY+d5vPyURHZjl2pc7dxPWzNJYYyJBqK9t2XRVKaUUkVafHwSL764nBdeWEpCgpsaNcrQrvUFXot+50x+SiKfAw+IyFygEtAVOyGYUkqp87B69T4iIxewYcNhAPr1a8bLL7en3LGD7D3PY/triO+bQHegGvCDiBw1xjQWka+BkcaYNcAH2IXn/3F2e84Ys8Mf8SmlVGG1adNhrrpqBm63h/r1KzBtWhfatr3QPnjs/I/vr9FZg4BB6Wzv5HU7Cejvj3iUUqqoaNw4nB49LqFmzbI891xbSpYMydXj56fmLKWUUufpxIk4hg37gX79mnP55XZZjI8/vo2gIFeePJ8mEaWUKiQWLdrCgw8uZN++GH7//QArV0bicrkyTiAnT573c+an60SUUkrlQFTUKXr1mkvnzrPZty+GVq1qMH36Lbhc6SQPtxt+/BHuvhuaNz/v59aaiFJKFVAej4dPPtnIoEHfcuTIaUqUKMaYMe0YNOhKgoPTqSPs2AEdOsDWrfa+ywVt28LenI/R0iSilFIFVFTUaR58cCExMfG0bVuXadO6UL9+xdQCHg/8/TfExdn7CxbYBFKlCjz8MNx/PwQFQUREjmPQJKKUUgWIx+PB44GgIBfh4aWYMOEmkpI8REZecW7z1Zgx8Mwz5x7k5pth1Ch7+zxqIaBJRCmlCoytW/+lX7+v6NbtYgYOvBKA+++/IrXAihW2tpFs4UL7u1YtqOjUUIoXh969cy0mTSJKKZXPJSa6mTDh/3jmmZ84cyaRnTujeeihFoSEBJ9dsG9f+Ouvcw/w5pvQtWuexKZJRCml8rENGw4RGbmA1av3A3DPPZfx+us3nptAAE6dsr+feAIqVLC3q1SBTp3OLZtLNIkopVQ+lJCQxAsvLGXs2OUkJrqpWbMsU6Z0plOni7LeecAAqFMn74NEk4hSSuVLLpeLb77ZSmKim/79W/DiizdQtmzxQId1Dk0iSimVT5w6FU9sbCKVKpWkWLEg3n33Vo4ePcN11/mnVpETesW6UkrlA0uW7OCyy96hb9+v8Hg8gJ08MT8nENCaiFJKBVR0dCxDhy5m+vQ/AChdOpTjx+MoXz4s8x2TkqBjR1i/PnXb4cN5GGn6NIkopVSAzJ//N/37L+LAgZOEhgYzcuR1PPHEf+zIq8RE+O03iI9Pf+cDB+D778/dXqsWVK2at4F70SSilFJ+5vF46NVrLrNnbwSgdeuazJhxC40aVUktNHq0veI8K3XrwsqVqfcrVoTQ0FyNNzOaRJRSys9cLhfVq5ehZMkQxo2L4OGHW547YeLu3fa3CFSrlvHB/vvfzB/PY5pElFLKD/bsOc6+fTFcdVVNAJ57ri0PP9ySCy+skPmOI0bk6jQluU1HZymlVB5yuz28/fZqGjeeTI8en3PihJ1Rt2TJkKwTSAGgNRGllMojW7Yc5YEHvmLp0l0A3HBDPeLjkwIcVe7SJKKUUrksMdHNa6+tZNSon4mNTaRq1VK89VYnbrvtkkCHlus0iSilVC7r0eNzvvzybwB6927K66/fSMWKJQIcVd7QJKKUUrnsgQea8fvvB5gypTM33dQg0OHkKU0iSil1nlau3MOqVft49NGrAOjU6SKMeYSwsML/FVv4/0KllMojJ0/G8/TTS3jzzVW4XC6uuaY2LVpUBygSCQQ0iSilVI58//02+vVbyM6d0QQHu3jiif/QpEl4oMPyO00iSimVDceOneHxxxczc+Y6AC6/vBozZtxCs2YXBDiywNAkopRS2TB06PfMnLmO4sWDGTWqDY8/fnX6S9UWEZpElFIqG55/vi0HD57k1Vc7cPHFlXN2kPh4aNECNmzI3eACQJOIUkplwOPx8MEH6/n44w0sXNiTYsWCuOCCMixc2DN7B3K7YckSOHrU3j940LcEUqkStGyZ/cD9SJOIUkqlY9euaB58cCHffbcNgDlzNnPXXU1ydrCvv4YuXc7dfvHFsHlz5vu6XDl7Tj/RJKKUUl6SJ0wcNuxHTp6Mp0KFMF5//UbuvLOxLeDxwMiRsHGj7wfdudP+rlsXWrWyt10u6Nkz3yeJrPgtiYhIQ+A9oBJwFOhtjPknTZlw4F2gFhAC/AQMMsYk+itOpVTRZcwR+vb9iuXL7Voet99+CRMndqRalRKQkGAL7d4NL7yQsyfo2dO3haYKEH/WRN4B3jLGfCgi9wBTgHZpyowA/jLG3CwiIcByoDvwmR/jVEoVUd9/v53ly3dTrVpp3nqrE927N4J160DawIkTZxeuWhXeftv3g4eFQdu2uRtwPuCXJOLUMJoB7Z1Ns4FJIlLFGBPlVdQDlBGRIKA4EArs80eMSqmi6fjxWMqVCwNgwICWxOw+xEP1/6XCydXw/mr4+WebQFwuKOZ8Zbpc0KsXdOsWuMDzCX/VRGoB+4wxSQDGmCQR2e9s904izwNfAAeAUsAkY8yvaQ8mIuWB8mk218yLwJVShVNsbCLPPfcL77yzhnWr7qf2ul8Imj6d4d9/b/s90howACZN8n+g+Vx+61jvAawHIoAywDcicrsxZk6aco8Co/wdnFKqgDl8GIYMgX//PWvzr8dKE7mhHuZUCVx4WNysB31PLrUPFi8ON94I5cql7hAWBgMH+jHwgsNfSWQPUENEgp1aSDBQ3dnubSDQxxjjBo6LyHygLZA2ibwBzEqzrSawLNcjV0oVXAsXwkcfpdyNIZQRRPAWl+DBRSOimMF8Wp/cC5ddBn372maqihUDGHTB4pckYow5LCLrgLuBD53ff6TpDwHYAdwE/CYiocANwNx0jhcNRHtvE5G8CF0pVZAlOgM7b7yR5RH30+uV7eyOiqdYsIthParx9F3NKR7SCWrUsEmkgA+3DQR/Nmc9BLwnIiOBY0BvABH5GhhpjFmDbaZ6R0Q2AMHYIb7T/BijUio/W7oU/v47e+UBatemdPs27B+xhebNL2DGjFto2rRa3sRYxPgtiRhj/gauTGd7J6/b20gdwaWUUqkOHYLrr0+/0zsDK6jF1QBhYVx+eTWWLOlN69a1KFYsKK+iLHJ8TiIi0h64Cwg3xnQRkRZAWWPMkjyLTilVdB04YDuzkzvFT52yCaRsWbjzzsx3PR3EI7+WZ+7OEnzRMYbu/+sPwLXX1snrqIscn5KIiAwE/gdMB253Np8B3gSb6JVSKls8HohK2y3qZfZs+OKLc7e3bAlTp2ZwSA+zZq1jyJDFREfHUqZMKKfuvhfq18+loFVavtZEHgUijDE7ReRJZ9vfgPZmK6VypmtXWLAg63KdO8Pgwfa2y5XhrLY7d0bTr99XfP/9dgA6dmzAlCmdqVWrXLrlVe7wNYmUIXU4bnKDZAgQn+sRKaUKp8WLz+4UX+K0hFesCMEZLOoUFmYv8muXdoaksy1duotOnT7i1KkEKlUqwYQJN9Gz56W4dLRVnvM1iSwFhgHeM4cNwo6eUkqpzO3fby/gS8vlgq1boUKF8zp88+YXEB5eipYtazBxYkfCw0ud1/GU73xNIgOBr0TkAezcVgaIATrnWWRKqYJr71546KHUTvHTp+3v8uXh3ntTyzVvnqMEkpCQxKRJv9G3bzPKlClOqVKhrF79AJUqlcyF4FV2+JREjDEHRKQl0BKog23a+s25slwppc62aJH9Seuqq+DNN8/r0GvX7icycgF//nmIbduOMWmSvUpAE0hg+Do6a74x5lbgN+cneftcY0z3vApOKVVAuZ3zy65dYehQe9vlgiuuyPEhz5xJ4Nlnf+HVV1eQlOShXr0Kdqp2FVC+NmdlNAn+9bkUh1KqMKpWDa4+/6sAli7dRd++C/jnn38JCnIxZMhVPPdcW0qVCs2FINX5yDSJiMhzzs1Qr9vJ6gG78iQqpZRybNx4mDZtZgFwySVVmDnzFq68Uld+yC+yqonUcn4Hed0GO8x3DzA6D2JSShU0ixfbdceTl5DN7CLCbGrSJJzevZty4YXlGT78GooXz28rWBRtmf43jDH3A4jICmOMToSolErfzJmwatW52xs0yPahjhw5zWOPLWbgwFa0aFEdgFmzbtVrPvIpX0dnTQMQkTJAZcDl9dj2vAlNKVVgJHekjxsH7Z05VEuUgEa+d3x7PB4+/3wzjzzyNVFRp9my5SgrVvTB5XJpAsnHfB2d1Qj4GGiKbcpykXrlegaXmiqlCq3ffoPvvku9v3mz/V2vnr32I5v2749hwIBFzJ9vALj++rpMm9ZFk0cB4Gvj4tvYq9PbYheOqguMA1bkTVhKqXzt7rthezqNEKVLZ+swHo+HmTP/4LHHFnP8eBxlyxbnlVfa07dvM4KCNIEUBL4mkaZAe2NMgoi4jDHHRWQosBG7UqFSqrDYvh1694ZjxzIus3On/T14MJRyphipWhVuuCFbT3X48CmGDFnMiRNxdO7ckLffvpmaNcvmLG4VEL4mkVjshIsJwBERqY1dnbBSXgWmlAqQb7+FX3/NulzVqvDiixCavWs1kpJs/0lwcBBVq5Zm8uROBAW5uOuuJtp8VQD5mkSWAXcAs4A5wDdAHKALUilV2CSvHHjXXfD00xmXq1072wlk06bDREYu4M47GzN4cGsAevW6LKeRqnzA19FZd3jdHQFsAkoD7+VFUEqpfKBiRWjcOFcOFR+fxIsvLueFF5aSkODm2LFYHnmkFSEhOi6noMv2VTvOpIsfiEgo8ADwVq5HpZQqNFav3kdk5AI2bDgMwIMPNuell27QBFJIZJlERCQCuBzYaoyZLyLFgAHAk8C/aBJRqmD78ksYNiz1avPjx3PlsHFxiTz99BJee+3/cLs91K9fgenTb+H66+vmyvFV/pDV3FlPAs9gm68ai8hk7KSLcUA/Y0w6cz0rpQqUjz4CY87dfuml53XYYsWCWL7cLoj6+OOtefbZtpQsGXJex1T5T1Y1kQeBNsaYtSJyFfAr8Jgx5o28D00p5Vdvvgmd7NochIVBjRrZPsTx47HExSURHl6K4OAg3n33VmJi4mjZMvvHUgVDUBaPVzbGrAUwxvwftgYyIc+jUkr5X7VqUL++/clBAlm4cAuNG08mMnIBHmeE18UXV9YEUsj50ifiwk5z4sJeL4KIpCQfXd1QqaItKuoU//vft8yevRGAmjXLcuJEHOXKhQU4MuUPWSWR0kCi132X1/3k+bN0iIVSRZDH4+GTTzYyaNC3HDlympIlQxgzph0DB7YiODirRg5VWGSVRC70SxRKqQLF7fZw++2fMW/e3wBERFzI1KldqFevQoAjU/6W1XoiunKhUuocQUEuRCpRrlxxxo/vQJ8+V+iUJUWULhGmlPLJ1q3/sn9/DNddVweAkSPbMHDglVSvXibAkalA0oZLpVSmEhPdjB+/gssue5u77ppDdHQsACVKhGgCUVoTUarIGTwYPvgg9f6JExkW3bDhEJGRC1i9ej8At99eL2X4rlKQzSQiIrWAGs41I0qpgujDD+Ho0bO3lSoFTZqk3I2LS2Ts2GWMHbucxEQ3tWqVZcqUznTseJGfg1X5na/L49YGZmPn0PIApUXkduAmY0xfH4/REDvrbyXgKNDbGPNPOuXuwE61kjyE+AZjzCFfnkMplQ1//QWVK9vbpUrZNdEdt932GYsW2Y/ngAEtGDfuBsqWLR6IKFU+52tNZAqwCLgWmwAAvgfGZ+O53gHeMsZ8KCL3OMds511ARFoAo4F2xpiDIlIOe5W8UionPB4YPx7+/jt1W0yM/V2pUmoSSWPQoCv5559/mTatS0pHulLp8TWJtAJuNsa4RcQD4CyRW86XnUUkHGgGtHc2zQYmiUgVY0yUV9HBwKvGmIPJz+FjfEqp9GzbBkOHnrs9LAxKlky5u2TJDuiApUMAACAASURBVH77bR/Dhl0DQIcO9dm4sb9O166y5GsSOQQ0ALYkbxCRS4DdPu5fC9hnjEkCMMYkich+Z7t3ErkE2CEiS7FXy88FxhhjzurJE5HyQPk0z1HTx1iUKjxmzbId5cnTuKeVlGR/X3ABPPdc6vamTaFUKaKjYxk6dDHTp/+By2UvGkye60oTiPKFr0nkVWChiIwDionI3dgVDl/M5XiCgcuwNZZQ4Ftsono/TblHgVG5/NxKFTwLF0J0dNblunWDvmd3X86f/zf9+y/iwIGThIYGM3LkdVx+ebU8ClQVVr4ujztTRI5ip4bfA/QGnjHGfOnj8+wBaohIsFMLCQaqO9u97QbmGGPigDgRmY9tSkubRN7ArvfurSZ2LXilip7337eJIj0ul+04dxw+fIpBg77h0083AdC6dU1mzLiFRo2q+CNSVcj4Ojor2BgzH5ifkycxxhwWkXXA3cCHzu8/0vSHAHwMdBKRD5zYIoA56RwvGjjr9EtEchKaUoVDiRJQurRPRUeM+JFPP91EqVIhjBsXwYABLXXCRJVjvjZnHRSRz4GPjDG/5vC5HgLeE5GRwDFsbQYR+RoYaYxZA3wCtAA2A27gO2BGDp9PKeXweDwpc1uNGdOOEyfiePnl9tStm7ZrUans8TWJdMDWHmaLSBL2y/5jY8wGX5/IGPM3cGU62zt53XYDQ5wfpdR5crs9TJmyhk8+2cQPP9xLSEgwVauW5rPPegQ6NFVI+FSHNcb8YYx5whhTG7gPqAAsEZH1eRmcUirntmw5yvXXz2LAgK9ZunQX8+ens466UucpJw2hfwN/YTvB6+ZqNEqp85aY6Obll3+ladN3WLZsN1WrlmLOnB7cfvslgQ5NFUK+dqyXB24DegJXAYuBl4AFeReaUiq7/vzzIH36LOD33w8A8N//NuW1126kYsUSWeypVM742ieyH1iBHT11mzM6SimVz6xcuZfffz9A7drlmDq1Mzfe2CDQIalCztckUt8YcyBPI1FK5cgRSpI8A1a/fs2JjU0kMvIKypTRCRNV3sswiYjIdcaYpc7dRiLSKL1yxpgleRKZUipTJ0/G89S6C3iX/7H+cDx1scvWPvroVYEOTRUhmdVEJgPJCwxkdK2GB6iXqxEppTIXH8/3z39Mv/GGnWcqE4ybpVvidJSLCogMk4gxponX7Qv9E45SKjPHXpnIY88s5924S4AwrnAdZGanBC5//vFAh6aKKJ+G+DpzWKW3fW7uhqOUysiSJTu4ZNge3o27hOKuJMZ1LsGq/c9y+cLpUEbXOleB4WvHetsMtl+fS3EopbIQHl6Ko+4wrmEX01eMQK7SkVcq8DJNIiKSvABBqNftZPWAXXkSlVIKj8fDjz/uICLiQlwuF02ahPNr2c9pfsIQdNErgQ5PKSDr5qxazk+Q1+1a2GnX9wA6AY9SeWDXrmg6dvyI9u0/4PPPN6dsb1nsMEF4MtlTKf/KtCZijLkfQERWGGOm+SckpYqofftwDx7C5PXFGba1LqeSgqkYkoBrzAsw/agtc+JEYGNUKo3MrhOpa4zZ6dz9UUTSHcprjNmeF4EpVdSYKXOI/Lw0v1IbgB5sYmLC11Rdf+rsguXK+bx2iFJ5LbOayAYgecjHVuw1Ia40ZTzYJW2VUufhp5920HFcNHHUplrxBCY/2Zhu/2kN9D23cKNGUFyvRlf5Q2bXiZTxuq3LnimV26Ki4MsvISGBK+M91A6L49qTm3i1d10qPHt3oKNTyie+DvE9i9O05fZq7lJKZUNsbCLjO47lkbVvU444SgJrCaUM8VDhiUCHp5TPfJ0KfjYw0RizQkTux06J4haRQcYYXb5WqWxYvnw3kZEL2LKlPLtpz5QbE6BePdt2XKIEPPxwoENUyme+1kQigP86t4cANwDRwJfoGuhKWcePw6lTGT4cczKe4eNW89YsO2S3UanT3HdqHfSbAN27+ytKpXKVr0kk1BgTLyI1gIrGmF8BRKRq3oWmVAHy00/QoQMkJqb78HfUpx9d2E15ipHEcJbz1KmlFCfJz4Eqlbt8TSLrRGQ4UAdYBOAkFB20rhTA+vU2gZQsaYfgej+UUJGbjtjrcpuHRDGj3M80DfkXCIdq1eDqqwMQsFK5w9ckEgk8DyQAyb1+rYGP8iIopQqsvn1hwoSzNl0G9HvwKxo0qMjgwa0pVkwHO6rCw6ckYozZhl1f3XvbHGBOXgSlVEF24EAMAwd+w+OPX81VV9UEYMqULgGOSqm84fMQX2dU1r1ADWAf8IEx5t28CkypgsYDzNocypBLJhMdHcv+/TH8+msfXK601+gqVXj4up7IU8Aw4BNgkPP7CWe7UkXejqNJdOBe+vxQmujoWDp1uohPP71dE4gq9HytifQFrjfGpEz9LiLfAUuBMXkRmFIFQVKSm0mTfmPESzGcpj6VwtxMmH4bPXteqglEFQm+9vCVAqLSbDsKlMjdcJQqWI4ePcPo0b9wOh7uYgOb7zlOr16XaQJRRYavSeRb4COxSojIxcB7wHd5F5pS+VNCQhKJiW7ArjY49cX/MP+aPczmC8JL6lofqmjxtTnrEWASsN7ZJwH4DNs/olSRsXbtfvr0WUCvuxvzRKN/Yfp0enz9NbhtUqFeuismKFVoZZlERKQcUB94GLgPqAwcMca48zY0pfKPM2cSGD36Z159dSVut4ekffsZcnQMxXBDSIidtiQyEm68MdChKuVXWa2xfjO2xlECiAG6GmN+8kdgSuUXS5fuom/fBfzzz78EBbkYMvgqnv/zdYotcUP//jB6NISHBzpMpQIiq5rI88CTwEzgAexILJ2jQRUJsbGJDBnyHW+/vQaAxg3LM+P6E1z5ZX/YscMWuvVWTSCqSMsqidQzxkwCEJG3AL0uRBUZoaHBbNhwmJCQIJ5qfprhqx4ndIszwWLt2vDgg9CuXWCDVCrAskoiKaO3jDGJIpKjRawARKQhdkRXJezw4N7GmH8yKCvAH8BkY8zjOX1OpbLryJHTxMcnUb16GYKCXLz77q3ExibS5JqG4Em0fR8PPggRERCsK0MrlVVSKCkiS73ul0lzH2PMdT4+1zvAW8aYD0XkHmAKcM5pnIgEO4996eNxlTpvHo+Hzz7bxMCB39Cs2QV8800vXC4XDRpUPLvgjBlQvnxgglQqH8oqiUSmuZ+jBahEJBxoBrR3Ns0GJolIFWNM2osYhwELgdLOT3rHKw+k/STXzElsSu3fH0P//otYsMAAEBeXRExMPGXLFg9wZErlf5kmEWPMe7n0PLWAfcaYJOe4SSKy39mekkREpClwI9AWeCaT4z0KjMql2FQR5fF4mDHjDx5/fDHHj8dRtmxxXn21PX37NtMrzpXyUY77OHKbiIQAU4H7nSSTWfE3gFlpttUEluVNdKqwcbs93Hzzx3z77VYAunRpyNtv30yNGmUhLg5uvhn+8eqyO348QJEqlb/5K4nsAWqISLCTIIKB6s72ZBdgL2r82kkg5QGXiJQ1xvTzPpgxJhq7xnuKLJKOUmcJCnLRsmV11qzZz8SJHbnzzsaptY+//oIffzx3p3r1oHS6LaxKFVl+SSLGmMMisg64G/jQ+f2Hd3+IMWY39mp4AERkNFBaR2ep3LJx42EOHTpJRISdmuSpp65l4MBWVKlSKv0dGjWCb75JvV+tGhTLN5V3pfIFf34iHgLeE5GRwDGgN4CIfA2MNMas8WMsqgiJj09i3LhljBmzjAoVSvDX/PZUnPMBxRMTqZLeDlHOuU1oKNSp489QlSpwfEoiIlIcGImtQVQyxpQTkQ5Aw+SLEbNijPkbuDKd7Z0yKD/al+MqlZnVq/fRp88CNm48DED37hdT7NWX4YuPs95Zh/IqlSVfayKvY5fF7QUk1+83Odt9SiJK+dPp0wmMHPkTr7/+f7jdHhoERzO97M+0mfc2HDtmC/XpA5demv4BXC7o3Nl/AStVQPmaRLoBDYwxp0TEDWCM2SciNfIuNKVyrlu3T1m8eBtBQS6Ghq5mdPxiSh5LSC1QsiQMGwYXXRS4IJUqBHxNIvFpy4pIFez0JUrlO08++R8OHjzJjBm30CJiPMQngDFQtqwtUKYMlMqgQ10p5TNfk8jn2E7xwQAicgH2Wo1P8iowVQQcPQovv5wr12B8tbM4a4+EMLrFScDOp/PHVRA0fR2cOWMLVa0K5cqd93MppVL5mkRGAC8BG4CSwD/ANODZPIpLFQWffGKTyHmIoiT/oyOzsX0bN6+dTUv2A2nWfi5RAorrNCZK5TafkogxJh4YDAx2mrGOGGN0MemiZPp024eQmJh7x4yNtb8jIuD227O1q8fjYfbqWAZ9FsPRUx5KhsLYW8vQ7PqnISidKUuaNYOwsFwIWinlzdchvmkXji6TfIW4MWZ7bgel8qF582zzU24LDoZ+/eCOO3zeZc+e4/Tvv4hFi+yw3YiIC5k6tQv16lXI/fiUUpnytTlrK+ABvE/xkmsiuqhCUfLJJ7m7jnhoqB0plQ3PPvsLixb9Q7lyxXnttRu5//7LdcJEpQLE1+ass5qXRaQadhZdnfCwqCldOiAX4bndHoKcZqpx4yJITHQzdmwE1auX8XssSqlUQVkXOZcx5iB2OvZxuRuOUmdLTHTz6qsruPrqGcTHJwFQpUopZs3qqglEqXzgfObOEuxILaXyxPr1h4iMXMCaNXa01aJFW+jWrVGAo1JKefO1Y30ZqX0gYJNHY+C5vAhKFW1xcYmMGbOMceOWk5joplatskyZ0pmOHfXqcqXyG19rItPT3D8F/GmM+Se9wkrl1KpVe+nTZwGbN9uZdB9+uCXjxkVQpoxe46FUfpRlEnEWkGoH9DPGxOV9SKoo27DhMJs3R9GwYSWmT+/CtdfqVOxK5WdZJhFnJcIOgNsP8agiaN++E3ZZWiAy8grcbg/33nsZJUqEBDgypVRWfB2d9TrwrLMOulK5Ijo6lsjI+TRsOImtW/8FwOVy0a9fc00gShUQmdZERORuY8xsYCBQDRgiIlF4dbIbY2rnbYgqIHbuhFWrUu/v35+rh58//2/691/EgQMnCQ0NZvXqfTRoUDFXn0Mplfeyas6aAswG7vFDLCo/advWJpK0QkPP67CHDp1k0KBv+eyzTQBcfXUtZsy4hYsvrnxex1VKBUZWScQFYIz5xQ+xqPxi82bYt8/e7t4dijlvk+rV4brrcnzY777bSs+ec/n33zOUKhXCuHERPPxwq5Qr0ZVSBU9WSSRYRNpy9pxZZzHGLMndkJRfJCZCUlLq/dhYmDvXzta7YoXdVro0fPppahI5T7Vrl+PkyXjat6/H1KldqFtX1zBXqqDL6tuhODCDjJOIB0g7w6/K75YuhY4d4fTp9B8vUwbuvhsGDjyvBOJ2e1i4cAtdujTE5XLRqFEV1qx5gCZNwnXCRKUKiay+IU4ZYzRJFDb/9382gQQFpSYJlwuaN4e+fe207Oe5dOyWLUfp23cBy5bt5sMPu9Gr12UAXHpp1fONXimVj+ROO4UqmB577LxXFkwrMdHN+PErGDXqZ+LikqhatRRly+rV5koVVj51rCvliz//PEifPgv4/fcDANx33+WMH9+BihVLBDgypVReyTSJGGN0ru3C4uTJ1I70M2dy/fDff7+NTp0+JjHRTZ065Zg6tQsdOtTP9edRSuUv2pxVFLzwAjzzTJ4+xbXX1uGiiypyww31GDs2gtKlz+96EqVUwaBJpCj49Vf7u0QJCHGmEylV6ryWuT15Mp6xY5cxdOjVVKhQgrCwYqxd20+nK1GqiNEkUpR88YUd2nueFi/eRr9+X7Fr13Giok4xbdotAJpAlCqCNIkon/377xkee2wxs2atA+CKK6oxYEDLAEellAokTSLKJ198sZmHH/6aQ4dOUbx4MKNHX89jj7UmJCQ40KEppQJIk4jK0rp1B7n99s8BuOaa2kyf3gURnTBRKaVJRPng8surMWhQK0Qq89BDLXTCRKVUCr8lERFpCLwHVAKOAr3TrtEuIs8AdwFJQAIwwhjznb9iVNbOndH077+Ip5++lv/8xy4XM2HC+XfIK6UKH19XNswN7wBvGWMaAm9h1ypJ6zegpTHmMqAP8KmI6OXOfuJ2e5g4cRVNmkzm22+38sQTPwQ6JKVUPueXmoiIhAPNgPbOptnAJBGpYoyJSi6XptaxHjvtSiVgrz/iLDQSE89eidCHK9T/+iuKvn2/YsWKPQD06HEJEydq7UMplTl/NWfVAvYZY5IAjDFJIrLf2R6VwT69gW3GmHMSiIiUB9IuRlEzF+Mt2K67Dlau9KloQkISr7yygmef/YX4+CSqVSvN5Mmd6NatUR4HqZQqDPJlx7qItAGeJ7XmktajwCj/RZTPzZkD27en3v/tN/u7Rg073Xvy7SuvPGfX6OhYxo9fSXx8EpGRV/DKK+2pUEFbEJVSvvFXEtkD1BCRYKcWEgxUd7afRURaAx8CtxpjTAbHewOYlWZbTWBZ7oVcQPz9N/Toce720FCbWNJZE/3MmQSKFQsiJCSYKlVKMXPmLZQqFcoNN+jSMUqp7PFLEjHGHBaRdcDd2ARxN/CHd38IgIi0BD4FbjfG/J7J8aKB6DT75nrcBcLx4/Z3tWpw772p26++Ot0EsmzZLvr2/Yr//rcpI0ZcC8Ctt17sj0hVEeZ2uzly5AjR0dEkeS/LrPwmLCyMmjVrEhKSu9MT+bM56yHgPREZCRzD9nkgIl8DI40xa4DJQAlgildSuNcYs8GPcRZMtWtnusBUTEwcw4b9wOTJawCYM2czTzzxH4oV8+cAPVVU7d27F5fLRd26dQkJCdHlkf3M4/Fw9OhR9u7dy4UXXpirx/ZbEjHG/A2c0yhvjOnkdVsnYsoD3367lQcfXMju3ccpViyI4cOv4amnrtUEovzm1KlTiAhBQfqeCwSXy0WlSpWIispoHFPO5cuOdZWJmBiYMQNOnLD392Y8+vn06QT691/E++//CUDz5hcwc+atXHaZrnOu/E8TSGDlVe1Pk0hB8/77MHjwudvLnLsIZVhYMXbvPk5YWDGee+56Bg9urbUPpRzt2rUjNDSU0NBQEhIS6NOnDz28Bqls2bKFl156id27d+N2u2ncuDHDhw/nggsuSCmzYMECZs6cSWxsLCVLlqROnToMHTqU6tWrB+JPCghNIgVNTIz9ffXVEBFhbwcFpYzQOnAghsREN7VqlSMoyMXMmbeQmOjmoosqBShgpfKvN998k4YNG7Jlyxa6d+/OddddR9WqVTl+/Dj3338/Tz/9NB2dNXhmzZpFnz59WLBgASEhIXz++ee8++67TJ48mbp16wKwatUqjhw5kutJJDExkWLF8ufXdf6MSmXt2mvhuedS7no8Ht6d+QdDhnxHixbV+f77e3G5XFx4YYUABqlUwdCwYUPKli3LoUOHqFq1Kh988AGtWrVKSSAA9913H4sWLWLRokV07dqVSZMm8cILL6QkEIAr07kWCyAmJoaxY8eyceNGXC4XLVq0YOTIkQwbNowmTZpwzz33AJx1f9iwYQQHB7Njxw5OnTrFjTfeSHR0NCNGjADg2LFj3HTTTfz0008UK1aM119/ndWrVxMfH4+IMHr0aEqVKpV3L5pD2zYKgR07jtGhw4dERi7g+PE4QkODOXkyPtBhKZWxm28Glytvfm6+OdvhrF27lgoVKnDxxXa4+5YtW2jatOk55Zo2bYoxhqNHj3Lw4MF0y6Rn7NixlCxZkvnz57NgwQIeeeQRn/b766+/mD59OvPnz6dr1658/fXXJCYmArBw4ULatWtHyZIlmT59OmXKlGHOnDksWLCA8PBwpk6d6uNff360JlKAJSW5mTTpN0aMWMLp0wlUqlSCCRNuomfPS3UIpVI+GDRoEB6Ph927dzNhwgRCnWurPB5Prj7PTz/9xNy5c1MGF1SsWNGn/W666SZKliwJQPXq1WnQoAG//PILERERzJs3j+HDhwOwZMkSTp48yXff2ekH4+PjUxJiXtMkUkC5PRAR8T6//LILgLvuasKECTcRHp731VelztuiRYGOAEjtE/nmm28YPnw4zZo1o3LlyogIf/755znl169fT8+ePalUqRJVq1Zl/fr1XHPNNTl+/uDgYNxud8r9uLi4sx5PTiDJunXrxpdffknNmjWJiYmhRYsWgE16o0aNonXr1jmOJae0OauACnJBu3YXUr16GebPv4vZs2/TBKJUDnXs2JH//Oc/TJliV6i45557WLVqFd98801KmVmzZnHixAludprLBgwYwIsvvsju3btTyqxevZr169efc/y2bdsyY8aMlBrOv//+C0CdOnXYsMFeS3348GFWrVqVaZwdOnRg9erVvPvuu3Tr1i2lxaFdu3bMmjWL2NhYAE6ePMm2bdty9Fpkl9ZE8rukJFi/HhITWbM5mqjlh0ju6hs27BoGDbqS8uXDAhqiUoXBY489Rvfu3XnggQcIDw9n5syZvPzyy4wfPx6Px0OjRo2YMWNGyrQhd911F2FhYQwaNIjY2FiCgoK4+OKLGTp06DnHHj58OGPHjqVz584EBwfTqlUrnn76aXr06MGgQYPo1KkTdevW5bLLLss0xhIlShAREcHcuXP58ccfU7b369ePSZMmcfvtt+NyuXC5XDzyyCPUr18/d1+kdLhyu+0vUESkLrDjxx9/pGbNQjQr/MMPc2byVEbRlvG0piJn+Iu3qDz8fzB2bKCjU8onf/31F40a6fICgZbe/2Hv3r1E2MsFLjTG7MzuMbUmks/9sjqKvvRnK5UIwkPvqgcpcdGVcOedgQ5NKaU0ieRXJ07E8eST3/PO6sYANK5Tghmf9uTKKwtRLUspVeBpEsmnunX7lCVLdhDicvOU5xeGT3yCUE0gSql8RpNIPjVqVBtOn05gWtBCmqz4BUKGBTokpZQ6hw7xzQc8Hg+ffLKR4cN/SNl23XV1WLGiD03KnAlgZEoplTmtiQTYvn0n6N9/EV99tQWA7hUP03LnrwC4ADZvDlxwSimVBU0iAeLxeJg+/Xcef/x7TpyIo2zZ4rz6antaPB4BJ46fu4OP0yQopZQ/aXNWAGzb9i8REe/Tr99CTpyIo0vIdjYHvc0Dw2/ElZxAXn8dJk2yP/PmQUtd9FGp3NSuXTuuueaas9Z8nzt3LiLChx9+GMDIChatiQTAiy8u56efdlKlfAgToz/mjoRNuKK9ClxxBfzvf3ZGUqVUngkPD2f58uW0adMGgHnz5tG4ceMAR1WwaBLxk8REd8qqgi+91J6QkGCe61Kayp2egubN4dtvUwtXqKAJRCk/6NatG3PnzqVNmzbs2bOH06dP07BhQ8DOhJvRGh1fffUV77//PgkJCQA8+eSTKZMftmvXjltvvZUVK1YQFRVFnz59UtYLKYw0ieSx+Pgkxo5dxvz5hv/7v0iKFy9GxYolmDz5Zli50hYKCYHKlQMbqFJ+5nI9m+FjU6Z0pl+/5gBMnbqWBx9cmGFZj2dUyu3mzaeydm0/n2No1aoVH3/8McePH2fevHl07dqVTZs2AZy1RgfAK6+8wtSpUxk8eDDXXHMNnTt3xuVysX37du677z6WLl2actzY2Fg+/fRT9u7dS5cuXejWrZtfFogKBE0ieei33/bRp898Nm2KAmDx4m106SIBjkoplczlctGxY8eUFQs/+eSTlCSS2Rode/bs4bHHHuPQoUMUK1aMI0eOEBUVRZUqVQDo1KkTADVr1qRs2bIcPHjQL5MhBoImkTxw+nQCzzyzhDfeWIXb7aFBg4pMn96FNm3qBjo0pfIN7xpEZvr1a55SK8lKdmohybp160aPHj1o2bIlFSqkLied2RodQ4YMYdiwYdxwww243W6aNm161logxYsXT7kdHBx8Vud9YaNJJJf98stO+vRZwPbtxwgKcjF06NWMHn09JUuGBDo0pVQ6atWqxeDBg8+Zhj15jY4rrriCsLAwTp48yaFDh6hfvz4xMTEps4V/8cUXxMcX3eWoNYnksl27jrN9+zEuvTScmTNvpUWL6oEOSSmVhTvTmRU7szU6hg8fzoABAyhXrhzXXnst5cuXD0DU+YOuJ5ILtm8/Rr16thrs8Xj46KMN3HFHY0JDgzPfceVKuPpquOqq1E52pQohXU8kf8iL9UT0YsPzEBV1irvv/oJLLnkLY44AtqPunnsuyzqBKKVUIaDNWTng8XiYPXsjgwZ9w9GjZyhZMoQNGw4jTz0EX36ZnQPlXZBKKeUHmkSyac+e4/Tvv4hFi/4BIOLSUkx9sDr12GwTSE5GYbRvn8tRKqWUf2gSyYavvjL06jWXmJh4ypUJ4bWYOdy/4Q9cj3gVCgqCkyftBYS+Kqb/BlX4ud1ugoK0BT1Q8qr/W7+9skGkMgkJbrp2vZi37qtC9a5P2dl127ZNLdS2LZQoEbgglcqHSpUqxb59+6hatSohISG4dFofv/J4PBw9epSwsLBcP7YmkUwkJrqZM2czd97ZGJfLRcOGlVi//iEaNKiIK3k0lQg40yIopdJXs2ZNjhw5wq5du0hMTAx0OEVSWFhYnoxc1SSSgfXrDxEZuYA1a/YTF5fIf/97OQAXNagIa9fCxIkBjlCpgiMoKIjw8HDCw8MDHYrKZX5LIiLSEHgPqAQcBXobY/5JUyYYeBO4CfAALxpjpvsrRoC4uETGjFnGuHHLSUx0U6tWWS64oAwkJMCUKTB9Ovz5Z+oO7dr5MzyllMpX/FkTeQd4yxjzoYjcA0wB0n4D9wIaABdhk80fIvJDTi6AyYmVK/cQGbmAv/6y13w8/HBLxo2LoEyZ4jB5MgwcaAtWqgS9e0NkJOjaA0qpIswvSUREwoFmQPJY1tnAJBGpYoyJ8ip6JzDNGOMGokTkS6AH8Eqa45UH0s4zUAfg4MGDOYrxl5Ezuf+DE3hwcVHZOF66eC8tt67keI83OQ6wY4cdRdW9O4wbB6Ghdse9e3P0fEoplR94alpw1AAACblJREFUfWfm6Appf9VEagH7jDFJAMaYJBHZ72z3TiK1gV1e93c7ZdJ6FEh3CtBevXrlOMgL66XeHhYNRKcpUK8erFsHHTvm+DmUUiqfugjYlt2dCmrH+hvArDTbQoF6wD9ATuZdrgksA67l/9u78xi7yjKO419agYg0ghQIS6Ai9qeEVYECgaKyKBWSKlAsChKFStVWtqCpVIpAopGyW1MIoCyFskQkQBiscWhAFoECSuFXkF3KUqFIoSCbf7zvLbfDLOeezj0zc+/zSSaZe+bMPc8zc+e88573nueBdpxeRP7tm3875w6R/2bAbcATZb65qkHkWWATScPzLGQ4sHHeXu8Z0mWpv+fHXWcmANjubp4AsKhsgNKKZlHPVbUGM5hE/u2bfzvnDpF/Xf6l6tlXcvuo7ZeAB4CJedNEYEGX9RCAa4CjJA2TtD4wHoibMEIIYZCqsgbB0cAUSYuAKfkxkm6WtGPe5zLSlOox4C7gl7afrDDGEEIIDahsTcT2o8CYbraPq/v8PWByVTGFEEJYNVEN7UNLgVPofq2lHUT+7Zt/O+cOkf8q5d8ynQ1DCCFUL2YiIYQQSotBJIQQQmlD9WbD0oZKIchmKZj/dOBbpJs23wGm2e6oOtZmKJJ/3b4CFgCzbJ9QXZTNUTR3SROA6cBqpNf/3rZfrDLWZij42t8AuIRUKWN14K/AVNtDun69pDOAA4FRwDa2/9nNPqXOe+04E6kVghwN/JZUCLKr+kKQuwIzJI2qLMLmKpL/PcBOtrcFvgfMldQqnbaK5F/7g5oNXF9hbM3WZ+757fYzgH1sbw3sDql8XAso8rufBjySX/vbAl8EvlldiE1zPTCWbm7erlPqvNdWg0hdIcgr86YrgS/kGxvrrSgEmW+IrBWCHNKK5m+7w/ab+eFDpP9I16ss0CZp4PcP8DPgRlahCsJg0kDuxwJn2H4BwPZrtt+qLtLmaCD/D4ARkoYBa5LKKf27skCbxPbttrtWCOmq1HmvrQYRuikECdQKQdYrWghyqCmaf73DgX/ZboWaQoXyl7Qd8FXgrMojbJ6iv/utgC0kzZd0v6STJLVCL9ui+Z8KjAYWAy8AHbbvqDLQAVTqvNdug0hogKQ9SX9UE/vat1VIWh24ADi6dsJpM8NJl3H2AfYE9gMOG9CIqnUwafa9EbAJMFbSQQMb0uDWboPIikKQsOK6d2+FIGs262afoaho/kjaFbgcGG/blUbZPEXy3wj4DHCzpKdIbQeOknRBtaH2u0Ze+9faftv268CfgJ0rjbQ5iuY/BbgiX9J5jZT/lyuNdOCUOu+11SDS7oUgi+YvaSdgLnCQ7furjbJ5iuRv+xnbI22Psj2K1HbgQtuTKg+4HzXw2p8D7CtptTwr2wt4kCGugfyfJL07CUlrAHsDH3knU4sqdd5rq0Eka/dCkEXynwV8HJgt6YH8sc3AhNvviuTfqorkfhXwErCQdNJ9GLhoAGJthiL5HwPsIekfpPwXARcORLD9SdK5kp4j9U6ZJ+nhvH2Vz3tR9iSEEEJp7TgTCSGE0E9iEAkhhFBaDCIhhBBKi0EkhBBCaTGIhBBCKC0GkTCkSeqUdORAx9EbSd+WdGsvX99DUqvc0BnaTLzFNwwa+Q7xDUkl6GtG236+l+/pBC7vz1L9+Tl3Ad4F3gLmAz+yvbifnv8D4LO2H++P5+vlODOAnwNvk3JZCBxv+86C319JnGFoi5lIGGwOsL123UePA0iT/dj22qRifOswdIsxzs15jCT1xrhmgOMJLabtmlKFoUXSuqQ7aceQXq93kIojfqSqsKQtSXdXb09qpvUX24fkr30OOI/UH+JlYLrtq/s6vu1XJF0HTM7PsxtwDmlwWQT8xPbf8teOAH4BrA8sAU6yfUXefqTt3SXNz0/9YP5P//vAi6TZ1KaSfkrq5bKi6J+kc4DVbE+V9EngTGAc8D6pgdLJfRWLtP2upCuAaZLWt/2ypJ1zLp8HlgPXAcfZ/l93cdqeK2l/4DRSc6OFpN/FQ339HEPriplIGOyGkU6Um5MKwi0Hzu9h31OBW4F1SeUdzgOQ9Angz6S6UBuQujbOkrRVXweXNJLUEW6BpE8BN5G6v61HOpnfJGm9fIxzgf1sjwB2I5XNWIntsfnT7fJMa26XXa4CxkkakY8/HJiQYwf4PenS1JbADsC+QJ9rQrkO1OGkjn6v5s3vkfqHjCQ1IdoL+GFPcUraAbgY+EHOfzZwg6Q1+zp+aF0xEwmDzfWSaq1IO22PJ/2HDICk00mXZbrzDmmw2TjPVG7P2/cHnrJ9SX68IM8uDgZO6eG5zs0tRd8AOoHjgK8Dj9m+LO9zpaSpwAGky0TvA1tLeiavnzS8hmL7aUn3A98ALgW+Arxp+y5JG5JmIOvYXg68IeksYBI9dGgEJuTZwwhgKXBgrdWr7fvq9ntK0mxS+feze3iuScBs23fnx3+QNI20fnRbo7mG1hCDSBhsxtueV3sgaS3SesTXSDMMSJ3nhndzCedE0mzkHkmvAjNtX0waWMZIWlq378dIl8l6MrXrYr2kjfloe9GngU1svyHpEOAE4CJJd5AWsR8tkHNXc0hVZi8FDuXDWcjmpL7fiyXV9h1G7+W6r7b9nTyjuo50Oa8z5zOaNJvaEViL9DO5r4fnqR3/u5Km1G1bg1RSPbSpGETCYHc8IGCM7RckbQ8sILXsXUlu6XoUgKTdSdVK55NOsrfZ3mcVY3melfstQLrEdks+fgfQodSP/jRS9dc9ShznGmCmpE1JM5Jd8/ZnSe+0GlmbTRRle4mkScC9kubkmdLvSD/LibZfl3QM0FsDpmeB022f3mA+oYXFIBIGuxGkdZCleU3i5J52lHQwcGe+lPUqqV/2+6Re6b+SdBhpzQHS4vsy2480EMvNwHmSDgWuJq2VbAXcmC817QLMy/Euy8fuzovAFkC3b53Ni96dpLWgJ2sx2l6c7zeZKWl6PsangU1t93k5ybYldZBmbMeSfrb/BZblNx5MJr3poKc4LwT+KGkecA9p9vIlYH5uYBXaUCysh8HubFJvkyWkHge39LLvTsDdkpYBN5DeOfVEPsHtS1pQf57UO/vXQEMLwrb/Q1pfOZ60QH0isL/tJaS/pePy879CWluY3MNTzSCtJyyVNKGHfeaQGiLN6bL9cNIlpIWkgfJaUjfGon4DTJK0AenS26HA66QBousi/0px2r6XNNM7Px/7ceCIBo4dWlDcbBhCCKG0mImEEEIoLQaREEIIpcUgEkIIobQYREIIIZQWg0gIIYTSYhAJIYRQWgwiIYQQSotBJIQQQmkxiIQQQijt/2FzX83zRV1JAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "JKHS-Skq-IPp"
      },
      "cell_type": "markdown",
      "source": [
        "As we can see from the above plot, random predictions give the ROC curve nearly at the mean.  \n",
        "Contrast it with the previous curve, how our model has top left peak, due to its superior performance."
      ]
    },
    {
      "metadata": {
        "id": "wcQXLrk68FC1"
      },
      "cell_type": "markdown",
      "source": [
        "### ROC-AUC score\n",
        "Now, that we know about ROC curve, what it represents, **AUC score** is very easy to understand.  \n",
        "AUC stands for **Area under the Curve**, which is nothing but the area under the ROC curve formed by the predictions.  \n",
        "As we saw, a totally random prediction will have AUC score `0.5`, while a perfect classifier will have AUC score of `1`. "
      ]
    },
    {
      "metadata": {
        "id": "JETH5mgtIs9Y"
      },
      "cell_type": "markdown",
      "source": [
        "Lets check the AUC score of our model."
      ]
    },
    {
      "metadata": {
        "id": "IJFx9ubwddpn"
      },
      "cell_type": "markdown",
      "source": [
        "We can calculate the **ROC-AUC score** by summing up the areas under each observation of FPRs and TPRs."
      ]
    },
    {
      "metadata": {
        "id": "cyx-Y4nPd6oL"
      },
      "cell_type": "code",
      "source": [
        "def get_roc_auc_score(y_test, y_proba):\n",
        "    \n",
        "    # use the function get_roc_curve that we created.\n",
        "    FPRs, TPRs, _ = get_roc_curve(y_test, y_proba)\n",
        "    FPRs.reverse()\n",
        "    TPRs.reverse()\n",
        "    x1, y1 = FPRs[0], TPRs[0]\n",
        "    auc = 0.0\n",
        "    prev = 0.0\n",
        "    # cumulative differences in x-axis\n",
        "    diffs = [FPRs[i] - FPRs[i-1] for i in range(1, len(FPRs))]\n",
        "    for x, y in zip(diffs, TPRs[1:]):\n",
        "        auc += (x * y1)  # area of rectangle\n",
        "        auc += (x * (y - y1)/2)  # area of triangle formed (if any)\n",
        "        y1 = y\n",
        "    return auc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pqucApZPjn9H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8269b343-466f-45b6-c91c-086a62d271d5"
      },
      "cell_type": "code",
      "source": [
        "auc_score = get_roc_auc_score(y_test, y_proba)\n",
        "print(f\"Our ROC-AUC score of SVC model is {auc_score: .4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Our ROC-AUC score of SVC model is  0.9874\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "54g1PC2jK0MS"
      },
      "cell_type": "markdown",
      "source": [
        "This is a good ROC-AUC score as we expected. (Also pretty close to Scikit's implementation).  \n",
        "Lets try the ROC-AUC score of random predictions."
      ]
    },
    {
      "metadata": {
        "id": "p0970PIaLBYj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "80a6c520-7ee8-4bdf-c2d6-0e049de3691f"
      },
      "cell_type": "code",
      "source": [
        "auc_score = get_roc_auc_score(y_test, rand_proba)\n",
        "print(f\"Our ROC-AUC score of random predictions is:      {auc_score: .4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Our ROC-AUC score of random predictions is:       0.5080\n"
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "Ara7jLPbLPXQ"
      },
      "cell_type": "markdown",
      "source": [
        "As expected, it is close to `0.5`.  \n",
        "(The minor differences are due to different granularity of thresholds chosen by us, and in Scikit's implementation. )"
      ]
    }
  ]
}