unit04_logistic/demo04_breast_cancer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Breast Cancer Diagnosis via Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this demo, we will see how to visualize training data for classification, plot the logistic function and perform logistic regression.  As an example, we will use the widely-used breast cancer data set.  This data set is described here:\n",
    "\n",
    "https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin\n",
    "\n",
    "Each sample is a collection of features that were manually recorded by a physician upon inspecting a sample of cells from fine needle aspiration.  The goal is to detect if the cells are benign or malignant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and Visualizing the Data\n",
    "\n",
    "We first load the packages as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn import datasets, linear_model, preprocessing\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the data.  It is important to remove the missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>thick</th>\n",
       "      <th>size_unif</th>\n",
       "      <th>shape_unif</th>\n",
       "      <th>marg</th>\n",
       "      <th>cell_size</th>\n",
       "      <th>bare</th>\n",
       "      <th>chrom</th>\n",
       "      <th>normal</th>\n",
       "      <th>mit</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1000025</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002945</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1015425</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1016277</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1017023</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1017122</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  thick  size_unif  shape_unif  marg  cell_size  bare  chrom  \\\n",
       "0  1000025      5          1           1     1          2   1.0      3   \n",
       "1  1002945      5          4           4     5          7  10.0      3   \n",
       "2  1015425      3          1           1     1          2   2.0      3   \n",
       "3  1016277      6          8           8     1          3   4.0      3   \n",
       "4  1017023      4          1           1     3          2   1.0      3   \n",
       "5  1017122      8         10          10     8          7  10.0      9   \n",
       "\n",
       "   normal  mit  class  \n",
       "0       1    1      2  \n",
       "1       2    1      2  \n",
       "2       1    1      2  \n",
       "3       7    1      2  \n",
       "4       1    1      2  \n",
       "5       7    1      4  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = ['id','thick','size_unif','shape_unif','marg','cell_size','bare',\n",
    "         'chrom','normal','mit','class']\n",
    "df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/' +\n",
    "                 'breast-cancer-wisconsin/breast-cancer-wisconsin.data',\n",
    "                names=names,na_values='?',header=None)\n",
    "df = df.dropna()\n",
    "df.head(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After loading the data, we can create a scatter plot of the data labeling the class values with different colors.  We will pick two of the features.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x10bb5c588>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAERCAYAAACQIWsgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3X90XXWZ7/H301NiWywCtV4KLaQO\nlRYKQhsLBTpEOyT0Xi5oi85oekVBy7hEy9xxtRZWgQLTyuhgverMSlQGUOTHEFR0KKKdyaKYKKQt\nQvk1/DCUzpQhFMUfiKHpc//Y57Q52e1pTnKyv9/0fF5r7bXPd5999n6yz0me7O/e5/mauyMiItLX\nqNABiIhIfJQcREQkRclBRERSlBxERCRFyUFERFKUHEREJCXT5GBmN5rZy2a2ZS/Pfc7M3MzenmVM\nIiKSlvWZw03AOf0XmtkU4Gxga8bxiIjIXmSaHNz9AeDVvTz1ZWAZoG/kiYhEIPg1BzM7D/hPd/9l\n6FhERCQxOuTOzWwccAXQMMD1lwBLAA4++ODZ06dPH8boREQOPBs3bnzF3Sfub72gyQH4M2Aq8Esz\nA5gMbDKzOe7+Uv+V3b0FaAGoq6vzzs7OLGMVERnxzOyFgawXNDm4+2PAOwptM+sC6tz9lWBBiYhI\n5rey3gZ0AMeZ2TYzuzjL/YuIyMBkeubg7h/ez/O1GYUiIiIlhL7mICIyJG+++Sbbtm3jjTfeCB1K\nVMaMGcPkyZM56KCDBvV6JQcRGdG2bdvG+PHjqa2tJX9jS9Vzd3bs2MG2bduYOnXqoLYR/HsOIiJD\n8cYbbzBhwgQlhj7MjAkTJgzpbErJQURGPCWGtKEeEyUHEZGA2traOPfccwG45557+MIXvhA4ooSu\nOYiIROK8887jvPPOCx0GoDMHEZEh6+rqYvr06XziE59g5syZNDU18dOf/pQzzjiDadOm8dBDD/HQ\nQw9x+umnc8opp3D66afz9NNPp7Zz0003cemllwLw3HPPcdppp/Ge97yHK6+8kre+9a1AcqZRX1/P\nBRdcwPTp02lqasK98jVLlRxEpPp0dMCaNcm8Qp599lmWLl3Ko48+ylNPPcV3v/tdHnzwQb70pS+x\nevVqpk+fzgMPPMDmzZu55ppruPzyy0tub+nSpSxdupSHH36YI488sui5zZs3s3btWp544gmef/55\nfvazn1Xs5yhQt5KIVJeODpg/H3p6oKYG1q+HuXOHvNmpU6dy4oknAnDCCScwf/58zIwTTzyRrq4u\nXnvtNS688EKeeeYZzIw333xzP2F28P3vfx+Aj3zkI3zuc5/b/dycOXOYPHkyACeffDJdXV2ceeaZ\nQ/4Z+tKZg4hUl7a2JDH09ibztraKbPYtb3nL7sejRo3a3R41ahQ7d+5k5cqVvPe972XLli388Ic/\nHNJtpn33lcvl2Llz5+AD3wclBxGpLvX1yRlDLpfM6+sz2e1rr73GUUcdBSTXFvbntNNOo7W1FYDb\nb799OEPbKyUHEakuc+cmXUnXXluxLqWBWLZsGStWrOCMM86gt7d3v+uvXbuWG264gTlz5rB9+3be\n9ra3ZRDlHjYcV7mzoPEcRATgySefZMaMGaHDqLjXX3+dsWPHYmbcfvvt3HbbbfzgBz8oaxt7OzZm\nttHd6/b3Wl2QFhGJ0MaNG7n00ktxdw499FBuvPHGTPev5CAiEqF58+bxy1/+Mtj+dc1BRERSlBxE\nRCRFyUFERFKUHEREJEXJQURkiLq6upg5c+aQt9PZ2clnP/vZCkQ0dLpbSUQkEnV1ddTV7fcrCJnQ\nmYOISAXs3LmTCy+8kJNOOokLLriA119/nY0bN3LWWWcxe/ZsGhsb2b59OwD19fUsX76cOXPm8K53\nvYsNGzYAxQP/dHd3c/bZZzNr1iwuueQSjjnmGF555RW6urqYMWMGn/zkJznhhBNoaGjgj3/8Y8V/\nnkyTg5ndaGYvm9mWPsu+aGZPmdmjZvY9Mzs0y5hEpPp0vNjBmg1r6HixciW7n376aZYsWcKjjz7K\nIYccwte//nU+85nPcNddd7Fx40Yuuugirrjiit3r79y5k4ceeoi1a9eyatWq1PZWrVrF+973PjZt\n2sQHPvABtm7duvu5Z555hk9/+tM8/vjjHHroobtrMFVS1t1KNwFfA27ps+wnwAp332lm1wMrgOUZ\nxyUiVaLjxQ7m3zKfnt4eanI1rP/oeuZOGXp9pSlTpnDGGWcAsHjxYlavXs2WLVs4++yzAejt7WXS\npEm711+4cCEAs2fPpqurK7W9Bx98kO9973sAnHPOORx22GG7n5s6dSonn3xyydcPVabJwd0fMLPa\nfsvu79P8OXBBljGJSHVp62qjp7eHXu+lp7eHtq62iiQHMytqjx8/nhNOOIGOfQwoVCi7va+S26Xq\n3vUv2T3iu5UG4CJgXeggROTAVV9bT02uhpzlqMnVUF9bX5Htbt26dXciuO222zjttNPo7u7evezN\nN9/k8ccfH/D2zjzzTO68804A7r//fn79619XJM6BiiY5mNkVwE7g1hLrLDGzTjPr7O7uzi44ETlg\nzJ0yl/UfXc+17722Yl1KADNmzODmm2/mpJNO4tVXX919vWH58uW8+93v5uSTT6a9vX3A27vqqqu4\n//77mTVrFuvWrWPSpEmMHz++IrEOROYlu/PdSj9y95l9ll0I/DUw391fH8h2VLJbRODALdn9pz/9\niVwux+jRo+no6OBTn/oUjzzySFnbGNElu83sHJIL0GcNNDGIiBzotm7dyoc+9CF27dpFTU0N3/jG\nNzLdf6bJwcxuA+qBt5vZNuAqkruT3gL8JH9B5+fu/tdZxiUiEptp06axefPmYPvP+m6lD+9l8bey\njEFERPYvmgvSIiKDNVKHOx5OQz0mSg4iMqKNGTOGHTt2KEH04e7s2LGDMWPGDHobwS9Ii4gMxeTJ\nk9m2bRu6vb3YmDFjmDx58qBfr+QgIiPaQQcdxNSpU0OHccBRt5KIiKQoOYiISIqSg4iIpCg5iIhI\nipKDiIikKDmIiEiKkoOIiKQoOYiISIqSg4iIpCg5iIhIipKDiIikKDmIiEiKkoOIiKQoOYiISIqS\ng4iIpCg5iIhIipKDiIikKDmIiEhKpsnBzG40s5fNbEufZYeb2U/M7Jn8/LAsYxIRkbSszxxuAs7p\nt+zzwHp3nwasz7dFRCSgTJODuz8AvNpv8fnAzfnHNwPvH84YOl7sYM2GNXS82DGcuxk5WlqgsTGZ\nhzRpEpgl81BOPRUOOiiZh7R4MUyYkMxDWr4cpk1L5iHFcjw6OmDNmmQektmeaTi5e6YTUAts6dP+\nTb/nfz2Q7cyePdvL1b613cdeN9Zzq3I+9rqx3r61vextHFCam91hz9TcHCaOI44ojuOII7KPYc6c\n4hjmzMk+Bnf3pqbiOJqawsSxbFlxHMuWhYkjluPR3u4+dqx7LpfM2wP97eh7LApT2Zug0wfwN3ZE\nXZA2syVm1mlmnd3d3WW/vq2rjZ7eHnq9l57eHtq62iof5EjS2lq6nZWXXirdzsKmTaXbWVm3rnQ7\nK3ffXbqdlViOR1sb9PRAb28yb2sLE0eGYkgO/21mkwDy85f3taK7t7h7nbvXTZw4sewd1dfWU5Or\nIWc5anI11NfWDzroA8KiRaXbWTniiNLtLMyaVbqdlQULSrezsnBh6XZWYjke9fVQUwO5XDKvrw8T\nR5YGcnpRyYl0t9IXgc/nH38e+PuBbGcw3UruSdfS6gdWq0upoLnZvaEhXJdSQaFrKUSXUsGcOe6j\nR4frUipoanI//PBwXSgFy5a5H3tsuC6lgliOR3u7++rV4bqUCobQpZS8fGDdSpasmw0zuw2oB94O\n/DdwFfB94E7gaGAr8EF373/ROqWurs47OzuHL1gRkQOQmW1097r9rTc6i2AK3P3D+3hqfpZxiIhI\naTFccxARkcgoOYiISIqSg4iIpCg5iIhIipKDiIikKDmIiEiKkoOIiKQoOYiISIqSg4iIpCg5VLtY\natTHEofER5+NYhkdj0zLZ0hkOjpg/vykBHFNDaxfD3PnVm8cEh99NopleDx05lDNYqlRH0scEh99\nNopleDyUHKpZLDXqY4lD4qPPRrEMj0emJbsrSSW7K6SjI/nvo74+7Ol6LHFIfPTZKDbE4zHQkt1K\nDiIiVWSgyUHdSiIikqLkICIiKUoOIiKSouQgIiIpSg4iIpKi5CAiIilKDiIikhJNcjCzvzGzx81s\ni5ndZmZjQsckIlKtokgOZnYU8Fmgzt1nAjngr8JGJSJSvaJIDnmjgbFmNhoYB/xX4HiqQyzlkGOI\no7ERxo1L5iEtXgwTJiTzkGJ4TwBaWpL3pKUlbByxOOQQMEvmw8ndBzwBu4DefUw7gR3AT4CGcrab\n3/ZS4PdAN3Dr/tafPXu2yxC1t7uPHeueyyXz9vbqjaOhwR32TA0N2cfg7t7UVBxHU1OYOGJ4T9zd\nm5uLj0dzc5g4YjF+fPHxGD++7E0AnT6Av8nlnjlcC7yY/wN+E3A9cHO+vQ34NjARWGdm5w50o2Z2\nGHA+MBU4EjjYzFL/NpnZEjPrNLPO7u7uMkOXlFjKIccQx4YNpdtZWbeudDsrMbwnAK2tpdvV5ne/\nK92uoHKTwxvAr4Bad7/Y3S9394tI/qh3kSSJWcD9wOVlbPcvgF+5e7e7vwncDZzefyV3b3H3Onev\nmzhxYpmhS0os5ZBjiGPevNLtrCxYULqdlRjeE4BFi0q3q8348aXblTSQ04vCBLwAnLeP584HXsw/\nXgT8voztngo8TnKtwUjORj5T6jXqVqqQ9nb31avDdRvEFEdDQ9KFEqpLqaCpyf3ww8N1KRXE8J64\nJ11JDQ3qUioodC0NokvJfeDdSmWV7DazPwKL3T11bmdmHwRudvdxZnYWcK+7H1zGtlcBf0ly7WIz\n8Al3/9O+1lfJbhGR8g20ZHe5Y0h3AlebWbu7b++zsyOBq/LPAxxDmXcbuftV+W2IiEhg5SaHpcB6\n4Fdm1gG8DLwDmAu8DhQuIh8LfLdSQYqISLbKSg7uvsnMjgX+luQ6wYnAduAfgBvcfUd+vSsrHaiI\niGRnwMnBzHLATOC/3L2cO5FERGSEKedWVie5pnDKMMUiIiKRGHBycPddJF+AG/AdSCIiMjKV+yW4\nZuAyM6sZjmBERCQO5d6tNB74M+B5M7uP5GJ03y9KeP6WVBERGcHKTQ59L0RftJfnHX1XQURkxCv3\nVtaYSnyLiMgw0R97iUMMYwfEEENMYjkey5fDtGnJPKRYjsepp8JBByXz4TSQAkwxTiq8dwCJYeyA\nGGKISSzHY9my4vELli0LE0csx2POnOLjMWdO2ZtgmMZzKIypsNnMXjez3v7TMOQvOdDFMHZADDHE\nJJbjcffdpdtZieV4bNpUul1BZSUHM/so8FXgYWAM8M/Ad4DfAs8B11Q6QKkCMYwdEEMMMYnleCxc\nWLqdlViOx6xZpdsVVG7J7k3APSQjwr0J1HlSb+kwoA34hrt/bTgC7U8luw8wHR3Jf2P19TB3bvXG\nEJNYjsfy5ckZw8KFcP314eKI5XicempyxjBrFvziF2W/fKAlu8tNDr8jGdTn30mSw5nu/vP8cx8C\n/s7dp5Ud7SAoOYiIlG+gyaHcaw5/BEblL2q8BLyzz3O/Jxn/WURERrhyvwT3GMlYDT8FNgCXm9mv\nSEZvuxp4qqLRiYhIEOUmhxb2nC2sJEkSD5KM+/xb4P2VC01EREIp9xvSd/R5/KyZnUAyCtw4oN3d\nX6lwfCIiEkC5Zw4AmNkUYArJ7ay7SK43nGRmuPu/VTA+EREJoKzkYGbvBG4F5hQWsacqa+FxrmLR\niYhIEOWeOXwTOBq4jOTic0/FIxIRkeDKTQ7vAT7m7q2VDsTMDiVJPjNJzkAucndVQBMRCaDc5LCN\n4Ttb+Apwn7tfkB9pbtww7UdERPaj3C/BrQaWm1lFx5E2s0OAPwe+BeDuPe7+m0ruo6DlzuU0XjON\nljsDl/+NRMeLHazZsIaOFwOfpLW0QGNjMg+lsRHGjUvmIcVSojqG9ySmOGKR1edjIKVb+07A3wHd\nwA+BW/pNN5e7vfw2TwYeAm4CNpN0Lx1c6jWDKdndfMcy5yp2T813BCr/G4n2re0+9rqxnluV87HX\njfX2rYHKEDc3F5chbm7OPoaGhuIYGhqyj8E9nhLVMbwnMcURiwp8PhiOkt1m9jFgBXAoMAuYt5dp\nMEbnt/dP7n4K8Afg83vZ/xIz6zSzzu7u7rJ30vpUvtyv9WtXqbauNnp6e+j1Xnp6e2jragsTSGtr\n6XYWNmwo3c5KLCWqY3hPYoojFhl+PsrtVloFfA+Y6O5HufvUftM797eBfdgGbHP3QonBu0iSRRF3\nb3H3OnevmzhxYtk7WTQ9X+7X+7WrVH1tPTW5GnKWoyZXQ31tfZhAFi0q3c7CvHml21mJpUR1DO9J\nTHHEIsvPx0BOLwoTyZfd5pfzmjK2vQE4Lv/4auCLpdYf7EhwzXcs84ZVx1Z9l1JB+9Z2X/3A6nBd\nSgXNzUlXTshug4aGZJSvUF1KBcuWuR97bLgupYIY3pOY4ojFED8fDLBbqdyS3fcBP/JhGLPBzE4m\nudZQAzwPfNzdf72v9VWyW0SkfAMt2V3uraxLgTvN7NfAfUDqj7e77ypzm4XXPQLsN2ARERl+5SaH\nJ/PzW/bxvA9imyIiEply/5Bfw55aSiIicoAqt2T31cMUh4iIRKTcW1lFRKQKKDmIiEiKkoOIiKQo\nOYiISIqSg4iIpCg5iIhIipJDKB0dsGZNMg8ZxteWs+aiaXR8LezYAS1fXkzjZRNo+fLicEFE8p5E\nM56D4ijSsa6FNdc10rEu8LgSsY7nEMs02MJ7UWhvTwq85XLJvD1M0bv2ry7zsVfguSvxsVfg7V8N\nU+it+Yam4nE2bmjKPohI3pNoxnNQHEXa720u/l25N1ARwFjHc5AKaWuDnh7o7U3mbW1hwth0Nz05\n6B0FPaOSdgitL6xLHli/dpYieU+iGc9BcRRp29ha/LuyMdC4EhGP5yCVUF8PNTWQyyXz+vowYcxa\nSE0v5HqhZlfSDmHRMQuSB96vnaVI3pNoxnNQHEXqZy8q/l2ZHWhciVjHc4hpGtHdSu5Jt8Xq1eG6\nLwphfHWZr/74scG6lAqab2jyhqWHh+lSKojkPYlmPAfFUaT93mZffW1DuC6lghjHc4iJxnMQESnf\nQMdzULeSiIikKDmIiEiKkoOIiKQoOYiISIqSg4iIpCg5iIhIipKDiIikKDmIiEhKVMnBzHJmttnM\nfhQ6FhGRahZVcgCWAk8O5w5aNrbQ+O1GWjaGLbsbS/nfjsbjWXNWjo7G48PG8YlG1jSOo+MTjeGC\niKQ0dDSlwxcvhgkTknlIxx+f1Lw6PuxnNBotLdDYmMyH00BqbGQxAZOB9cD7gB/tb/3B1FZq7mx2\nrmb31NwZpkZKLOV/2xtmFMfRMCNMHBc3FMdxcUP2QURSGjqa0uFNTcXHoylQzasZM4rjmBHmMxqN\n5ubi49Fc/t8ORmDJ7rXAMmDXvlYwsyVm1mlmnd3d3WXvoPWJ1pLtrMRS/rftjaeL43jj6TBxvLih\nOI4XN2QfRCSloaMpHb5uXel2Vp5+unS72rS2lm5XUBTJwczOBV52942l1nP3Fnevc/e6iRMnlr2f\nRccvKtnOSizlf+vHHFccx5jjwsQxZV5xHFPmZR9EJKWhoykdvmBB6XZWjjuudLvaLFpUul1JAzm9\nGO4JWANsA7qAl4DXge+Ues1gS3Y3dzZ7wy0NwbqUCmIp/9veMMNX//moYF1Ku+O4uMFXN4wN06VU\nEElp6GhKhzc1uR9+eLgupYIZM9xHjVKXUkFzs3tDw6C6lNxHcMluM6sHPufu55ZaTyW7RUTKp5Ld\nIiIyaKNDB9Cfu7cBbYHDEBGpajpzEBGRFCUHERFJUXIQEZEUJQcREUlRchARkRQlBxERSVFyEBGR\nFCUHERFJqbrkYKts9xRS47cbGfd342j8dsDxC4CWyxtpvGQcLZcHjuPO5TReM42WOwOOpdDYCOPG\nJfOQYokjq3ED9ieW8S1iOR5ZjW8xkAJMMU6DKbzH1ThX9ZmupuxtVELDLQ1F40o03BKm2Fzzioai\n49G8IlAcdywrjuOOAIXvGhqK6+Q3BCoAGEscFRg3oCJiGd8iluNRgfEtGIHjOWQn7EkDG7ZuKNnO\nSuuO/H6tXzvrOJ66uziOpwKMpbBhQ+l2tcWR4bgBJcUyvkUsxyPD8S2qMzkELkQ77+h5JdtZWTQh\nv1/v1846jukLi+OYHmAshXnzSrerLY4sxw0oJZbxLWI5HlmObzGQ04sYp8GO59C3OyekhlsafOx1\nY4N1KRU0r2jwhiVjg3Up7Y7jjmXesOrYMF1KBQ0NSddFqK6c2OIY4rgBFRPL+BaxHI8hjm/BSB3P\nYaA0noOISPk0noOIiAyakoOIiKQoOYiISIqSg4iIpCg5iIhIipKDiIikKDmIiEiKkoOIiKREkRzM\nbIqZ/buZPWlmj5vZ0tAxiYhUs9GhA8jbCfytu28ys/HARjP7ibs/ETowEZFqFMWZg7tvd/dN+ce/\nA54EjhqOfcUynkPt2lpGrRpF7draoHGcunISB11pnLpyUtA4OlYsZs37J9CxYnG4IJYvh2nTknlI\nixfDhAnJPKRYxlGI5X2JRUafj+hqK5lZLfAAMNPdf7uv9QZTW2lvCcGvyv7nr11bywuvvbC7fczb\njqHrsq7M4zh15SQeyr20uz2n9wh+ce32zOPoWLGY+blb6clBTS+s721i7prvZBvE8uXw93+/p71s\nGVx/fbYxQPILf+ute9pNTfCdjI8FJAlh/vykTHZNDaxfD3PnZh9HLO9LLCrw+RiRtZXM7K1AK3DZ\n3hKDmS0xs04z6+zu7s4+wArZ+trWku2sbLJ8YrB+7Yy1PbmOnhz0joKeUUk7c3ffXbqdlXXrSrez\nEss4CrG8L7HI8PMRTXIws4NIEsOt7r7XT4C7t7h7nbvXTZw4MdsAK+jotx1dsp2VWX5E8sD7tTNW\nP2MBNb2Q64WaXUk7cwsXlm5nZcGC0u2sxDKOQizvSywy/HxE0a1kZgbcDLzq7pcN5DWDLdndt2sp\nRJdSQe3aWra+tpWj33Z0kC6lglNXTmKTvcQsD9OlVNCxYjFtT66jfsaC7LuUCpYvT/4zXbgwbNfF\n4sXJf4QLFoTpUiro6EjOGOrrw3QpFcTyvsRiiJ+PgXYrxZIczgQ2AI8Bu/KLL3f3e/f1Go3nICJS\nvoEmhyhuZXX3Bwk+srOIiBREc81BRETioeQgIiIpSg4iIpKi5CAiIilKDiIikqLkICIiKUoOIiKS\nouQgIiIpVZccbJVhV4Uv2W0r83GsDBtHLh9HLnAcLYtqafzoKFoW1YYLIpYS1ZHE0bGuhTXXNdKx\nrkVxxKSxEcaNS+bDKIryGYMx6JLdfX9cC1NfyVYa5Pos6AW/Nvs4ciuNXX3iGNULvQHiaFlUyyUn\n7ilh3vzYMSxp7co2iFhKVEcSR8e6Fub/7JI9ZdTPaGbugiVVG0c0Ghvh/vv3tBsa4Mc/LmsTI7Jk\n97ArVG2yfu2sFY669WtnbFe/OHYFiqP14K1FcexuZymWEtWRxNG2sbW4jPrG1qqOIxobNpRuV1B1\nJYfCT+v92lkrJCXv187YqH5xjAoUx6I/HF0Ux+52lmIpUR1JHPWzFxWXUZ+9qKrjiMa8eaXbFVRV\n3UqQ71raBYwKW7LbVlqSnHaF6VIqyK00do1KEkOILqWClkW1tB68lUV/ODr7LqWCWEpURxJHx7oW\n2ja2Uj97UdCunFjiiEZjY3LGMG9e2V1KMMJKdg+GSnaLiJRP1xxERGTQlBxERCRFyUFERFKUHERE\nJEXJQUREUpQcREQkRclBRERSlBxERCQlmuRgZueY2dNm9qyZfT50PCIi1SyK5GBmOeDrwALgeODD\nZnZ82KhERKpXFMkBmAM86+7Pu3sPcDtwfuCYRESqVizJ4SjgxT7tbfllIiISwOjQAeTtbRiyVEVA\nM1sCFMoy/t7Mnh7WqIbf24FXQgcRER2PPXQsiul4FBvK8ThmICvFkhy2AVP6tCcD/9V/JXdvAQ6Y\nsQLNrHMg1RGrhY7HHjoWxXQ8imVxPGLpVnoYmGZmU82sBvgr4J7AMYmIVK0ozhzcfaeZXQr8mGR0\n5Rvd/fHAYYmIVK0okgOAu98L3Bs6jowdMF1kFaLjsYeORTEdj2LDfjxG7EhwIiIyfGK55iAiIhFR\ncgjAzKaY2b+b2ZNm9riZLQ0dU2hmljOzzWb2o9CxhGZmh5rZXWb2VP4zMjd0TKGY2d/kf0e2mNlt\nZjYmdExZMrMbzexlM9vSZ9nhZvYTM3smPz9sOPat5BDGTuBv3X0GcBrwaZULYSnwZOggIvEV4D53\nnw68myo9LmZ2FPBZoM7dZ5LcrPJXYaPK3E3AOf2WfR5Y7+7TgPX5dsUpOQTg7tvdfVP+8e9Ifvmr\n9hvhZjYZ+F/AN0PHEpqZHQL8OfAtAHfvcfffhI0qqNHAWDMbDYxjL99/OpC5+wPAq/0Wnw/cnH98\nM/D+4di3kkNgZlYLnAL8ImwkQa0FlgG7QgcSgXcC3cA/57vZvmlmB4cOKgR3/0/gS8BWYDvwmrvf\nHzaqKPwPd98OyT+awDuGYydKDgGZ2VuBVuAyd/9t6HhCMLNzgZfdfWPoWCIxGpgF/JO7nwL8gWHq\nNohdvi/9fGAqcCRwsJktDhtV9VByCMTMDiJJDLe6+92h4wnoDOA8M+siqcb7PjP7TtiQgtoGbHP3\nwpnkXSTJohr9BfArd+929zeBu4HTA8cUg/82s0kA+fnLw7ETJYcAzMxI+pSfdPcbQscTkruvcPfJ\n7l5LcrHx39y9av87dPeXgBfN7Lj8ovnAEwFDCmkrcJqZjcv/zsynSi/O93MPcGH+8YXAD4ZjJ9F8\nQ7rKnAH8H+AxM3skv+zy/LfERT4D3JqvM/Y88PHA8QTh7r8ws7uATSR3+G2myr4pbWa3AfXA281s\nG3AV8AXgTjO7mCSBfnBY9q1vSIuISH/qVhIRkRQlBxERSVFyEBGRFCUHERFJUXIQEZEUJQepGmZW\nb2ZuZvWhY6kEM+sys5v6LfvHVQkkAAADdElEQVTfZvaYmb2R/1kPDRSejHC6lVWqRr6o3fHAEwdC\nuRIzOwX4rbs/l2+PJinS1g6sBnqAh929N1yUMlIpOYgcIMzsGKALuNjdbwwcjoxw6laSA4qZvcvM\nvpcfIOUNM9tqZv9iZqP7dyuZ2dX59t6mj/XZ5lQzu9XMus3sT2b2iJl9oMy4rjaz1H9iZnZTvq5U\noV2b3/8lZnaNmW03s9+Y2Q/zpc37vnZ3t5KZXU2SGAC+ld9GWzkxivSl8hlyoPkR8BvgU8ArJONk\n/E/2/o/QN4H7+i37HEl9/P+AZNQ+knLqLwN/Q1JO+y+BVjN7v7vfMww/A8AKku6hi0hKMv8DcCtw\n1j7W/yawBfgX4DrgX4ER33Um4Sg5yAHDzN4OTAPO7/dH+7v554vWd/dtJFVQC6//ILCQpIR6e37x\n1YABZ7n7jvyyH+eTxjUkRdCGwwvu/pE+sU0EvmhmR7p7asAbd9/Wp07Xc+7+82GKS6qEupXkQLKD\npFDdF8zsk2Y2baAvNLM6klG1/tHd/1+fp84B7gVey3dNjc5f+P0x8O78Re7h8K/92o/l50cP0/5E\niig5yAHDk7srzgY6gTXAf5jZ82b2qVKvy/fl3wO0kYxl3dc7gI8Cb/abvph/fkKl4u+n/9CQf8rP\nxwzT/kSKqFtJDiju/jzw0Xz9/3cDlwL/mL/o+8f+6+eH4LyH5PrEX+7lts8dwAbg+n3scqBjGr+R\n31+Nu/f0WT5cyUVkSJQc5ICUP4t4xMz+L3AxMBN4uO86+QTyHZIhKOe4++/2sqn7gLnA4+6eSi5l\neCE/n0kyPgH5L6idDuxtvyJBKTnIAcPMTgK+AtwBPAvkgI+RDBTzb8D4fi9ZTnJn0lLgSDM7ss9z\nz7l7N3Al8BDwgJl9jeR20cNI/si/090vGmB464DXgG+Y2VXAW4BlwO/L+ylFsqHkIAeSl0hGxvq/\nwGSSrpzHgHPdfeNeymZMz8+/spdtfRy4yd235i9WX03yreOJJF1NW0guYA+Iu//GzM4FvgzcSXKX\n1DUk4yT3j0skOH1DWkREUnS3koiIpKhbSWSI8t97KKXXdYouI4zOHESGwMxqSX8Hov+0r5IXItHS\nNQeRITCzGuCk/az29D5ukxWJlpKDiIikqFtJRERSlBxERCRFyUFERFKUHEREJEXJQUREUv4/Dp3K\nxcaA0ScAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a12bf12b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get the response.  Convert to a zero-one indicator \n",
    "yraw = np.array(df['class'])\n",
    "BEN_VAL = 2   # value in the 'class' label for benign samples\n",
    "MAL_VAL = 4   # value in the 'class' label for malignant samples\n",
    "y = (yraw == MAL_VAL).astype(int)\n",
    "Iben = (y==0)\n",
    "Imal = (y==1)\n",
    "\n",
    "# Get two predictors\n",
    "xnames =['size_unif','marg'] \n",
    "X = np.array(df[xnames])\n",
    "\n",
    "# Create the scatter plot\n",
    "plt.plot(X[Imal,0],X[Imal,1],'r.')\n",
    "plt.plot(X[Iben,0],X[Iben,1],'g.')\n",
    "plt.xlabel(xnames[0], fontsize=16)\n",
    "plt.ylabel(xnames[1], fontsize=16)\n",
    "plt.ylim(0,14)\n",
    "plt.legend(['malign','benign'],loc='upper right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above plot is not informative, since many of the points are on top of one another.  Thus, we cannot see the relative frequency of points.  \n",
    "\n",
    "One way to improve the plot is to draw circles on each point whose size is proportional to the count of samples at that point.  We will re-use this code, so we define a function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'matplotlib.pyplot' from '/Users/yao/anaconda3/lib/python3.6/site-packages/matplotlib/pyplot.py'>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAERCAYAAACQIWsgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xt8VPW57/HPkwv3AHITQlBQQa6K\nEhClVVu31VOt2h7t3tVWW2ttPdVSd3u0tq/Wy97Hul92W91ttbVCwapQRa3UKl5r1YqFBOR+v4dr\nSCAQbrk95481QMgEyOQyv2Hyfb9eeWVmzZrf78lkZj2zfmut52fujoiISG0ZoQMQEZHUo+QgIiJx\nlBxERCSOkoOIiMRRchARkThKDiIiEiepycHMJprZNjNbWM9jPzQzN7MeyYxJRETiJXvPYRJwed2F\nZtYPuBRYn+R4RESkHklNDu7+PlBaz0O/BO4CdEWeiEgKCH7MwcyuAja6+7zQsYiISCQrZOdm1gH4\nCfC5Bq5/K3ArQMeOHUcNHjy4BaMTEUk/hYWF29295/HWC5ocgNOBAcA8MwPIA+aY2Rh331J3ZXd/\nEngSID8/3wsKCpIZq4jICc/M1jVkvaDJwd0XAL0O3jeztUC+u28PFpSIiCT9VNYpwEzgTDMrMrNv\nJrN/ERFpmKTuObj7V47zeP8khSIiIscQ+piDiEiTVFZWUlRUxP79+xu0/pa9WyirKKOqpoo2mW3o\n2a4nXdt2beEok69du3bk5eWRnZ3dqOcrOYjICa2oqIicnBz69+9P7MSWOAeqDjBv6zxmb5xN8d7i\nww9Ugu03Bpw0gNG5ozmzx5lkWPAz/JvM3SkpKaGoqIgBAwY0qg0lBxE5oe3fv/+YiWFD2QamLpzK\nnso99T7uOKt3rGb1jtXk5uRy/Yjr6dSmU0uG3OLMjO7du1NcXHz8lY/ixE+RItLqHS0xrC9bz9Pz\nnj5qYqhr0+5NTJw7kT0VDVs/lR3tNWkoJQcRSUu7DuxiyoIpVNZUJvS80n2lTFk4JaHnrF27luHD\nhyf0nPoUFBTwve99r8ntNAcNK4lIWirYVMC+qn2Nem7RriJW71jNaSed1sxRHVt+fj75+flJ7fNo\ntOcgImmnuqaaOZvnNKmN2RtnJ7R+VVUVN910E2eddRbXXnste/fupbCwkIsuuohRo0Zx2WWXsXnz\nZgAuvvhi7r77bsaMGcOgQYP44IMPAHjvvfe48sorASguLubSSy/l3HPP5dvf/jannnoq27dvZ+3a\ntQwZMoRvfetbDBs2jM997nPs29e4JHgsSg4iknaWbl9KeUV5k9pYVrKM3Qd2N3z9Zcu49dZbmT9/\nPp07d+Y3v/kNd9xxB9OmTaOwsJCbb76Zn/zkJ4fWr6qqYtasWTz66KPcf//9ce3df//9fPazn2XO\nnDl88YtfZP36wzMarFixgu9+97ssWrSIrl278uKLLzbpb62PhpVEJO0ccbpqI9V4DSX7Sshpm9Og\n9fv168e4ceMA+OpXv8qDDz7IwoULufTSSwGorq6mT58+h9b/0pe+BMCoUaNYu3ZtXHsffvghL7/8\nMgCXX345J5100qHHBgwYwMiRI4/5/KZSchCRtHOg6kDS26l7dlBOTg7Dhg1j5syZ9a7ftm1bADIz\nM6mqqop73P3o09scfO7B52tYSUSkAdpktkl6O+vXrz+UCKZMmcLYsWMpLi4+tKyyspJFixY1uL1P\nfepTPP/88wC8+eab7NixI4HIm07JQUTSTo8OTZ+K3jC6d+je4PWHDBnC5MmTOeussygtLT10vOHu\nu+/m7LPPZuTIkXz00UcNbu/ee+/lzTff5Nxzz+X111+nT58+5OQ0bIirOdixdl1SmeZzEBGAJUuW\nMGTIkCOWVdVU8cjMR9hbubfR7Q7qPojrR1zf1PAa7cCBA2RmZpKVlcXMmTO57bbb+OSTTxJqo77X\nxswK3f2458vqmIOIpJ2sjCxG9h7JRxsa/k29rtG5o5sxosStX7+eL3/5y9TU1NCmTRt+//vfJ7V/\nJQcRSUtj+o6hcFMhB6oTPzjdu1Nvzuh2RgtE1XADBw5k7ty5wfrXMQcRSUtd23XlumHXkWmZCT2v\nc9vOfGX4V5pcm+hEp+QgImnrjG5ncP2I62mb2fb4KwM9O/Tk5nNupku7Li0cWerTsJKIpLXTu53O\nHefdwZzNcyjcVEjZgbK4dXJzchmdO5rhvYaTndm4yXHSjZKDiKS9Tm06ceGpF/KpUz7FqtJV7Ny/\nk4rqCtpltaNPTh9yc3JDh5hyNKwkIq1GhmUwsPtARvcdzbhTxjEqd1TwxFC72N706dN56KGHgsZz\nkPYcRERSxFVXXcVVV10VOgxAew4iIk22du1aBg8ezC233MLw4cO54YYbePvttxk3bhwDBw5k1qxZ\nzJo1iwsuuIBzzjmHCy64gGXLlsW1M2nSJG6//XYAVq1axdixYxk9ejQ/+9nP6NQpmrr0vffe4+KL\nL+baa69l8ODB3HDDDcesw9RYSU0OZjbRzLaZ2cJayx42s6VmNt/MXjazrsmMSURamS1bYMIE+PnP\n4YUXoJmK1q1cuZLx48czf/58li5dynPPPceHH37IL37xCx588EEGDx7M+++/z9y5c3nggQf48Y9/\nfMz2xo8fz/jx45k9eza5uUcOfc2dO5dHH32UxYsXs3r1av7xj380y99QW7L3HCYBl9dZ9hYw3N3P\nApYD9yQ5JhFpLdzh+edhwwY4cAAWLYI332yWpgcMGMCIESPIyMhg2LBhXHLJJZgZI0aMYO3atZSV\nlXHdddcxfPhw7rzzzuMW4Zs5cybXXXcdANdff2QZjzFjxpCXl0dGRgYjR45skZLdSU0O7v4+UFpn\n2ZvufrBe7cdAXjJjEpFWZPduKC09ctm6dc3SdO0y2hkZGYfuZ2RkUFVVxU9/+lM+85nPsHDhQv7y\nl7+wf//+ZunraCW/myrVjjncDLweOggRSVOdOkGXOhe49e2blK7LysroG+tr0qRJx11/7Nixh2Z4\nmzp1akuGVq+USQ5m9hOgCnj2GOvcamYFZlZQXNz0mZ5EpJXJyIBrr4WePaP7p58Ol12WlK7vuusu\n7rnnHsaNG0d1dfVx13/00Ud55JFHGDNmDJs3b6ZL3aTWwpJestvM+gOvuvvwWstuAr4DXOLuDaqx\nq5LdIgL1l6VukJqaKFmkqL1799K+fXvMjKlTpzJlyhReeeWVhNo4oUt2m9nlwN3ARQ1NDCIiTZbC\niQGgsLCQ22+/HXena9euTJw4Man9JzU5mNkU4GKgh5kVAfcSnZ3UFngrVgXxY3f/TjLjEhFJNZ/+\n9KeZN29esP6Tmhzc/Sv1LJ6QzBhEROT4Unu/SkSkAU7U6Y5bUlNfEyUHETmhtWvXjpKSEiWIWtyd\nkpIS2rVr1+g2gh+QFhFpiry8PIqKitDp7Udq164deXmNv6ZYyUFETmjZ2dkMGDAgdBhpR8NKIiIS\nR8lBRETiKDmIiEgcJQcREYmj5CAiInGUHEREJI6Sg4iIxFFyEBGROEoOIiISR8lBRETiKDmIiEgc\nJQcREYmj5CAiInGUHEREJI6Sg4iIxFFyEBGROEoOIiISR8lBRETiJDU5mNlEM9tmZgtrLetmZm+Z\n2YrY75OSGZOISMpyh82bYeNGqKlJatfJnkN6EvBr4Olay34EvOPuD5nZj2L3705yXCKSaqqqIKsV\nT3O/cSO89BKUlET3u3SBq6+G005LSvdJ3XNw9/eB0jqLrwYmx25PBq5JSjBbtkQZOaSqKtizJ2wM\ny5fDG2/AmjVh45DUsWkTzJ4Nu3eH6X/vXpgwAf7zP+Gxx2DbtjBxhHTgADz77OHEAFBWBlOmJO3/\nkgrHHE52980Asd+9WrzHJUvgd7+LfhYuPP76LaGyEn77W/jFL2DevDAxFBVFb7aZM+GZZ2D79jBx\n7NsHL74IK1aE6R+gvBwefxyefhqqq8PE4A7PPQe//jXs3Bkmhm3bog3zX/8KEycmfSgDgLffhg0b\nots7dsArryQ/hoNWrQrzv1i8OEqSdVVWJm17kQrJocHM7FYzKzCzguLi4sY3tGlT9EGEaNcthD17\noo2xO6xfHyaGkpLDr0N1NZTW3alLkuJiWLAg+kCEsn17tGFcswb27w8TQ1UVrFwZxbJ1a5gYtm8/\nnBx37Ig2Rsm2a9ex7yfT3/8e/U+S7VgjCkkabUiFAb2tZtbH3TebWR/gqPuQ7v4k8CRAfn6+N7rH\n0aOjb801NXDeeY1upkm6doXLL48S1YUXholh0CDo1SvaKOblwYABYeI45RS45Rbo2TNM/wD9+8NV\nV0HHjtFPCNnZcP310UZ50KAwMQwcGL0P1q+H88+Htm2TH8PQoUdukIcOTX4MB918c5h+Tznl6I+d\nempSQjD3xm9jG9WhWX/gVXcfHrv/MFBS64B0N3e/63jt5Ofne0FBQYvG2ipUV0dDKjk5kHFC7UhK\nOlu0KNqL69kTxowBs9ARJd+f/hQNgdfWvz/ceGOTPqtmVuju+cddL5nJwcymABcDPYCtwL3An4Hn\ngVOA9cB17n7c8Q0lBxFJazU1UFAQJcqaGhgyJBr1yM5uUrMNTQ5JHVZy968c5aFLkhmHiEjKy8iI\n9prGjAnTfZBeRUQkpSk5iIhIHCUHERGJo+QgIiJxlBxERCSOkoOIiMRRchARkThKDiIiEkfJQURE\n4ig5tHbuR9aMFxFByUFmzYJf/Qrmzw8dSXCznvgpC/78ZOgwwtu0KZprpKoqdCQC8P77MGNG0rtN\nhZLdElKfPtCvX1S6u5Xrcc4FtO/SI3QY4XXpAiNHQmZm6EgEohLdAebVSHrJ7uaiqqwiIolraFVW\nDSuJiEgcJQcREYmj5CAiInGUHEREJI6Sg4iIxFFyEBGROEoOIiISR8lBRETiKDmIiEiclEkOZnan\nmS0ys4VmNsXM2oWOSUSktUqJ5GBmfYHvAfnuPhzIBP4tbFQiIq1XSiSHmCygvZllAR2ATYHjkWQp\nLY0qTy5dGjoSkdRTUwPV1UnvNqHkYGY1ZlZ9lJ8qMysxs7fM7HOJtOvuG4FfAOuBzUCZu7+ZSBuJ\nKlm/jO1rl7RkF8f3xhtRaeTt28PFUFMDy5dH8zqEUFEBEyfCu+/C1KmweHGYOPbujeJ4/vnoNQll\n/nx4++1w/w+AmTPhN7+BdevCxVBeDtOnw+bN4WIA2LAhiiWUnTvhkUfg4Ydh69akdp3onsN/ABuA\nYmAS8F/A5Nj9IuCPQE/gdTO7sqGNmtlJwNXAACAX6GhmX61nvVvNrMDMCoqLixMM/Uir/voMK/8y\nqUltNNncubBlC6xZEy6GkpJogxjqA1BefmTfoTYGW7bA+vVRctq7N0wMAB9/DP/4B+zZEy6GefOg\nuBiWLQsXw5o1MGcOLFoULgaA11+HhQvD9b9pU/T52L8/en8mUUIlu83sHuBzwP9y9/21lrcHXgfe\nAn4O/BXo4u4XNLDd64DL3f2bsfs3AmPd/f8c7TlNLdntsW+HlhFwZG3lSigqgnHjIDs7XBw1NRDq\ndXCHZ5+NXov27eEb3wgzt4R79I25Y0c4++zk93/Qjh3RxqBfv3AxbNwYDfGNHRu9HiG4w4oV0L8/\ntGkTJgYI+9mAaMKl116L5nO44gpo1/TzdBpasjvR5LAOuMPdp9fz2NXAr929n5n9b2Cyu3dqYLvn\nAROB0cA+or2SAnf/1dGeo/kc0khNTfRNtXPnKEGISItpaHJIdCa4XsDRvuK2AbrHbm8HrKGNuvs/\nzWwaMAeoAuYCmq+xtcjIgJNPDh2FiNSSaHIoAO4zs4/c/dDgsJnlAvfGHgc4lQTPNnL3e2NtiIhI\nYIkmh/HAO8AaM5sJbCPamzgf2AscPIh8BvBccwUpIiLJlVBycPc5ZnYG8APgPGAE0amn/w084u4l\nsfV+1tyBiohI8jQ4OZhZJjAc2OTuP265kEREJLREztFyomMK57RQLCIikiIanBzcvYboArhAJz6L\niEiyJHp1x++A75tZwKtSRESkpSV6tlIOcDqw2sxmEB2Mrn0VncdOSRURkRNYosmh9oHom+t53NG1\nCiIiJ7xET2VNpRLfIiLSQrSxl7AVQCGqgvr3v4etTgtR1cskl0WOU10NBw6EjUFSy5YtUYHOJGu1\nyWH+i4/zydTHwgaxcmVULjukwsKoVnzI0sgzZsDf/hZVZ62oCBPDxo3RfA6//z3s2xcmBoDnnoNf\n/jJsDFu2wKRJQSaYOaSiAv7wh7BznaSC3buj9+SECbBtW1K7TvSYA2Z2K3AbcCbQtu7j7p7ZDHG1\nuJy80/GagG9+iKqQnnJK2Bi6doWePaFLl3AxdO4c/e7QATIDvX06dIC2baPfWQl/LJpP797RRjlk\nCfeOHaNS2SFLVWdlRTG09iq92dnR/6OyslnKdSci0ZLdNwK/J5rg5xaiMtvZwFVEE/486+73t0Cc\ncVSyO424R7OO9ewZbv4AiD6AGRnhEpRIfaqqos9IM31haGjJ7kS/GnyfaDKf22L3H3f3m4DTiOZh\nKEmwPREwi74lhkwMEH34lBgk1WRlBdmTTDQ5DATeB2piP20A3H0H8P+IqraKiMgJLtHksA/I8Ggs\nagvRHsNB5UTzP4uIyAku0SNvC4jmangb+AD4sZmtIZq97T5gabNGJyIiQSSaHJ7k8N7CT4mSxIdE\nU4LuAq5pvtBERCSURK+Q/lOt2yvNbBjRLHAdgI/cvZWflCwikh4adUK3mfUD+gHtiA5MlwNnmRnu\n/m4zxiciIgEklBzM7DTgWWDMwUUcrsp68LbOBRQROcEluufwFHAK0fUOS4FAtQ5ERKQlJZocRgNf\nd/cXmzsQM+tKlHyGE+2B3OzuM5u7HxEROb5Ek0MRLbe38Bgww92vjc0016GF+hERkeNI9CK4B4G7\nzaxZ6xyYWWfgQmACgLtXuPvO5uxDjqK6GkpLoaYmdCQikkISPZX1j2Y2GFhrZh8DO+JX8ZsaEcdp\nRIX7/mBmZwOFwHh3b7mJBsrKomJWXbu2WBfHVF0NzzwTlYq+7joYODD5MbjD5MnRPAaDBsH11yc/\nhoM2bYJevcJVRD1wAF56Karv9IUvRPWeQnj9dSgpgS99KaoQ21r985/w1ltw9tnR/6O12r07Kl1e\nVQU33gg9eiSt64T2HMzs68A9QFfgXODT9fw0RlasvSfc/RxgD/Cjevq/1cwKzKyguLi4kV3FvPce\nvPNO09poirKyaHKbigpYvDhMDPv2RYkBYMWKsHsPkyfD8uXh+i8qgmXLYM4cKC8PE0NFRbRRXLkS\n1q4NE0OqmD8/2iDOmxc6krDWr4/27HftgtWrk9p1ol/T7gdeBr7ZzMM+RUCRu/8zdn8a9SQHd3+S\n6Cpt8vPzG15rvD5XXNGkpzdZt24wdmz0jXns2DAxdOgAo0bBggUwenTY+v133pn0evVHGDAALrww\n2nPIyQkTQ5s2cNVV0Z7DoEFhYkgVF10UzQ44YkToSMIaOBCGDo3KyQ8bltSuE53PoRy42t2b/Su3\nmX0A3OLuy8zsPqCju//fo62v+RxERBLX0PkcEt1z+BAYArTEeMwdwLOxM5VWA99ogT5ERKQBEk0O\n44HnzWwHMIP4A9K4e6MGrt39E+C42UxERFpeoslhSez300d53BvRpoiIpJhEN+QPcLiWkoiIpKlE\nr3O4r4XiEBGRFBLw3EUREUlVSg4iIhJHyUFEROIoOYiISBwlBxERiaPkICIicZQcQtqzJ6oGGtLK\nlTB9OqxbFzaO9eujKrGtmTt8+GFUtntPy1WrPyFs3Bi9L5csOf666ay8HP70J3juOdgRV5CiRbXe\n5DBrFswMPAvphAnw1FOwaFGY/g8cgKlTozLVzz0XbZxCKCiAiROj+S1Ceu+9sCWiFy6Et9+Oyna/\n+Wa4OIqLo/dFdXW4GA6+L194IVwJ9VTw7rtRgly+HGbMSGrXrbfURYcO4Wc/a9cumlSmbdsw/Wdk\nRJPrVFVBdnaYGOBwqe6QJbsBOnWC9u3D9d+mTf23ky07O5oEK2QJ94N/f2Zm2DhCq/25TPJnNKGS\n3akkLUp2V1XB3r3QuXO4GDZvjib6GTIEevYMF0dZWbRxzswMF0MqWLAgmv1r9OiwCTu00tJoT+q0\n0yAvL3Q04Rw4EO3RVlXBZz7TLLMDNrRkt5KDiEgr0tDk0Ir310RE5GiUHEREJI6Sg4iIxFFyEBGR\nOEoOIiISR8lBRETiKDmIiEgcJQcREYmj5CAiInFSKjmYWaaZzTWzV0PHIiLSmqVa4b3xwBKg5YsN\nFRdHVUh79WrxrkREGqW8PCrjXl0N48ZFBRGTJGX2HMwsD7gCeKrFO/vkE4ofvp9tD98XlYsOpOqj\nDyl/+qmo0Foguw/sZmbhK+yrDDiXwrJl7Pn5A1S/9FK4GIBlb/+JtbPeChoDa9YEfU8C0Zem4uKw\nMcDhL3Ah7dwJFRXh+v/jH+Hjj2H2bJg0KSrAlyQpkxyAR4G7gKPW0TazW82swMwKipvy5l26lKXF\nS1i6fSksW9b4dppoy/QpLPj7C7B4cbAYFsx/mwO/+iWLVgWc22LuXOYs/zulM98JNuFPxb5yNj/7\nW9Y993iQ/g+ZMQNefTXoFwZWr4bf/japG6I4+/fDE0/Ahg3hYoBonpPZs8P0XVEBW7cevr9zZ1Ln\ntkiJYSUzuxLY5u6FZnbx0dZz9yeBJyGqytroDocPZ9i8EXhNDQwf3uhmmir3yzfTfdXyoDGMHHkZ\ni+/qzIgzxgWLgTFjyN9SRLtBQ4PNp9CmfSf63fID2uacFKT/Q77whWgjkJMTLobTT4fx46O5PkJp\n1y6KoUuXcDEA3HRTuHlG2rSBvn2jWfEAundP6vsiJUp2m9nPga8BVUA7omMOL7n7V4/2nCaX7N61\nK9plDf3mExE5mn37omGlmho477xozpMmOmHnc4jtOfzQ3a881nqaz0FEJHGaz0FERBotJY451Obu\n7wHvBQ5DRKRV056DiIjEUXIQEZE4Sg4iIhJHyUFEROIoOYiISBwlBxERiaPkICIicVLuOodkWLB1\nAQWbCnCcUX1GcXbvs0OHJCKSUlpdcli4bSEzPnqaQR8tw4C3zl+MX+CM7D0yqXHUeA1/WzqDsu1F\njB15Jbk5uUnt/3AgNbBtG5x8MpiFiWHtWpg5E3Jz4cILw8UhIoe0umGlgk0FDJq5nC7Fu+hcvItB\nHy2jcFNh0uP4ZMsn7P/db+g24Tneeu3XSe//oHXPPsEH99zAppf/GCYAd5g6lWX/mE75G3+BpUvD\nxFFdzer/+HeKfvdwmP5jVpSs4P117weNISVUVcHLL0NZWdg43n03KmEe0uTJUfnyJNfBa3XJoS4L\nVHfwiIKHgYsfOqlVfDEYd6gJ/79ItWKYQYV+LUL3H1DKVWVtqMZWZV2wdQEz/jGZM2cuB3eWnz+I\nfxn3tTDDSkteZ9f2jZx3TsBhperqaFipd+/ww0p9+sBFF2lYSaQFnbAluxuqKSW7dUBaRFqrhiaH\nVndAGmDEySMYcfKI0GGIiKSsVn/MQURE4ik5iIhIHCUHERGJo+QgIiJxlBxERCSOkoOIiMRRchAR\nkThKDiIiEiclLoIzs37A00BvoAZ40t0fa4m+qmuq+bjoYxZuWwjAsF7DGJs3lqyMlHgpRERSQqps\nEauAH7j7HDPLAQrN7C13X9zcHb245EUWFy+m7Z4D4M7b5ZvZULaBr4z4SnN3dVzLti+j7EAZQ3sO\npVObTknv/6AaryHDtBMpIoelRHJw983A5tjt3Wa2BOgLNGty2Fq+lcXFi+m5ZhtD318COEs/NYRl\nwKbdm5Ja/G7mhpm8seqNQ7dvG30bbTLbJK1/iCrDTls8jUXFi8jPzefKQVcmtf/adm/fRKduvbGM\ngElq3z7IyoLs7HAxiKSIlPu6aGb9gXOAfzZ321v3bAXg5DXbMHfModfqaNmW8i3N3d0xrSxdeej2\njv072L53e1L7B9hbuZdFxYuAaJ6LGq9JegwHFf7kG6z66NVg/QMwbVpUv19EUis5mFkn4EXg++6+\nq57HbzWzAjMrKC4uTrj9Hh16ALC9X3c8VhW65JRoWc8OPRsdd2Oc2vXUQ7dz2uTQrX23pPYP0CG7\nA2d0OwOA4b2GBx1aGvGT/2HAeZcH6x+Aq6+OZqITkdQp2W1m2cCrwBvu/sjx1m9sye5n5j/DytKV\ndNi5B3PYc1JH+nftz9dHfj3xoJvA3Zm/dT479+/krJPP4qT2JyW1/9px7KvaR4fsDkH6F5HkOqFK\ndpuZAROAJQ1JDI21p2IP5/Y+l+I9xSzN3AzAoDa9yc/Np7yiPKkHhc0sJeaRMDMlBhGJkxLJARgH\nfA1YYGafxJb92N1fa2rD7s6K0hXM2jiLVaWrDk2J2SenDwC7K3YzbfE0DOO0k05jdN/RnNn9TEyz\nkYlIK5YSycHdPwSafWu8c/9OXln6Cmt2rjl+DDirdqxi1Y5VnNrlVK4efHWQ4wAiIqkgpQ5IN6fl\nJct5fPbjDUoMda0rW8cTs59gSfGSFohMRCT1pWVyWLZ9GVMXTqWiuqLRbVTWVPLC4hdYXNzs1+GJ\niKS8tEsOpftKeXHJi81yzn6N1/DSkpeCXIMgIhJSWiUHd+eVpa80aY+hrqqaKv689M+kyim/IiLJ\nkFbJYcn2JawrW9fs7RbtKjpUqE9EpDVIq+Qwe+PsFmt71sZZLda2iEiqSZvkULa/rFFnJjXUhl0b\nKNlb0qxtlu4rZUPZBqpqqpq13USUV5Szvmw9eyv3BotBRFJPSlzn0ByKdhW1eB8bd2+ke4fuTW5n\nX+U+pi2exqodq4CoxtHnB36e4b2GN7nthnJ3Xl/5+qGCe1kZWZyfdz6XnHZJ0mIQkdSVNnsOm8s3\nt3wfu5unj9dWvHYoMUBUHfWlJS9Ruq+0WdpviNmbZjNr46xDZ3VV1VTxwfoPWLB1QdJiEJHUlTbJ\nYU/FnoTWH/H2As5+Y15ifVQm1kd9qmqq6r12osZrknrQe96W6G/POlDJwI9XkFlZDcD8rfOTFsNB\nO/bt4LGPH+Pjoo+T3neqefedCbz0h7uorK4MHYqkiCXFS/hkyyfHX7GZpc2wUqLa7tlPRk1ip6c2\nx+ms7n7UazCSeeyh2qNkkFHo5saQAAAJNUlEQVRdQ7vy/Vjsbwtx/KOyppLdFbsTTvDpaN/uUqp3\n7qDaq8lGkw5J9KV0f9X+pPebNsmhfXb7hNYv/MJxK9Y2uY/6ZGdmc0a3M1hRuiLusaE9hza5/YYa\n2nMoW8q3UNGhLQv+ZUSQGA7q1bEXP/rUjzSPN/D5q39IjdeQmZEZOhRJEfm5iW+rmkPaDCv17tQ7\nofU9w/CMxGr99enUJ6H1j+aKQVccmngIIMMyuPS0SxP+G5rign4XMLDbwCOWDe81nFG5o5IWQ21K\nDBEzU2KQlJA2n8i+OX1bvI/mmmO6a7uufHf0d1m9YzV7KvcwoOsActrmNEvbDZWVkcUNZ93Apt2b\nKN5TTO9OvTm508lJjUFEUlfaJIfuHbqTm5PLpt2bWqT9kzue3KwbTzPj9G6nN1t7jZWbk9tsSU9E\n0kfaDCsBjM4d3WJthxr3ExEJIa2Sw1knn0Wvjr2avd0eHXpwTp9zmr1dEZFUlVbJITMjk2sGX0OG\nNd+fZRjXDL5GB0xFpFVJq+QA0Rj65wd+vtnau/yMy8nrnNds7YmInAjS8utwfm4+7s5rK17DadyF\na4Zx2RmXcV7eec0cnYhI6kvL5AAwuu9o+uT04c9L/5zwTG7d2nfjmsHXcEqXU1ooOhGR1Ja2yQEg\nr3Me38n/DoWbCinYVEDx3uJjrt+9fXdG9x3NqD6jyM5U6QIRab1SJjmY2eXAY0Am8JS7P9Qc7WZl\nZHFe3nmcl3ceG3dtZOPujWzevfnQ/AUdsjvQJ6cPuTm59M3pi1liV02LiKSjlEgOZpYJ/Aa4FCgC\nZpvZdHePL1/aBH0796Vv55a/klpE5ESXKmcrjQFWuvtqd68ApgJXB45JRKTVSpXk0BfYUOt+UWyZ\niIgEkBLDSkB9A/1x56Ca2a3ArbG75Wa2rEWjSo4eQGKnU6UvvRYRvQ6H6bWINOfrcGpDVkqV5FAE\n9Kt1Pw+Iq6Dn7k8CTyYrqGQwswJ3V+Em9FocpNfhML0WkRCvQ6oMK80GBprZADNrA/wbMD1wTCIi\nrVZK7Dm4e5WZ3Q68QXQq60R3XxQ4LBGRVislkgOAu78GvBY6jgDSapisifRaRPQ6HKbXIpL018Hc\nG1d7SERE0leqHHMQEZEUouQQiJn1M7O/mdkSM1tkZuNDxxSSmWWa2VwzezV0LCGZWVczm2ZmS2Pv\njfNDxxSCmd0Z+1wsNLMpZtYudEzJYmYTzWybmS2staybmb1lZitiv09q6TiUHMKpAn7g7kOAscB3\nzWxo4JhCGg8sCR1ECngMmOHug4GzaYWviZn1Bb4H5Lv7cKKTVP4tbFRJNQm4vM6yHwHvuPtA4J3Y\n/Ral5BCIu2929zmx27uJNgKt8qpwM8sDrgCeCh1LSGbWGbgQmADg7hXuvjNsVMFkAe3NLAvoQD3X\nPaUrd38fKK2z+Gpgcuz2ZOCalo5DySEFmFl/4Bzgn2EjCeZR4C6gJnQggZ0GFAN/iA2xPWVmHUMH\nlWzuvhH4BbAe2AyUufubYaMK7mR33wzRF0ugV0t3qOQQmJl1Al4Evu/uu0LHk2xmdiWwzd0LQ8eS\nArKAc4En3P0cYA9JGD5INbHx9KuBAUAu0NHMvho2qtZHySEgM8smSgzPuvtLoeMJZBxwlZmtJarG\n+1kzeyZsSMEUAUXufnAPchpRsmht/gVY4+7F7l4JvARcEDim0LaaWR+A2O9tLd2hkkMgFs0qNAFY\n4u6PhI4nFHe/x93z3L0/0UHHd929VX5LdPctwAYzOzO26BKgWec0OUGsB8aaWYfY5+QSWuGB+Tqm\nAzfFbt8EvNLSHabMFdKt0Djga8ACM/sktuzHsSvFpfW6A3g2VmNsNfCNwPEknbv/08ymAXOIzuqb\nSyu6UtrMpgAXAz3MrAi4F3gIeN7MvkmUPK9r8Th0hbSIiNSlYSUREYmj5CAiInGUHEREJI6Sg4iI\nxFFyEBGROEoO0mqY2cVm5mZ2cehYmoOZrTWzSXWWfcHMFpjZ/tjf2jVQeHKC06ms0mrECtsNBRan\nQ6kSMzsH2OXuq2L3s4gKtn0EPAhUALPdvTpclHKiUnIQSRNmdiqwFvimu08MHI6c4DSsJGnFzAaZ\n2cuxyVL2m9l6M3vBzLLqDiuZ2X2x+/X9fL1WmwPM7FkzKzazA2b2iZl9McG47jOzuG9iZjYpVlfq\n4P3+sf6/bWYPmNlmM9tpZn+JlTav/dxDw0pmdh9RYgCYEGvjvURiFKlN5TMk3bwK7ARuA7YTzZHx\neer/IvQUMKPOsh8S1cpfDtGMfUSl1LcBdxKV1P5X4EUzu8bdp7fA3wBwD9Hw0M1E5Zn/G3gWuOgo\n6z8FLAReAP4T+Ctwwg+dSThKDpI2zKwHMBC4us5G+7nY40es7+5FRJVQDz7/OuBLROXTP4otvg8w\n4CJ3L4kteyOWNB4gKojWEta5+/W1YusJPGxmue4eN/GNuxfVqtG1yt0/bqG4pJXQsJKkkxKiYnUP\nmdm3zGxgQ59oZvlEM2w97u7/U+uhy4HXgLLY0FRW7MDvG8DZsYPcLeGvde4viP0+pYX6EzmCkoOk\nDY/OrrgUKAB+Diw3s9Vmdtuxnhcby58OvEc0l3VtvYAbgco6Pw/HHu/eXPHXUXeayAOx3+1aqD+R\nI2hYSdKKu68GbozNA3A2cDvweOyg776668em4ZxOdHziX+s57bME+AD4r6N02dC5jffH+mvj7hW1\nlrdUchFpEiUHSUuxvYhPzOzfgW8Cw4HZtdeJJZBniKaiHOPuu+tpagZwPrDI3eOSSwLWxX4PJ5qn\ngNgFahcA9fUrEpSSg6QNMzsLeAz4E7ASyAS+TjRhzLtATp2n3E10ZtJ4INfMcms9tsrdi4GfAbOA\n983s10Sni55EtJE/zd1vbmB4rwNlwO/N7F6gLXAXUJ7YXymSHEoOkk62EM2S9e9AHtFQzgLgSncv\nrKdsxuDY78fqaesbwCR3Xx87WH0f0VXHPYmGmhYSHcBuEHffaWZXAr8Enic6S+oBovmS68YlEpyu\nkBYRkTg6W0lEROJoWEmkiWLXPRxLtWsXXU4w2nMQaQIz60/8NRB1f45W8kIkZemYg0gTmFkb4Kzj\nrLbsKKfJiqQsJQcREYmjYSUREYmj5CAiInGUHEREJI6Sg4iIxFFyEBGROP8fPQWrewgUbowAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10bb77940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_cnt(X,y):\n",
    "    \n",
    "    # Compute the bin edges for the 2d histogram\n",
    "    x0val = np.array(list(set(X[:,0]))).astype(float)\n",
    "    x1val = np.array(list(set(X[:,1]))).astype(float)\n",
    "    x0, x1 = np.meshgrid(x0val,x1val)\n",
    "    x0e= np.hstack((x0val,np.max(x0val)+1))\n",
    "    x1e= np.hstack((x1val,np.max(x1val)+1))\n",
    "\n",
    "    # Make a plot for each class\n",
    "    yval = list(set(y))\n",
    "    color = ['g','r']\n",
    "    for i in range(len(yval)):\n",
    "        I = np.where(y==yval[i])[0]\n",
    "        cnt, x0e, x1e = np.histogram2d(X[I,0],X[I,1],[x0e,x1e])\n",
    "        x0, x1 = np.meshgrid(x0val,x1val)\n",
    "        plt.scatter(x0.ravel(), x1.ravel(), s=2*cnt.ravel(),alpha=0.5,\n",
    "                    c=color[i],edgecolors='none')\n",
    "    plt.ylim([0,14])\n",
    "    plt.legend(['benign','malign'], loc='upper right')\n",
    "    plt.xlabel(xnames[0], fontsize=16)\n",
    "    plt.ylabel(xnames[1], fontsize=16)\n",
    "    return plt\n",
    "\n",
    "plot_cnt(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In-Class Exercise\n",
    "\n",
    "Based on the above plot, what would be a good \"classifer\" using the two features.  That is, write a function that makes a prediction `yhat` of the class label `y`.  Code up your classifier function.  Measure the accuracy of the classifier on the data.  What percentage error does your classifier get?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Simple Classification ideas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some ideas for classification that you may have already thought of.  We can see that the benign samples are located in the lower left.  So, one simple classification rule is to define the region with a line.  We plot the region for one possible line with the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.patches.Polygon at 0x10b7ce588>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAERCAYAAACQIWsgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VNX5+PHPkwVCIBB2iEECCgRZ\nlUUUFZS6VFFSFKxYpVWLWrXULvq1/bUubalbXVqr1lYEFUHZEtwQEBAta9jXsAYIa0jYISHJPL8/\n7hBDJiEzWeZOkuf9euWVzJ1zz3kyyZxn7rn3niOqijHGGFNUmNsBGGOMCT2WHIwxxviw5GCMMcaH\nJQdjjDE+LDkYY4zxYcnBGGOMj6AmBxEZKyIHRWRdCc/9VkRURJoFMyZjjDG+gn3kMA64qfhGEWkD\nXA/sCnI8xhhjShDU5KCqC4DsEp56FXgCsDvyjDEmBLh+zkFEbgP2qOpqt2MxxhjjiHCzcRGJBv4A\n3OBn+VHAKID69ev3SkxMrMLojDGm5lm+fPkhVW1eVjlXkwNwEdAOWC0iAPHAChHpq6r7ixdW1XeA\ndwB69+6tqampwYzVGGOqPRHZ6U85V5ODqq4FWpx9LCLpQG9VPeRaUMYYY4J+KetEYBHQSUQyROT+\nYLZvjDHGP0E9clDVu8p4PiFIoRhjjDkPt885GGNMheTl5ZGRkUFOTo5f5fef2s/RM0fJ9+RTJ7wO\nzaOaE1s3toqjDL6oqCji4+OJjIws1/6WHIwx1VpGRgYxMTEkJCTgvbDFR25+LqsPrGbZnmVknsr8\n/ok8kByhXeN29InrQ6dmnQgT16/wrzBVJSsri4yMDNq1a1euOiw5GGOqtZycnPMmht1HdzNp3SRO\n5p0s8XlF2X54O9sPbycuJo4R3UbQoE6Dqgy5yokITZs2JTMzs+zCpaj+KdIYU+uVlhh2Hd3F+6vf\nLzUxFLf3+F7GrhzLyTP+lQ9lpb0m/rLkYIypkY7lHmPi2onkefIC2i/7dDYT100MaJ/09HS6du0a\n0D4lSU1N5Ze//GWF66kMNqxkjKmRUvemcjr/dLn2zTiWwfbD22nfuH0lR3V+vXv3pnfv3kFtszR2\n5GCMqXEKPAWs2LeiQnUs27MsoPL5+fmMHDmS7t27c8cdd3Dq1CmWL1/OgAED6NWrFzfeeCP79u0D\nYODAgTz55JP07duXjh078u233wIwf/58Bg8eDEBmZibXX389l112GQ8++CBt27bl0KFDpKen07lz\nZ37+85/TpUsXbrjhBk6fLl8SPB9LDsaYGmfToU2cOHOiQnWkZaVxPPe4/+XT0hg1ahRr1qyhYcOG\n/Otf/+Kxxx5jypQpLF++nPvuu48//OEPheXz8/NZunQpr732Gs8++6xPfc8++yzXXXcdK1as4Ec/\n+hG7dn2/osGWLVt45JFHWL9+PbGxsUydOrVCv2tJbFjJGFPjnHO5ajl51EPW6Sxi6sb4Vb5Nmzb0\n798fgJ/85CeMGTOGdevWcf311wNQUFBA69atC8sPHToUgF69epGenu5T33fffcf06dMBuOmmm2jc\nuHHhc+3ataNnz57n3b+iLDkYY2qc3PzcoNdT/OqgmJgYunTpwqJFi0osX7duXQDCw8PJz8/3eV61\n9OVtzu57dn8bVjLGGD/UCa8T9Hp27dpVmAgmTpxIv379yMzMLNyWl5fH+vXr/a7vqquu4pNPPgFg\n1qxZHD58OIDIK86SgzGmxmkWXfGl6AWhaXRTv8t37tyZ8ePH0717d7KzswvPNzz55JP06NGDnj17\nsnDhQr/re/rpp5k1axaXXXYZX375Ja1btyYmxr8hrsog5zt0CWW2noMxBmDjxo107tz5nG35nnxe\nWfQKp/JOlbvejk07MqLbiIqGV265ubmEh4cTERHBokWLePjhh1m1alVAdZT02ojIclUt83pZO+dg\njKlxIsIi6NmqJwt3+/9Jvbg+cX0qMaLA7dq1i+HDh+PxeKhTpw7/+c9/gtq+JQdjTI3U94K+LN+7\nnNyCwE9Ot2rQioubXFwFUfmvQ4cOrFy50rX27ZyDMaZGio2KZViXYYRLeED7NazbkLu63lXhuYmq\nO0sOxpga6+ImFzOi2wjqhtctuzDQPLo59116H42iGlVxZKHPhpWMMTXaRU0u4rHLH2PFvhUs37uc\no7lHfcrExcTRJ64PXVt0JTK8fIvj1DSWHIwxNV6DOg24pu01XHXhVWzL3saRnCOcKThDVEQUrWNa\nExcT53aIIceGlYwxtUaYhNGhaQf6XNCH/hf2p1dcL9cTQ9HJ9mbMmMHzzz/vajxn2ZGDMcaEiNtu\nu43bbrvN7TAAO3IwxpgKS09PJzExkQceeICuXbty9913M2fOHPr370+HDh1YunQpS5cu5corr+TS\nSy/lyiuvJC0tzaeecePG8eijjwKwbds2+vXrR58+ffjTn/5EgwbO0qXz589n4MCB3HHHHSQmJnL3\n3Xefdx6m8gpqchCRsSJyUETWFdn2kohsEpE1IjJdRGKDGZMxppbZvx/efRf+9jeYPBkqadK6rVu3\nMnr0aNasWcOmTZv46KOP+O6773j55ZcZM2YMiYmJLFiwgJUrV/Lcc8/x+9///rz1jR49mtGjR7Ns\n2TLi4s4d+lq5ciWvvfYaGzZsYPv27fzvf/+rlN+hqGAfOYwDbiq2bTbQVVW7A5uBp4IckzGmtlCF\nTz6B3bshNxfWr4dZsyql6nbt2tGtWzfCwsLo0qULgwYNQkTo1q0b6enpHD16lGHDhtG1a1cef/zx\nMifhW7RoEcOGDQNgxIhzp/Ho27cv8fHxhIWF0bNnzyqZsjuoyUFVFwDZxbbNUtWz89UuBuKDGZMx\nphY5fhyys8/dtnNnpVRddBrtsLCwwsdhYWHk5+fzxz/+kWuvvZZ169bx6aefkpOTUyltlTbld0WF\n2jmH+4Av3Q7CGFNDNWgAjYrd4HbBBUFp+ujRo1zgbWvcuHFllu/Xr1/hCm+TJk2qytBKFDLJQUT+\nAOQDE85TZpSIpIpIamZmxVd6MsbUMmFhcMcd0Ly58/iii+DGG4PS9BNPPMFTTz1F//79KSgoKLP8\na6+9xiuvvELfvn3Zt28fjYontSoW9Cm7RSQB+ExVuxbZNhJ4CBikqn7NsWtTdhtjoORpqf3i8TjJ\nIkSdOnWKevXqISJMmjSJiRMnkpKSElAd1XrKbhG5CXgSGOBvYjDGmAoL4cQAsHz5ch599FFUldjY\nWMaOHRvU9oOaHERkIjAQaCYiGcDTOFcn1QVme2dBXKyqDwUzLmOMCTVXX301q1evdq39oCYHVb2r\nhM3vBjMGY4wxZQvt4ypjjPFDdV3uuCpV9DWx5GCMqdaioqLIysqyBFGEqpKVlUVUVFS563D9hLQx\nxlREfHw8GRkZ2OXt54qKiiI+vvz3FFtyMMZUa5GRkbRr187tMGocG1Yyxhjjw5KDMcYYH5YcjDHG\n+LDkYIwxxoclB2OMMT4sORhjjPFhycEYY4wPSw7GGGN8WHIwxhjjw5KDMcYYH5YcjDHG+LDkYIwx\nxoclB2OMMT4sORhjjPFhycEYY4wPSw7GGGN8WHIwxhjjw5KDMcYYH0FNDiIyVkQOisi6ItuaiMhs\nEdni/d44mDEZY0zIUoV9+2DPHvB4gtp0sNeQHge8AbxfZNv/AV+r6vMi8n/ex08GOS5jTKjJz4eI\nWrzM/Z49MG0aZGU5jxs1giFDoH37oDQf1CMHVV0AZBfbPAQY7/15PJAUlGD273cyspvy8+HkSXdj\n2LwZvvoKduxwNw4TOvbuhWXL4Phxd9o/dQrefRf+8hd4/XU4eNCdONyUmwsTJnyfGACOHoWJE4P2\ndwmFcw4tVXUfgPd7iypvceNG+Pe/na9168ouXxXy8uDtt+Hll2H1andiyMhw/tkWLYIPP4RDh9yJ\n4/RpmDoVtmxxp32AEyfgzTfh/fehoMCdGFTho4/gjTfgyBF3Yjh40OmYP/8cxo4N+lAGAHPmwO7d\nzs+HD0NKSvBjOGvbNnf+Fhs2OEmyuLy8oPUXoZAc/CYio0QkVURSMzMzy1/R3r3OGxGcQzc3nDzp\ndMaqsGuXOzFkZX3/OhQUQHbxg7ogycyEtWudN4RbDh1yOsYdOyAnx50Y8vNh61YnlgMH3Inh0KHv\nk+Phw05nFGzHjp3/cTB9843zNwm2840oBGm0IRQG9A6ISGtV3ScirYFSjyFV9R3gHYDevXtruVvs\n08f51OzxwOWXl7uaComNhZtuchLVNde4E0PHjtCihdMpxsdDu3buxHHhhfDAA9C8uTvtAyQkwG23\nQf36zpcbIiNhxAinU+7Y0Z0YOnRw/g927YIrroC6dYMfwyWXnNshX3JJ8GM467773Gn3wgtLf65t\n26CEIKrl72PL1aBIAvCZqnb1Pn4JyCpyQrqJqj5RVj29e/fW1NTUKo21VigocIZUYmIgrFodSJqa\nbP165yiueXPo2xdE3I4o+D7+2BkCLyohAe69t0LvVRFZrqq9yywXzOQgIhOBgUAz4ADwNJAMfAJc\nCOwChqlqmeMblhyMMTWaxwOpqU6i9Higc2dn1CMyskLV+pscgjqspKp3lfLUoGDGYYwxIS8szDlq\n6tvXneZdadUYY0xIs+RgjDHGhyUHY4wxPiw5GGOM8WHJwRhjjA9LDsYYY3xYcjDGGOPDkoMxxhgf\nlhyMMcb4sORQ26meO2e8McZgycEsXQr//CesWeN2JK5b+tYfWZv8jtthuG/vXmetkfx8tyMxAAsW\nwMyZQW82FKbsNm5q3RratHGm7q7lml16JfUaNXM7DPc1agQ9e0J4uNuRGHCm6HZhXY2gT9ldWWxW\nVmOMCZy/s7LasJIxxhgflhyMMcb4sORgjDHGhyUHY4wxPiw5GGOM8WHJwRhjjA9LDsYYY3xYcjDG\nGOPDkoMxxhgfIZMcRORxEVkvIutEZKKIRLkdkzHG1FYhkRxE5ALgl0BvVe0KhAM/djcqY4ypvUIi\nOXhFAPVEJAKIBva6HI8JluxsZ+bJTZvcjsSY0OPxQEFB0JsNKDmIiEdECkr5yheRLBGZLSI3BFKv\nqu4BXgZ2AfuAo6o6K5A6ApW1K41D6RursomyffWVMzXyoUPuxeDxwObNzroObjhzBsaOhblzYdIk\n2LDBnThOnXLi+OQT5zVxy5o1MGeOe38PgEWL4F//gp073YvhxAmYMQP27XMvBoDdu51Y3HLkCLzy\nCrz0Ehw4ENSmAz1y+DOwG8gExgEvAOO9jzOAD4DmwJciMtjfSkWkMTAEaAfEAfVF5CcllBslIqki\nkpqZmRlg6Ofa9vmHbP10XIXqqLCVK2H/ftixw70YsrKcDtGtN8CJE+e27VZnsH8/7NrlJKdTp9yJ\nAWDxYvjf/+DkSfdiWL0aMjMhLc29GHbsgBUrYP1692IA+PJLWLfOvfb37nXeHzk5zv9nEAU0ZbeI\nPAXcAPxQVXOKbK8HfAnMBv4GfA40UtUr/ax3GHCTqt7vfXwv0E9Vf1HaPhWdslu9nw4lzMWRta1b\nISMD+veHyEj34vB4wK3XQRUmTHBei3r14Gc/c2dtCVXnE3P9+tCjR/DbP+vwYaczaNPGvRj27HGG\n+Pr1c14PN6jCli2QkAB16rgTA7j73gBnwaUvvnDWc7jlFoiq+HU6/k7ZHWhy2Ak8pqozSnhuCPCG\nqrYRkduB8arawM96LwfGAn2A0zhHJamq+s/S9rH1HGoQj8f5pNqwoZMgjDFVxt/kEOhKcC2A0j7i\n1gGaen8+BIi/larqEhGZAqwA8oGVgK3XWFuEhUHLlm5HYYwpItDkkAo8IyILVbVwcFhE4oCnvc8D\ntCXAq41U9WlvHcYYY1wWaHIYDXwN7BCRRcBBnKOJK4BTwNmTyBcDH1VWkMYYY4IroOSgqitE5GLg\nN8DlQDecS0//Dryiqlnecn+q7ECNMcYEj9/JQUTCga7AXlX9fdWFZIwxxm2BXKOlOOcULq2iWIwx\nxoQIv5ODqnpwboBz6cJnY4wxwRLo3R3/Bn4lIi7elWKMMaaqBXq1UgxwEbBdRGbinIwuehedei9J\nNcYYU40FmhyKnoi+r4TnFbtXwRhjqr1AL2UNpSm+jTHGVBHr7I27M4CCMwvqN9+4OzstOLNeBnla\nZB8FBZCb624MJrTs3+9M0BlktTY5rJn6Jqsmve5uEFu3OtNlu2n5cmeueDenRp45E+bNc2ZnPXPG\nnRj27HHWc/jPf+D0aXdiAPjoI3j1VXdj2L8fxo1zZYGZQmfOwHvvubvWSSg4ftz5n3z3XTh4MKhN\nB3rOAREZBTwMdALqFn9eVcMrIa4qFxN/Eepx8Z8fnFlIL7zQ3RhiY6F5c2jUyL0YGjZ0vkdHQ7hL\n/z7R0VC3rvM9IuC3ReVp1crplN2cwr1+fWeqbDenqo6IcGKo7bP0RkY6f4+8vEqZrjsQgU7ZfS/w\nH5wFfh7AmWY7ErgNZ8GfCar6bBXE6cOm7K5BVJ1Vx5o3d2/9AHDegGFh7iUoY0qSn++8RyrpA4O/\nU3YH+tHgVziL+Tzsffymqo4E2uOsw5AVYH3GgIjzKdHNxADOm88Sgwk1ERGuHEkGmhw6AAsAj/er\nDoCqHgb+ijNrqzHGmGou0ORwGghTZyxqP84Rw1kncNZ/NsYYU80FeuZtLc5aDXOAb4Hfi8gOnNXb\nngE2VWp0xhhjXBFocniH748W/oiTJL7DWRL0GJBUeaEZY4xxS6B3SH9c5OetItIFZxW4aGChqtby\ni5KNMaZmKNcF3SLSBmgDROGcmD4BdBcRVHVuJcZnjDHGBQElBxFpD0wA+p7dxPezsp792a4FNMaY\nai7QI4f/Ahfi3O+wCXBprgNjjDFVKdDk0Af4qapOrexARCQWJ/l0xTkCuU9VF1V2O8YYY8oWaHLI\noOqOFl4HZqrqHd6V5qKrqB1jjDFlCPQmuDHAkyJSqfMciEhD4BrgXQBVPaOqRyqzDVOKggLIzgaP\nx+1IjDEhJNBLWT8QkUQgXUQWA4d9i+jIcsTRHmfivvdEpAewHBitqlW30MDRo85kVrGxVdbEeRUU\nwIcfOlNFDxsGHToEPwZVGD/eWcegY0cYMSL4MZy1dy+0aOHejKi5uTBtmjO/0623OvM9ueHLLyEr\nC4YOdWaIra2WLIHZs6FHD+fvUVsdP+5MXZ6fD/feC82aBa3pgI4cROSnwFNALHAZcHUJX+UR4a3v\nLVW9FDgJ/F8J7Y8SkVQRSc3MzCxnU17z58PXX1esjoo4etRZ3ObMGdiwwZ0YTp92EgPAli3uHj2M\nHw+bN7vXfkYGpKXBihVw4oQ7MZw543SKW7dCero7MYSKNWucDnH1arcjcdeuXc6R/bFjsH17UJsO\n9GPas8B04P5KHvbJADJUdYn38RRKSA6q+g7OXdr07t3b/7nGS3LLLRXavcKaNIF+/ZxPzP36uRND\ndDT06gVr10KfPu7O3//440Gfr/4c7drBNdc4Rw4xMe7EUKcO3Habc+TQsaM7MYSKAQOc1QG7dXM7\nEnd16ACXXOJMJ9+lS1CbDnQ9hxPAEFWt9I/cIvIt8ICqponIM0B9Vf1daeVtPQdjjAmcv+s5BHrk\n8B3QGaiK8ZjHgAneK5W2Az+rgjaMMcb4IdDkMBr4REQOAzPxPSGNqpZr4FpVVwFlZjNjjDFVL9Dk\nsNH7/f1Sntdy1GmMMSbEBNqRP8f3cykZY4ypoQK9z+GZKorDGGNMCHHx2kVjjDGhypKDMcYYH5Yc\njDHG+LDkYIwxxoclB2OMMT4sORhjjPFhycFNJ086s4G6aetWmDEDdu50N45du5xZYmszVfjuO2fa\n7pNVN1t9tbBnj/N/uXFj2WVrshMn4OOP4aOP4LDPhBRVqvYmh6VLYZHLq5C++y7897+wfr077efm\nwqRJzjTVH33kdE5uSE2FsWOd9S3cNH++u1NEr1sHc+Y403bPmuVeHJmZzv9FQYF7MZz9v5w82b0p\n1EPB3LlOgty8GWbODGrTtXeqi+ho91c/i4pyFpWpW9ed9sPCnMV18vMhMtKdGOD7qbrdnLIboEED\nqFfPvfbr1Cn552CLjHQWwXJzCvezv394uLtxuK3o+zLI79GApuwOJTViyu78fDh1Cho2dC+Gffuc\nhX46d4bmzd2L4+hRp3MOD3cvhlCwdq2z+lefPu4mbLdlZztHUu3bQ3y829G4JzfXOaLNz4drr62U\n1QH9nbLbkoMxxtQi/iaHWny8ZowxpjSWHIwxxviw5GCMMcaHJQdjjDE+LDkYY4zxYcnBGGOMD0sO\nxhhjfFhyMMYY48OSgzHGGB8hlRxEJFxEVorIZ27HYowxtVmoTbw3GtgIVP1kQ5mZziykLVpUeVPG\nGFMuJ04407gXFED//s6EiEESMkcOIhIP3AL8t8obW7WKzJee5eBLzzjTRbskf+F3nHj/v85Eay45\nnnucRctTOJ3n4loKaWmc/NtzFEyb5l4MQNqcj0lfOtvVGNixw9X/ScD50JSZ6W4M8P0HODcdOQJn\nzrjX/gcfwOLFsGwZjBvnTMAXJCGTHIDXgCeAUufRFpFRIpIqIqmZFfnn3bSJTZkb2XRoE6Sllb+e\nCto/YyJrv5kMGza4FsPaNXPI/eerrN/m4toWK1eyYvM3ZC/62rUFf86cPsG+CW+z86M3XWm/0MyZ\n8Nlnrn5gYPt2ePvtoHZEPnJy4K23YPdu92IAZ52TZcvcafvMGThw4PvHR44EdW2LkBhWEpHBwEFV\nXS4iA0srp6rvAO+AMytruRvs2pUuq7uhHg907Vruaioqbvh9NN222dUYeva8kQ1PNKTbxf1di4G+\nfem9P4Oojpe4tp5CnXoNaPPAb6gb09iV9gvdeqvTCcTEuBfDRRfB6NHOWh9uiYpyYmjUyL0YAEaO\ndG+dkTp14IILnFXxAJo2Der/RUhM2S0ifwPuAfKBKJxzDtNU9Sel7VPhKbuPHXMOWd3+5zPGmNKc\nPu0MK3k8cPnlzponFVRt13PwHjn8VlUHn6+credgjDGBs/UcjDHGlFvIJQdVnV/WUQPAuoPreHL2\nkyzavQiPurwWtDHG1DAhcUK6PNo3bk+d8DqM+mwUh04d4raOtzEkcQjXtbuOqAiXF6o3xphqLuTO\nOfir6DmHrdlbSdmUQnJaMmsPrGVQu0Hcfsnt3NzhZmKjgnfTiDHGhLpqe0LaX6WdkJ6+cTr3Jt/L\npa0uZdX+VVwefzlJnZIYkjiE+IbxLkRqjDGhw9/kUG2HlUoyP30+oz4bxad3fcrAhIGcOHOCWdtm\nkZKWwp/m/4n2jdszpNMQkhKT6NK8CyLidsjGGBOSasyRw/z0+QybPIzJwyYzMGGgT/l8Tz7f7vyW\n5E3JpKSlICLc2vFWhncZzhXxVxAeFh7E6I0xxh216sihrMQAEBEWwbXtruXadteSlJjE0E+Gcjz3\nOI9+8Sh7j+/l1o63kpSYxA/a/4B6ke7cpWuMMaGi2icHfxJD8fLDpwxn+p3TC8unH0knZVMKryx+\nhZ9M/wmD2g0iKTGJWzrcQtPoplX7CxhjTAiq1sNKL095OeDEMGzyMF4Y9AIigqL0at2LHq16FJY5\ndOoQn2/+nJS0FGZvn01i00Tu7n43SYlJJMQmVN0vZIwxQVDjh5WOnzlersTw4g9e5FD6BjouTEOA\n2VdsQK9UerbqCUCz6GaM7DmStrFtWbBzAUmJSaw5sIa/fvtX4hvGk9QpiaTEJLq37F6hE9oe9TBv\n00yOHsqgX8/BxMXElbuuCvF44OBBaNkS3DpBn54OixZBXBxcc417cRhjClXb5LA9ezuzh80OKDFM\nHjaZ9CPpdFy0mUaZxwDouDCN5QnLC5ND0fJThk8prL/AU8DC3QtJ3pTM0E+GUuApICnRSRRXXXgV\nEWGBvZSr9q8i59//osmRk8z+YToj7xwT0P6VZeeEt9g1dxoX3TqSuKH3Bj8AVZg0ibSM1VzQ8AIa\ntGgBnTsHP46CAraP+R11WrQm/sHfBb99ry1ZW9h3Yh/XtL3GtRhCQn4+fPopXHedu5Njzp0LCQnQ\nvr17MYwfD6dOwUMPBfWDU8hNn+Gv9k3aB5wYSiov6l/58LBwrm57NX+/8e9sfWwrn434jGbRzXjk\n80do/EJj7pl+D9M3TufkmZN+xX/OcJ7LQ3tK9RxarHSq4HH/b1Fdh3qrhNuvhdvtu6han3Moa1bW\nkjr6tQfWMvN/4+m0aDOosvmKjvyg/z30bNWzXCe3h00exhs/fIPMU5kkb0pm6Z6lzhVRnZIY3HEw\nzes3L3Ffj3qYt/FLjh3aw+WXujisVFDgDCu1auX+sFLr1jBggA0rGVOFau0d0medr6Nfe2AtqXtT\nzzkhXd7EULz84dOH+WLLFySnJTNr2yx6tOxBUmISQzoN4aImF5XztzXGmMpRq5NDZXX0FS2fk5/D\n3B1zeTv1bb7e8TUJjRIY2nkoSYlJXNb6MrtD2xgTdDX+aqXShEpiAIiKiCI6MppFGYuY8eMZ1Ius\nR/KmZO6aehc5+TkM6TSEIYlDGNB2AJHhkf79gsYYEwyqWi2/evXqpcXN2zFPm73YTOftmOfzXEnc\nKu/xeHTDwQ06ZsEY7fufvtr4+cZ699S7dfL6yXos55hfdRtjTHkAqepHH+t6J1/er+LJobokhpLs\nObZHH5/5uDZ5vok2GNNAb55ws76T+o7uO77Pr7aMMcZf/iaHanspa1GhNJRUnvKbszbzwZoPmHrn\nVDIez+Ce7vcwN30unf/Vmf5j+/Pi/15kc9bmMusxxphK408GCcWvs0cOoXQEUNnlc/JydOaWmfrQ\npw9p65dba+c3OuuTs5/UJRlLtMBT4Ff9xhhTFLVhWKk6dfQVLV/gKdA3l76pMWNitP3r7TXu73H6\n0KcP6cwtMzU3P9ev9owxpsYnh47dOoZMx+1G+bRDafridy/qle9eqbHPx+qPp/xYJ66dqEdOH/Gr\nPmNM7VTjk0PEBREh23EHu/y+4/v0ndR39OYJN2vMmBgdNH6Qvrn0Td1zbI9f9Rtjao9qlRyANsA8\nYCOwHhhd1j4du3X064Uo3rHmF+Trdzu/07eXva1vL3tbv935reYV5JVaPtD63S7/+ebPNWZMjN7w\nwQ3a+PnG2vc/fXXMgjG64eCoN42bAAAT1ElEQVQG9Xg8ftVhjKm5/E0OoXITXD7wG1VdISIxwHIR\nma2qG0rbIaZOTJmVlnTV0NSNU9mQuYG6J3NBlTkn9rH76G7u6nZX0K9iSjuUxtHco1zS/BIa1GlQ\nKfWPTB7JjLtmMDBhIHkFeXyz8xtSNqVww4c3EB0ZXbiGdr/4foSJc7GaRz2FPxtjDITo9BkikgK8\noaqzSytTnrmVDpw4wFupb9F8x0EuWbARUDZd1ZkDF7Wka4uuPPz5w0FLDIt2L+KrbV8B0DiqMQ/3\neZg64XUqrf7iVJUV+1YUrqG9/8R+brzoRuIbxhMeFk6/+H4M7ji4zHaqyvFDe2nQpBUS5mKSOn0a\nIiIg0u5WNzVXtZ0+Q0QSgEuBJeWto7SO8sDJAwC03HEQ8SbFFtsPsKTpaV5f8vo5S4eWp/5Aym/N\n3lr4/OGcwxw6dahwZtaqOIIREXrF9aJXXC8GtR/E0I+HEhURxUfrPuLAiQNc1PgiDuccZnCHwTSu\n17jMNivb8j/8jPh7HuHiq24LetuFpkyBFi3gxhvdi8GYEBFSyUFEGgBTgV+p6rESnh8FjAK48MIL\nS6zjfB1ls+hmABxq05Smuw8hCgtbnmHyhsm8d9t7Qb0hrm1sW7Yd3gY4Q2RN6jWp1PrLKj/tzmkM\naDuACWsnsHr/ak7lnWLKhik88vkj9L2gb+FMsm0atSmzzsrQ7Q//ILZ1u6C0VaohQ+yowZiz/Dkx\nEYwvIBL4Cvi1P+XLO7fSB6s/0KfnPa0vTP+tPvj+cI3+a7Q+OfvJss7h+F2/v+U9Ho+u2rdK5++Y\nr9mnsiu9fn/LezwePXnmZOHjE7kndNqGaTpy+kiNfT5WO/6zoz43/zlds3+NndA2pgagml2tJMD7\nwGv+7lOeuZVO5J7Q9QfW6ysLX9HBEwZr1F+i9KFPH9K1B9bq8dzj531BQ+2qpGCUb/pCU3110as6\n+svR2vbVttr+9fb665m/1gXpCzS/IN+veowxocXf5BASJ6RF5CrgW2At4PFu/r2qflHaPkVPSJ9v\naEVV2ZK9haV7lrItexuKkn4knckbJjPskmEkxCY4MSC0b9yePhf0oVPTTuestRBqczG5UV5VWXNg\nDcmbkklOSybjWAa3dryVpMQkrm9/PfUi65VZrzHGfbVmsZ/zdXxHco6QsimFHUd2FG4rKTEU17ZR\nW4YkDqFJvSbVouN2o3z6kXRmpM3ggzUfsO7gOm5ofwNDOw9lcMfBNI1uWmY7xhh3VNurlQJxvo5s\nc9ZmpmyYwpmCM4Xb/EkMADuP7uStZW8R3zCe387+resdcSiWT4hNoHvL7qQfSWfS7ZM4mnuU5E3J\n/HLmL7ms9WUkdUpiSOKQ877OxpjQVW2PHDp176TZ92SX2JGlHUrj4/Uf41FP4TZ/E0Px8v+6+V/8\ntOdPyywfSh23m+VP551mzvY5JG9K5tPNnxIXE0dSYhJJiUn0aNnDlkY1xmU1flgpMj5SZ38326cj\nyz6dzdupb5friKGk8hc3uZiHej9UeBlsSapLxx3s8gWeAhZlLOKNpW/wxZYvaBTViKGJzhraV7e9\nmoiwan3gaky15G9yqLZzJrRv0r7Ek88pm1IqLTEkxCaQ78kneVMypSXRUOmIQ7F8eFg4+Z58vt7x\nNSk/TuHLu7+kef3mPDHnCVq93IqRySOZtnEaJ8+cLLNdY0xwVdsjh5Kmz9iQuYFP1n9S+LiiiaGo\n2zvfTreW3c7ZFkodcXUrv/vobmakzSA5LZklGUsYmDCQpMQkbu14K83rNy+zbmNM+dT4I4eSLNuz\nrPDnykwMAEv3LD3ncXXqiEOxfJtGbXik7yPMvmc2ux7fRfeW3fntrN9y0T8u4ur3rubvC/9+zhQj\nxpjgqjGDvkdzjhZeslrZiQFg97HdZJ3Koml000rrKLNPZ3PyzElax7Q+Z/w9mB1377je7Dq6i2bR\nzYiOjK70+v0pv2r/Kv69/N9Mu3Ma/eL7MW/HPJI3JXPV2KtoXr954UyyvVr3shPaxgRJjRlWWn9w\nPZM3TK6SxHDW0M5DyT6dXeGO8nTeaaZsmFI4t1J0ZDQ3d7iZri26Bi0xfHLHJ5zOP03q3lQ86iEi\nLIIr4q9gUPtBlVJ/ZZT3qIclGUsKb7w7lXeKWzveytDOQxnQdgCR4TYPkjGBqvFXKxVPDnO2z+HD\nNR9WWWIApxN/aeFLFe74pm6YytqDa88pGyZh9GzVk/tn3B+Ujjg6MpovtvjegF703IrbQ0/Fvb/6\nfR75/BHaNW5HxrEMftjhhyR1SuKmi28ipm7Z63sYY2rhOYfFGYsD6ujl67lMXzUxoETy12//WuGO\nL9+Tz4ZM3zWMth/ezr3T7w1aR7x6/2oAInLz6LB4C+F5BQCsObCmUuoPpHyPlj14ffHrLM5YfN7y\nv5n1Gz4d8SlrHl7Dul+s45oLr2HsqrFc8MoF3PLRLbyz/B32n9hfZtuhbO7X7zLtvSfIK8hzOxQT\nIjZmbmTV/lVBb7dGnHOYnz6flxa+FFBHPz1sCU8d70lOAEcYT1z5RIU7SlU95+a8ovX/+do/B+0T\neoE6ySCswEPUiZzC9S3yPflBP2I4ePIgx88cL/WS1pLqj4uJ48HeD/Jg7wc5lnuML7d8SUpaCk/O\neZIOTTowtPNQfpT4Izo161RmPKHk9PFsCo4cpkALiMSGzQyczDtJTn5O0Nut9sNKZzuOJ658gpN5\nZV8vXziUlHgH7WIT0LDzn+AsOvQ0vMtwbu5w83nL+9NRTlgzgS3ZW3zqf/4Hz9OqQasK1+9P+QU7\nFzB3x1yf8q3qt+KP8/8Y9KGkfE9+iTfFBVr/7G2zuWPyHVzX7jqW7llKw7oNSerk3KHd54I+Ib8c\n6tkPD+Fh4W6HYmqoWjGsVLTjuPHislfvOuccQ5N2ASWGhNgEWjdo7Xc85+vIbul4C82imxXWP/yS\n4fz8sp8HLTEAXNnmSjo06XDONkFcSQxApSSG+enzGTFtBCk/TmH6ndPZ/fhuxieNJ0zCuG/GfcS/\nEs9Dnz3EzK0zyc3PLbM+N4iIJQYTEqrtkUPxuZWyTmXxz6X/LLV8ZVzF9HDvh2nZoGWJ5QPtyObt\nmMftn9zOS9e/xPAuw8s8oVpVQz17j+8l82Qm2w5v48HPHgyZk89VUX5z1mZSNqWQkpbC2gNruSbh\nGkZ0HcHNHW6mUVSjMtswpiao8UcO27O3n9MRNI1uWrgGc3GVkRha1m9ZaYlhfvp8hk8ZzrQ7p3H/\nZfe7lhjAGbs/nHO4xicGgI5NO/K7/r/jL9f9hYjwCBKbJjJh7QTavNqGGz+8kbeWvcWeY3vKbM+Y\n2qDanpAuaW6lPnF9SElLOWdbZd330Duu5EQbKh2flQ+s/NThUwvLnzhzgq+2fkVyWjJ/mPsHLm5y\nceFMsp2bdbYb70ytVG2TQ0wd30/b3Vt2Z1HGIg6ePAhUXmJoFt2MS1tf6lM+VDs+Kx9Y+QZ1GnD7\nJbdz+yW3k1eQx4KdC0hJS2HQ+4MIkzDu6noXP0r8Ef3i+9n5AFNrVNthpZKEh4WTlJhEmIRVWmIQ\nhKTEJJ8TptWl47PygZWPDI9kUPtBDO08lLyCPJ4e8DTRkdH84otfEPdKHD+f8XM+3/y5K5cWGhNM\n1fbIoTRxMXG0rN+SF/73QqXcKX3TxTcR3zD+nG2h0pFZ+aotP2X4lMLyz137HNsPbydlUwovLnyR\nEdNGcH3760lKTOKWDrfQuF7jMus1pjqptlcrlTRlN3z/xv7zwD9z4OQBlPP/fuc7Yrjx4hvpF9+v\nxPpDrSOz8sEtf+jUIT7b/BnvrXqPJRlL6Bffj9s7386QxCFc2OjCMtsxxi21Yg3p4oq/sTOOZZC8\nKZlDpw6VWL60xNCkXhOSEpN83uSh0jFZeffLN4tuRkJsAhsyNzD9zunkFuSSvCmZZ795lraxbQtv\nvOvaoqud0DbVk6pWy69evXppUfN2zNNmLzbTeTvmnbM9ryBPF+9erG8seUOfnvd04dfI6SM1+q/R\nOnL6yMJt/1j8D120e5GeyT+jxZVWf2msfO0sn1eQp/N3zNdfffkrTXgtQdu91k4fn/m4fpP+jeYV\n5PlVtzFVCUhVP/pY1zv5wkDgJiAN2Ar8X1nliyYHf9/YGUczdEnGEv3zN3/WmDEx+v++/n+avDFZ\nl2Qs0d1Hd6vH4ylxv+rSMVn50Crv8Xh09f7V+rPkn2ns87Ha5IUm+tPkn2ryxmQ9eeakX20ZU9mq\nVXIAwoFtQHugDrAauOR8+5xNDqHSEVh5K19W+fTD6fr64tf1uvHXacO/NdSkSUn63sr3NPNkpl91\nGVMZqltyuAL4qsjjp4CnzrdPr169QrojsPJW/nzls05l6fur3tehHw/Vhn9rqAPeG6CvLHxFt2dv\n96tuY8rL3+QQElcricgdwE2q+oD38T3A5ar6aGn7RMZHat1f1CU6MtqvG5MKPAWcyjtl5a18yJXP\nyc/hSM6RwsdLHlhC3wv6ltmGMeVRrVaCE5FhwI3FkkNfVX2sWLlRwCjvw0445yiqu2ZAyZdT1T72\nWjjsdfievRaOynwd2qpq87IKhcqlrBlAmyKP44G9xQup6jvAO8EKKhhEJNWfLF4b2GvhsNfhe/Za\nONx4HUJl+oxlQAcRaScidYAfAzNcjskYY2qtkDhyUNV8EXkU+ArnyqWxqrre5bCMMabWConkAKCq\nXwBfuB2HC2rUMFkF2WvhsNfhe/ZaOIL+OoTECWljjDGhJVTOORhjjAkhlhxcIiJtRGSeiGwUkfUi\nMtrtmNwkIuEislJEPnM7FjeJSKyITBGRTd7/jSvcjskNIvK4932xTkQmikiU2zEFi4iMFZGDIrKu\nyLYmIjJbRLZ4v1f5HPGWHNyTD/xGVTsD/YBHROQSl2Ny02hgo9tBhIDXgZmqmgj0oBa+JiJyAfBL\noLeqdsW5SOXH7kYVVONw5por6v+Ar1W1A/C193GVsuTgElXdp6orvD8fx+kELnA3KneISDxwC/Bf\nt2Nxk4g0BK4B3gVQ1TOqeuT8e9VYEUA9EYkAoinhvqeaSlUXANnFNg8Bxnt/Hg8kVXUclhxCgIgk\nAJcCS9yNxDWvAU8AHrcDcVl7IBN4zzvE9l8Rqe92UMGmqnuAl4FdwD7gqKrOcjcq17VU1X3gfLAE\nWlR1g5YcXCYiDYCpwK9U9Zjb8QSbiAwGDqrqcrdjCQERwGXAW6p6KXCSIAwfhBrvePoQoB0QB9QX\nkZ+4G1XtY8nBRSISiZMYJqjqNLfjcUl/4DYRSQcmAdeJyIfuhuSaDCBDVc8eQU7BSRa1zQ+AHaqa\nqap5wDTgSpdjctsBEWkN4P1+sKobtOTgEnHWjnwX2Kiqr7gdj1tU9SlVjVfVBJyTjnNVtVZ+SlTV\n/cBuEenk3TQI2OBiSG7ZBfQTkWjv+2QQtfDEfDEzgJHen0cCKVXdYMjcIV0L9QfuAdaKyCrvtt97\n7xQ3tddjwATvHGPbgZ+5HE/QqeoSEZkCrMC5qm8ltehOaRGZCAwEmolIBvA08DzwiYjcj5M8h1V5\nHHaHtDHGmOJsWMkYY4wPSw7GGGN8WHIwxhjjw5KDMcYYH5YcjDHG+LDkYGoNERkoIioiA92OpTKI\nSLqIjCu27VYRWSsiOd7fNdal8Ew1Z5eymlrDO7HdJcCGmjBViYhcChxT1W3exxE4E7YtBMYAZ4Bl\nqlrgXpSmurLkYEwNISJtgXTgflUd63I4ppqzYSVTo4hIRxGZ7l0sJUdEdonIZBGJKD6sJCLPeB+X\n9PXTInW2E5EJIpIpIrkiskpEfhRgXM+IiM8nMREZ551X6uzjBG/7D4rIcyKyT0SOiMin3qnNi+5b\nOKwkIs/gJAaAd711zA8kRmOKsukzTE3zGXAEeBg4hLNGxs2U/EHov8DMYtt+izNX/mZwVuzDmUr9\nIPA4zpTadwJTRSRJVWdUwe8A8BTO8NB9ONMz/x2YAAwopfx/gXXAZOAvwOdAtR86M+6x5GBqDBFp\nBnQAhhTrtD/yPn9OeVXNwJkJ9ez+w4ChONOnL/RufgYQYICqZnm3feVNGs/hTIhWFXaq6ogisTUH\nXhKROFX1WfhGVTOKzNG1TVUXV1FcppawYSVTk2ThTFb3vIj8XEQ6+LujiPTGWWHrTVX9R5GnbgK+\nAI56h6YivCd+vwJ6eE9yV4XPiz1e6/1+YRW1Z8w5LDmYGkOdqyuuB1KBvwGbRWS7iDx8vv28Y/kz\ngPk4a1kX1QK4F8gr9vWS9/mmlRV/McWXicz1fo+qovaMOYcNK5kaRVW3A/d61wHoATwKvOk96Xu6\neHnvMpwzcM5P3FnCZZ9ZwLfAC6U06e/axjne9uqo6pki26squRhTIZYcTI3kPYpYJSK/Bu4HugLL\nipbxJpAPcZai7Kuqx0uoaiZwBbBeVX2SSwB2er93xVmnAO8NalcCJbVrjKssOZgaQ0S6A68DHwNb\ngXDgpzgLxswFYort8iTOlUmjgTgRiSvy3DZVzQT+BCwFFojIGziXizbG6eTbq+p9fob3JXAU+I+I\nPA3UBZ4ATgT2WxoTHJYcTE2yH2eVrF8D8ThDOWuBwaq6vIRpMxK9318voa6fAeNUdZf3ZPUzOHcd\nN8cZalqHcwLbL6p6REQGA68Cn+BcJfUcznrJxeMyxnV2h7QxxhgfdrWSMcYYHzasZEwFee97OJ8C\ntUN0U83YkYMxFSAiCfjeA1H8q7QpL4wJWXbOwZgKEJE6QPcyiqWVcpmsMSHLkoMxxhgfNqxkjDHG\nhyUHY4wxPiw5GGOM8WHJwRhjjA9LDsYYY3z8f6CDiku5Vaj0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10bae4b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import patches\n",
    "plt = plot_cnt(X,y)\n",
    "\n",
    "ax = plt.gca() \n",
    "ax.add_patch(\n",
    "    patches.Polygon([[0,0],[0,4],[6,0]],\n",
    "        color='green',\n",
    "        fill=False, hatch = '//'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an alternate classification rule based on two linear constraints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.patches.Rectangle at 0x10ba75208>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAERCAYAAACQIWsgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8lOW99/HPLwmL7LJZEBGoUBBw\nYynVVm09Vp/WqqdH7Kn22FPb2vq0li4+Ll1E2x7rOfZY7Wmt1ULBUwEVN9w3oGpFIez7FgKENSQQ\nlrBk+T1/3BMMmSwzSWbuWb7v1yuvZGauue4fQ3J9517muszdERERqS0n7AJERCT1KBxERCSKwkFE\nRKIoHEREJIrCQUREoigcREQkSlLDwcwmm9luM1tRz2O3mpmbWc9k1iQiItGSvecwBbi87p1mdhpw\nKbAlyfWIiEg9khoO7v4OUFrPQ78DbgP0iTwRkRQQ+jkHM7sS2ObuS8OuRUREAnlhbtzMOgA/Az4f\nY/ubgJsAOnbsOGro0KEJrE5EJPMsXLhwj7v3aqpdqOEAfBwYCCw1M4B+wCIzG+vuO+s2dvdHgUcB\nRo8e7fn5+cmsVUQk7ZnZ5ljahRoO7r4c6F1z28wKgdHuvie0okREJOmXsk4H5gGfMLMiM/tmMrcv\nIiKxSeqeg7t/tYnHBySpFBERaUTY5xxERFqkoqKCoqIijhw5ElP7neU7KTtWRmV1JW1z29KrfS+6\nteuW4CqTr3379vTr1482bdo06/kKBxFJa0VFRXTu3JkBAwYQubAlytHKoyzdtZQF2xZQXF780QMV\nYEeMgScPZEzfMXyi5yfIsdCv8G8xd6ekpISioiIGDhzYrD4UDiKS1o4cOdJoMGwt28qMFTM4VHGo\n3scdp2BvAQV7C+jbuS/XjbyOTm07JbLkhDMzevToQXFxcdONG5D+ESkiWa+hYNhStoXHlz7eYDDU\ntf3AdiYvnsyhY7G1T2UNvSaxUjiISEbaf3Q/05dPp6K6Iq7nlR4uZfqK6XE9p7CwkBEjRsT1nPrk\n5+fzgx/8oMX9tAYdVhKRjJS/PZ/DlYeb9dyi/UUU7C1g0MmDWrmqxo0ePZrRo0cndZsN0Z6DiGSc\nquoqFu1Y1KI+FmxbEFf7yspKvv71r3PWWWdxzTXXUF5ezsKFC7nooosYNWoUl112GTt27ADg4osv\n5vbbb2fs2LEMGTKEd999F4C5c+dyxRVXAFBcXMyll17Keeedx3e+8x1OP/109uzZQ2FhIcOGDePb\n3/42w4cP5/Of/zyHDzcvBBujcBCRjLNmzxoOHjvYoj7WlqzlwNEDsbdfu5abbrqJZcuW0aVLF/74\nxz9yyy23MHPmTBYuXMiNN97Iz372s+PtKysrmT9/Pg8++CD33HNPVH/33HMPn/vc51i0aBH//M//\nzJYtH61osH79er73ve+xcuVKunXrxjPPPNOif2t9dFhJRDLOCZerNlO1V1NyuITO7TrH1P60007j\nggsuAOBrX/sa9957LytWrODSSy8FoKqqij59+hxv/+UvfxmAUaNGUVhYGNXfe++9x3PPPQfA5Zdf\nzsknn3z8sYEDB3LOOec0+vyWUjiISMY5Wnk06f3UvTqoc+fODB8+nHnz5tXbvl27dgDk5uZSWVkZ\n9bh7w8vb1Dy35vk6rCQiEoO2uW2T3s+WLVuOB8H06dMZN24cxcXFx++rqKhg5cqVMff36U9/mqee\negqAN954g71798ZRecspHEQk4/Ts0PKl6A2jR4ceMbcfNmwYU6dO5ayzzqK0tPT4+Ybbb7+ds88+\nm3POOYf3338/5v4mTpzIG2+8wXnnncerr75Knz596Nw5tkNcrcEa23VJZVrPQUQAVq9ezbBhw064\nr7K6kgfmPUB5RXmz+x3SYwjXjbyupeU129GjR8nNzSUvL4958+Zx8803s2TJkrj6qO+1MbOF7t7k\n9bI65yAiGScvJ49zPnYO72+N/Z16XWP6jmnFiuK3ZcsWrr32Wqqrq2nbti2PPfZYUrevcBCRjDT2\n1LEs3L6Qo1Xxn5z+WKePcUb3MxJQVewGDx7M4sWLQ9u+zjmISEbq1r4b44ePJ9dy43pel3Zd+OqI\nr7Z4bqJ0p3AQkYx1RvczuG7kdbTLbdd0Y6BXh17ceO6NdG3fNcGVpT4dVhKRjPbx7h/nlk/ewqId\ni1i4fSFlR8ui2vTt3JcxfccwovcI2uQ2b3GcTKNwEJGM16ltJy48/UI+3f/TbCzdyL4j+zhWdYz2\nee3p07kPfTv3DbvElKPDSiKSNXIsh8E9BjPm1DFc0P8CRvUdFXow1J5sb9asWdx3332h1lNDew4i\nIiniyiuv5Morrwy7DEB7DiIiLVZYWMjQoUP51re+xYgRI7j++ut56623uOCCCxg8eDDz589n/vz5\nnH/++Zx77rmcf/75rF27NqqfKVOm8P3vfx+AjRs3Mm7cOMaMGcNdd91Fp07B0qVz587l4osv5ppr\nrmHo0KFcf/31jc7D1FxJDQczm2xmu81sRa377jezNWa2zMyeM7NuyaxJRLLMzp0waRL85jfw9NPQ\nSpPWbdiwgQkTJrBs2TLWrFnDtGnTeO+99/jtb3/Lvffey9ChQ3nnnXdYvHgxv/zlL/npT3/aaH8T\nJkxgwoQJLFiwgL59Tzz0tXjxYh588EFWrVpFQUEB//jHP1rl31BbsvccpgCX17nvTWCEu58FrAPu\nTHJNIpIt3OGpp2DrVjh6FFauhDfeaJWuBw4cyMiRI8nJyWH48OFccsklmBkjR46ksLCQsrIyxo8f\nz4gRI/jRj37U5CR88+bNY/z48QBcd92J03iMHTuWfv36kZOTwznnnJOQKbuTGg7u/g5QWue+N9y9\nZr7aD4B+yaxJRLLIgQNQWnrifZs3t0rXtafRzsnJOX47JyeHyspKfvGLX/DZz36WFStW8OKLL3Lk\nyJFW2VZDU363VKqdc7gReDXsIkQkQ3XqBF3rfMDt1FOTsumysjJOjWxrypQpTbYfN27c8RXeZsyY\nkcjS6pUy4WBmPwMqgScaaXOTmeWbWX5xcctXehKRLJOTA9dcA716Bbc//nG47LKkbPq2227jzjvv\n5IILLqCqqqrJ9g8++CAPPPAAY8eOZceOHXStG2oJlvQpu81sAPCSu4+odd/Xge8Cl7h7THPsaspu\nEYH6p6WOSXV1EBYpqry8nJNOOgkzY8aMGUyfPp0XXnghrj7SespuM7scuB24KNZgEBFpsRQOBoCF\nCxfy/e9/H3enW7duTJ48OanbT2o4mNl04GKgp5kVARMJrk5qB7wZmQXxA3f/bjLrEhFJNZ/5zGdY\nunRpaNtPaji4+1fruXtSMmsQEZGmpfZ+lYhIDNJ1ueNEaulronAQkbTWvn17SkpKFBC1uDslJSW0\nb9++2X2EfkJaRKQl+vXrR1FREbq8/UTt27enX7/mf6ZY4SAiaa1NmzYMHDgw7DIyjg4riYhIFIWD\niIhEUTiIiEgUhYOIiERROIiISBSFg4iIRFE4iIhIFIWDiIhEUTiIiEgUhYOIiERROIiISBSFg4iI\nRFE4iIhIFIWDiIhEUTiIiEgUhYOIiERROIiISBSFg4iIRElqOJjZZDPbbWYrat3X3czeNLP1ke8n\nJ7MmEZGU5Q47dsC2bVBdndRNJ3sN6SnAH4DHa913B/C2u99nZndEbt+e5LpEJNVUVkJeFi9zv20b\nPPsslJQEt7t2hauugkGDkrL5pO45uPs7QGmdu68CpkZ+ngpcnZRidu4MEjlMlZVw6FC4NaxbB6+/\nDps2hVuHpI7t22HBAjhwIJztl5fDpEnw61/DQw/B7t3h1BGmo0fhiSc+CgaAsjKYPj1p/y+pcM7h\nFHffARD53jvhW1y9Gv785+BrxYqm2ydCRQU88gj89rewdGk4NRQVBb9s8+bB3/4Ge/aEU8fhw/DM\nM7B+fTjbBzh4EB5+GB5/HKqqwqnBHaZNgz/8AfbtC6eG3buDgfnll2Hy5KQfygDgrbdg69bg5717\n4YUXkl9DjY0bw/m/WLUqCMm6KiqSNl6kQjjEzMxuMrN8M8svLi5ufkfbtwd/iBDsuoXh0KFgMHaH\nLVvCqaGk5KPXoaoKSuvu1CVJcTEsXx78QYRlz55gYNy0CY4cCaeGykrYsCGoZdeucGrYs+ejcNy7\nNxiMkm3//sZvJ9Pf/x78nyRbY0cUknS0IRUO6O0ysz7uvsPM+gAN7kO6+6PAowCjR4/2Zm9xzJjg\nXXN1NXzyk83upkW6dYPLLw+C6sILw6lhyBDo3TsYFPv1g4EDw6mjf3/41regV69wtg8wYABceSV0\n7Bh8haFNG7juumBQHjIknBoGDw5+D7ZsgU99Ctq1S34NZ5554oB85pnJr6HGjTeGs93+/Rt+7PTT\nk1KCuTd/jG3WBs0GAC+5+4jI7fuBklonpLu7+21N9TN69GjPz89PaK1ZoaoqOKTSuTPkpNWOpGSy\nlSuDvbhevWDsWDALu6Lke/LJ4BB4bQMGwA03tOhv1cwWuvvoJtslMxzMbDpwMdAT2AVMBJ4HngL6\nA1uA8e7e5PENhYOIZLTqasjPD4KyuhqGDQuOerRp06JuYw2HpB5WcvevNvDQJcmsQ0Qk5eXkBHtN\nY8eGs/lQtioiIilN4SAiIlEUDiIiEkXhICIiURQOIiISReEgIiJRFA4iIhJF4SAiIlEUDiIiEkXh\nkO3cT5wzXkQEhYPMnw//8z+wbFnYlYRu/p9+wfLnHw27jPBt3x6sNVJZGXYlAvDOO/Daa0nfbCpM\n2S1h6tMHTjstmLo7y/U893xO6toz7DLC17UrnHMO5OaGXYlAMEV3COtqJH3K7taiWVlFROIX66ys\nOqwkIiJRFA4iIhJF4SAiIlEUDiIiEkXhICIiURQOIiISReEgIiJRFA4iIhJF4SAiIlFSJhzM7Edm\nttLMVpjZdDNrH3ZNIiLZKiXCwcxOBX4AjHb3EUAu8K/hViUikr1SIhwi8oCTzCwP6ABsD7keSZbS\n0mDmyTVrwq5EJPVUV0NVVdI3G1c4mFm1mVU18FVpZiVm9qaZfT6eft19G/BbYAuwAyhz9zfi6SNe\nJVvWsqdwdSI30bTXXw+mRt6zJ7waqqth3bpgXYcwHDsGkyfD7NkwYwasWhVOHeXlQR1PPRW8JmFZ\ntgzeeiu8/w+AefPgj3+EzZvDq+HgQZg1C3bsCK8GgK1bg1rCsm8fPPAA3H8/7NqV1E3Hu+fwK2Ar\nUAxMAf4TmBq5XQT8L9ALeNXMroi1UzM7GbgKGAj0BTqa2dfqaXeTmeWbWX5xcXGcpZ9o48t/Y8OL\nU1rUR4stXgw7d8KmTeHVUFISDIhh/QEcPHjitsMaDHbuhC1bgnAqLw+nBoAPPoB//AMOHQqvhqVL\nobgY1q4Nr4ZNm2DRIli5MrwaAF59FVasCG/727cHfx9HjgS/n0kU15TdZnYn8Hng/7j7kVr3nwS8\nCrwJ/AZ4Gejq7ufH2O944HJ3/2bk9g3AOHf/vw09p6VTdnvk3aHlhHhkbcMGKCqCCy6ANm3Cq6O6\nGsJ6HdzhiSeC1+Kkk+Ab3whnbQn34B1zx45w9tnJ336NvXuDweC008KrYdu24BDfuHHB6xEGd1i/\nHgYMgLZtw6kBwv3bgGDBpVdeCdZz+OIXoX3Lr9OJdcrueMNhM3CLu8+q57GrgD+4+2lm9i/AVHfv\nFGO/nwQmA2OAwwR7Jfnu/j8NPUfrOWSQ6urgnWqXLkFAiEjCxBoO8a4E1xto6C1uW6BH5Oc9gMXa\nqbt/aGYzgUVAJbAY0HqN2SInB045JewqRKSWeMMhH7jbzN539+MHh82sLzAx8jjA6cR5tZG7T4z0\nISIiIYs3HCYAbwObzGwesJtgb+JTQDlQcxL5DGBaaxUpIiLJFVc4uPsiMzsD+AnwSWAkwaWn/w08\n4O4lkXZ3tXahIiKSPDGHg5nlAiOA7e7+08SVJCIiYYvnGi0nOKdwboJqERGRFBFzOLh7NcEH4EK6\n8FlERJIl3k93/Bn4oZmF+KkUERFJtHivVuoMfBwoMLPXCE5G1/4UnUcuSRURkTQWbzjUPhF9Yz2P\nO/qsgohI2ov3UtZUmuJbREQSRIO9hDsDKASzoP797+HOTgvBrJdJnhY5SlUVHD0abg2SWnbuDCbo\nTLKsDYdlzzzMkhkPhVvEhg3BdNlhWrgwmCs+zKmRX3sN5swJZmc9diycGrZtC9ZzeOwxOHw4nBoA\npk2D3/0u3Bp27oQpU0JZYOa4Y8fgr38Nd62TVHDgQPA7OWkS7N6d1E3He84BM7sJuBn4BNCu7uPu\nntsKdSVc534fx6tD/OWHYBbS/v3DraFbN+jVC7p2Da+GLl2C7x06QG5Ivz4dOkC7dsH3vLj/LFrP\nxz4WDMphTuHesWMwVXaYU1Xn5QU1ZPssvW3aBP8fFRWtMl13POKdsvsG4DGCBX6+RTDNdhvgSoIF\nf55w93sSUGcUTdmdQdyDVcd69Qpv/QAI/gBzcsILKJH6VFYGfyOt9IYh1im7431r8EOCxXxujtx+\n2N2/DgwiWIehJM7+RMAseJcYZjBA8MenYJBUk5cXyp5kvOEwGHgHqI58tQVw973AfxDM2ioiImku\n3nA4DOR4cCxqJ8EeQ42DBOs/i4hImov3zNtygrUa3gLeBX5qZpsIVm+7G1jTqtWJiEgo4g2HR/lo\nb+EXBCHxHsGSoPuBq1uvNBERCUu8n5B+stbPG8xsOMEqcB2A9909yy9KFhHJDM26oNvMTgNOA9oT\nnJg+CJxlZrj77FasT0REQhBXOJjZIOAJYGzNXXw0K2vNz7oWUEQkzcW75/AXoD/B5x3WACHNdSAi\nIokUbziMAf7d3Z9p7ULMrBtB+Iwg2AO50d3ntfZ2RESkafGGQxGJ21t4CHjN3a+JrDTXIUHbERGR\nJsT7Ibh7gdvNrFXnOTCzLsCFwCQAdz/m7vtacxvSgKoqKC2F6uqwKxGRFBLvpaz/a2ZDgUIz+wDY\nG93Ev96MOgYRTNz3VzM7G1gITHD3xC00UFYWTGbVrVvCNtGoqir429+CqaLHj4fBg5NfgztMnRqs\nYzBkCFx3XfJrqLF9O/TuHd6MqEePwrPPBvM7felLwXxPYXj1VSgpgS9/OZghNlt9+CG8+SacfXbw\n/5GtDhwIpi6vrIQbboCePZO26bj2HMzs34E7gW7AecBn6vlqjrxIf39y93OBQ8Ad9Wz/JjPLN7P8\n4uLiZm4qYu5cePvtlvXREmVlweI2x47BqlXh1HD4cBAMAOvXh7v3MHUqrFsX3vaLimDtWli0CA4e\nDKeGY8eCQXHDBigsDKeGVLFsWTAgLl0adiXh2rIl2LPfvx8KCpK66Xjfpt0DPAd8s5UP+xQBRe7+\nYeT2TOoJB3d/lOBT2owePTr2ucbr88UvtujpLda9O4wbF7xjHjcunBo6dIBRo2D5chgzJtz5+3/0\no6TPV3+CgQPhwguDPYfOncOpoW1buPLKYM9hyJBwakgVF10UrA44cmTYlYRr8GA488xgOvnhw5O6\n6XjXczgIXOXurf6W28zeBb7l7mvN7G6go7v/v4baaz0HEZH4xbqeQ7x7Du8Bw4BEHI+5BXgicqVS\nAfCNBGxDRERiEG84TACeMrO9wGtEn5DG3Zt14NrdlwBNppmIiCRevOGwOvL98QYe92b0KSIiKSbe\ngfyXfDSXkoiIZKh4P+dwd4LqEBGRFBLitYsiIpKqFA4iIhJF4SAiIlEUDiIiEkXhICIiURQOIiIS\nReEQpkOHgtlAw7RhA8yaBZs3h1vHli3BLLHZzB3eey+YtvtQ4marTwvbtgW/l6tXN902kx08CE8+\nCdOmwd6oCSkSKnvDYf58mBfyKqSTJsFf/gIrV4az/aNHYcaMYJrqadOCwSkM+fkweXKwvkWY5s4N\nd4roFSvgrbeCabvfeCO8OoqLg9+Lqqrwaqj5vXz66fCmUE8Fs2cHAbluHbz2WlI3nb1TXXToEP7q\nZ+3bB4vKtGsXzvZzcoLFdSoroU2bcGqAj6bqDnPKboBOneCkk8Lbftu29f+cbG3aBItghTmFe82/\nPzc33DrCVvvvMsl/o3FN2Z1KMmLK7spKKC+HLl3Cq2HHjmChn2HDoFev8OooKwsG59zc8GpIBcuX\nB6t/jRkTbmCHrbQ02JMaNAj69Qu7mvAcPRrs0VZWwmc/2yqrA8Y6ZbfCQUQki8QaDlm8vyYiIg1R\nOIiISBSFg4iIRFE4iIhIFIWDiIhESdurlayvOd8JuwqpcXrX0yn8YWHYZYhIE2K9Wgl3T8uvUaNG\neWPmbJrjPf+rp8/ZNKfRdmrfOu25m5j6EZFwAfkewxgb+iDf3K/GwiHsgTIb2yscRNJD1oZDKgyU\n2dhe4SCSHtIyHIBcYDHwUlNt6wuHVBkos7G9wkEkPaRrOPwYmNaccIh74Fv8nPe8r0dKDKyZ0F7h\nIJIABw64v/qq+0svue/d2ypdpl04AP2At4HPxRsOcQ98bz7mPSa292dvv9J9wYKm2ydoYK34x7t+\nYOpjPmfly6EN9PuP7Pf385/38mPlLeq/ReGwZo0fvPcer3zmmeb30QrWvDnDN334Rqg1eEFBTL+T\nCVVd7b57d7g1uAc1VFeHW8Peve5Hj4a3/Ycfdp84Mfj63e/cKypa3GWs4ZBKn3N4ELgNaHAebTO7\nyczyzSy/uLgYgLmFcxn/9HieHv80Fw+4uMmNzC2cy/gPfsxdu4bRY9cBWLu26fbx9h9j+52zpjP1\ngz8z/vnrE9J/LO2XL3uLo//zO1ZunNei/ltk8WIWrfs7pfPeDm3Bn2OHD7LjiUfYPO3hULZ/3Guv\nwUsvBTOzhqWgAB55JJgJNCxHjsCf/gRbt4ZXAwTrnCxYEM62jx2DXbs+ur1vX3LXtoglQRL9BVwB\nPBz5+WJi3HNo9jvo2ZO95I4Jvue2W9yXLGm6fYLe0b/9+iPe81edfc6qVxLSfyztDx075AtWvuVH\nKo60qP8W7Tls3Ojlv7vfq19+ufl9tIIN773oW5e+F2oNvnWr+/Ll4dbg7l5WFnYF7vv2hV2B+8GD\n7pWV4W3/0Uc/2nP4/e9bpRbS6bAS8BugCCgEdgLlwN8ae86QkUNaNlCWlTX6y5dqx/RTvb3OOYgk\nQHm5++zZ7m+9FZx/aAVpFQ4nFBTjnkPeqXkpO1BmY3uFg0h6yPhwGDJySEwvRDoMrJnQXuEgkh7S\nNhxi/Wpq+gz39BlYM6G9wkEkPWR9OKTTwJoJ7RUOIukhq8Mh7IEyG9srHETSQ9aGQyoMlNnYXuEg\nkh6yMhxSZaDMxvYKB5H0kHXhkEoDZTa2VziIpIesCodUGyizsb3CQSQ9ZE04pOJAmY3tFQ4i6SEr\nwqG5A9mkhZN88qLJPmnRJF+yI7y5lTKpvcJBJD1kfDg0d26lyYsm+389e6s/f+uX/IVbv+T3P/MT\nX7xjcYPtEzWwvl3wtnf9TRe/54Uf+7b921q9/5jbV1W579jhcwpmhze30qZN7tOmuc+dG/4UzSIZ\nLtZwSKUpu+NSUFrQrGmrzYwh89bRtXg/XYr3M+T9tSzcvrDB9omaRvtfnvoXbt8zjAufXcibr/yh\n1fuPtf3mJ/7E7+/5Av/yxFUJ+/c2yh1mzGDtP2Zx8PUXYc2alvXXXFVVFPzqxxT9+f5wth+xvmQ9\n72x+J9QaUkJlJTz3HJSVhVvH7NnBFOZhmjo1mL7cPambTdtwGNR9UKsMZObxtY+3/4ba33fJfZxV\n2SO4s5H/9ETXM+/wOu7qtYI/9/9e8oMh1bhDdXL/AKNKILJLL4GwX4uwtx+mWHYvUvGruXMrLdu5\nzP/rmZ/4C7d+yV/4yRV+/8wfHz+slMxj9FXVVf7Wypf82b//ucHDSkmrJ39mTIdzGuu/VQ4rzZmj\nw0oiCUamn3NoydxKy3Yuizohncone9OhvU5Ii6SHrA6HsAfKbGyvcBBJD1kbDqkwUGZje4WDSHrI\nynBIlYEyG9srHETSQ9aFQyoNlNnYXuEgkh6yKhxSbaDMxvYKB5H0kDXhkIoDZTa2VziIpIesCIdU\nHSizsb3CQSQ9ZHw4NHduJbVPTHuFg0h6yPhwyDs1L2UHymxsr3AQSQ9pFQ7AacAcYDWwEpjQ1HOG\njBwS0wtRdyCrrKr09za/548seMQfWfCIv7v5Xa+oqmiwfbz9Z2t7hYNIeki3cOgDnBf5uTOwDjiz\nsec0d26lJ1c86RPnTPR7X7rD733xdp84Z6JPWzatwfbx9h9P+zXFa/zDog/9wNEDCek/1vZvF7zd\n4v4VDiLpIdZwyEvWBH+NcfcdwI7IzwfMbDVwKrCquX3WN3voroO7WFW8il6bdnPmO6sBZ82nh7EW\nmLlqJje/fHPCZ2OtaT9v6zxe3/g6APO2zuPmMTfTNrdtq/Ufa/vvjf4e72x+h/KKcq4YckWr9R+v\nA3u206n7x7CcECcKPnwY8vKgTZvwahBJESk3ZbeZDQDOBT5sbh8NDWS7Du0C4JRNuzF3zKF3wS4K\n9xXy7Re/nbRgANhQuuH443uP7GVP+Z5W7T+W9o9f/ThYcF/+9nyqvbpV+m+OhT/7Bhvffykhfcds\n5sxg/n4RSY09hxpm1gl4Bvihu++v5/GbgJsA+vfvX28fjQ1kPTv0BGDPaT3osXUP5vD+Kcd4etXT\n/PXKvyYtGABO73Y6G/duBKBz2850P6l7q/YfS/uLTr+IksMlbCjdwIjeI8ix6PcKyVq/YeTPfk+3\nPgMT1n9MrrpKew0iNWI59pSML6AN8Drw41jaN3dupf9d+r8+cc5E/8/nbvXvPH6td/iPDn77m7fH\ncqiuVc8BVFdX+5IdS3zuprleWl7a6v3H2r66utoPHTvU4v51zkEkPZBmJ6QNeBx4MNbnNGdupYNH\nD/rKXSv9gfcf8CueuMLb/7q9f/fF7/ryXcsbPCkcT//Z3F7hIJIeYg2HVDmsdAHwb8ByM1sSue+n\n7v5KLE9u7NCHu7O+dD3zt81nY+lGHKdwXyGzC2fzleFf4ZROpzBz1UwMY9DJgxhz6hg+0eMTmFlM\n/cdbT6a2F5HMkhLh4O7vcfzUaHwaG8j2HdnHC2teYNO+TcfvK9xXyNOrnmb8meMZ0G3ARzXgbNy7\nkY17N3J619O5auhVdD+pe0rPjaOyAAAMfklEQVQOxKnYXkQyS0qEQ3M1NpCtK1nHzFUzOVZ17Ph9\nDQVDXZvLNvOnBX+iX5d+3PrmrSk3EKdi+89O/WyT7UUkfaRtOBw4dqDBgWztnrU8ufLJEy7NjDUY\naqwvXc+9793LH7/wx5QbiNOxvYikl5T7nEOsCkoL6h2YSg+X8szqZ1oUDLXbF+0vOuEzCPVJtYE4\n1dqLSPpJ23AY1H1QvSefX1jzQrMOJTXUvrK6kufXPF9zVVWUVBuIU629iKSntA2Hzm07R923es9q\nNpdtPn67pcFQo2h/ESt2r4hqn2oDcaq1F5H0lbbhUJ8F2xYc/7m1gqHG/G3zT7idagNxqrUXkfSW\nMeFQdqTs+CWrrR0MAFv3b6WkvARovYG19HApW8u2UlldGVP7ePuPpf3ovqPZUraF8oryVutfRNJf\n2l6tVFfR/iIgMcFQY9uBbSzfvbzFA/fhisPMXDXz+NxKHdp04AuDv8CI3iOSFgxPXfMU5RXlPDDv\nAaq9mrycPD7V71NcMuiSFvUvIpkhY/YcdhzckdBgAHhtw2utMnC/sv6V48EAUF5RzrOrn2XW2llJ\n22Po2LYj87fNP35VV2V1Je9ueZflu5Y3u38RyRwZEw4fFH0Q10Bvb8/muSXT4wqS/3j3P1o8cFdW\nV7KqOHqZioK9Bdzw3A1JO2ewdOdSAPKOVjD4g/XkVlQBsGzXsmb13xJ7D+/loQ8e4oOiDxK6nXQw\n++1JPPvX26ioqgi7FEkRq4tXs2TnkqYbtrKMOKw0t3Au979/f1wD/XM5H3LngXM4Escexm3n39bi\ngdvdo9ZNqOn/V5/9VdJOJld5EAY5VdW0P3gEi1yqW1ldmfS5lSqqKzhw7ACHjh1qUT+Z4PCBUqr2\n7aXKq2iDpg8XOFRxiCOVR5K+3bQPh5qB7Oef+TmHKpoeXI4fSjr7Xzka56GnMaeOibmehgbWNrlt\nOKP7GawvXR/V/3Ujr2tx/7G2P7PXmew8uJNjHdqx/J9GHr//cMXhpM+t1Ltjb+749B3k5aT9r2OL\nfeGqW6n2anJzcsMuRVLE6L6jQ9luWh9Wqj3wXXbGZU22P+EcQ/eBeE7jc/3VPSfRp1OfmOtpbGD9\n4pAv0rNDz+P9X3vmtXz7vG/zsU4fa5X+Y2l//mnnM7j74BPuM4xfzP1Fs/pvKQVDwMwUDJIS0vYv\nsu7cSjWXmTakNU5W9+3ct8H28Qzc3dp3Y3iv4dw15y5+f/nvuXb4tXRuF/2hvub2H0v7vJw8rj/r\nerYf2E7xoWI27t3Id176jj73ICJAGodDQWkBb45/8/jA1KNDD/p27sv2A9uj2rZGMJzS8RRO6XRK\nve2bM3BfO/Nanv3Ks6F/YK1v576sK1mnYBCRE6TtYaX65lYa0zf6nEBrXd7a0HG/VPtkcqq1F5H0\nlLbhUN/cSmedcha9O/Y+fru1gqFnh56c2+fcqPapNhCnWnsRSV9pGw71yc3J5eqhV5NjOa0WDIZx\n9dCro06YptpAnGrtRSS9ZVQ4QHAM/ZSOp7TaJ6UvP+Ny+nXpd8J9qTYQp1p7EUl/aXtCuiFzC+fy\n8zk/578v/W92HdqFU/86DDUa22O47IzL+GS/T0b1n0oDcaq1F5HMkFHhUHcgK9pfxPNrnm9wJbeG\ngqH7Sd25eujV9O/av9H+460n09uLSObImHCobyDr16Uf3x39XRZuX0j+9nyKy4uPt68vGHqc1IMx\np45hVJ9RtMlt02T/8daTye1FJMO4e0p8AZcDa4ENwB1NtR81apTXmLNpjvf8r54+Z9Mcb0xRWZF/\nWPSh/+rvv/LO93b2n7/9c39+9fP+YdGHvrVsq1dXV9f7vFj7z+b23E1MbUUkXEC+xzImx9Io0V9A\nLrARGAS0BZYCZzb2nJpwSMWBMhvbKxxE0kO6hcOngNdr3b4TuLOx54waNSplB8psbK9wEEkP6RYO\n1wB/qXX734A/NPacISOHpOxAmY3tFQ4i6SHWcLCgbbjMbDxwmbt/K3L734Cx7n5LnXY3ATdFbn6C\n4BxFuusJ1H85VfbRaxHQ6/ARvRaB1nwdTnf3Xk01SpWrlYqA02rd7gdEzaDn7o8CjyarqGQws3x3\nD2fC9hSj1yKg1+Ejei0CYbwOqfIJ6QXAYDMbaGZtgX8FZoVck4hI1kqJPQd3rzSz7wOvE1y5NNnd\nV4ZclohI1kqJcABw91eAV8KuIwQZdZishfRaBPQ6fESvRSDpr0NKnJAWEZHUkirnHEREJIUoHEJi\nZqeZ2RwzW21mK81sQtg1hcnMcs1ssZm9FHYtYTKzbmY208zWRH43PhV2TWEwsx9F/i5WmNl0M2sf\ndk3JYmaTzWy3ma2odV93M3vTzNZHvp+c6DoUDuGpBH7i7sOAccD3zOzMkGsK0wRgddhFpICHgNfc\nfShwNln4mpjZqcAPgNHuPoLgIpV/DbeqpJpCMNdcbXcAb7v7YODtyO2EUjiExN13uPuiyM8HCAaB\nU8OtKhxm1g/4IvCXsGsJk5l1AS4EJgG4+zF33xduVaHJA04yszygA/V87ilTufs7QGmdu68CpkZ+\nngpcneg6FA4pwMwGAOcCH4ZbSWgeBG4DqsMuJGSDgGLgr5FDbH8xs45hF5Vs7r4N+C2wBdgBlLn7\nG+FWFbpT3H0HBG8sgd6J3qDCIWRm1gl4Bvihu+8Pu55kM7MrgN3uvjDsWlJAHnAe8Cd3Pxc4RBIO\nH6SayPH0q4CBQF+go5l9Ldyqso/CIURm1oYgGJ5w92fDrickFwBXmlkhMAP4nJn9LdySQlMEFLl7\nzR7kTIKwyDb/BGxy92J3rwCeBc4Puaaw7TKzPgCR77sTvUGFQ0jMzAiOLa929wfCrics7n6nu/dz\n9wEEJx1nu3tWvkt0953AVjP7ROSuS4BVIZYUli3AODPrEPk7uYQsPDFfxyzg65Gfvw68kOgNpswn\npLPQBQRTky83syWR+34a+aS4ZK9bgCcic4wVAN8IuZ6kc/cPzWwmsIjgqr7FZNEnpc1sOnAx0NPM\nioCJwH3AU2b2TYLwHJ/wOvQJaRERqUuHlUREJIrCQUREoigcREQkisJBRESiKBxERCSKwkGyhpld\nbGZuZheHXUtrMLNCM5tS574vmdlyMzsS+bd2C6k8SXO6lFWyRmRiuzOBVZkwVYmZnQvsd/eNkdt5\nBBO2vQ/cCxwDFrh7VXhVSrpSOIhkCDM7HSgEvunuk0MuR9KcDitJRjGzIWb2XGSxlCNmtsXMnjaz\nvLqHlczs7sjt+r7+vVafA83sCTMrNrOjZrbEzP45zrruNrOod2JmNiUyr1TN7QGR7X/HzH5pZjvM\nbJ+ZvRiZ2rz2c48fVjKzuwmCAWBSpI+58dQoUpumz5BM8xKwD7gZ2EOwRsYXqP+N0F+A1+rcdyvB\nXPnrIFixj2Aq9d3Ajwim1P4K8IyZXe3usxLwbwC4k+Dw0I0E0zP/N/AEcFED7f8CrACeBn4NvAyk\n/aEzCY/CQTKGmfUEBgNX1Rm0p0UeP6G9uxcRzIRa8/zxwJcJpk9/P3L33YABF7l7SeS+1yOh8UuC\nCdESYbO7X1ertl7A/WbW192jFr5x96Jac3RtdPcPElSXZAkdVpJMUkIwWd19ZvZtMxsc6xPNbDTB\nClsPu/vvaz10OfAKUBY5NJUXOfH7OnB25CR3Irxc5/byyPf+CdqeyAkUDpIxPLi64lIgH/gNsM7M\nCszs5saeFzmWPwuYS7CWdW29gRuAijpf90ce79Fa9ddRd5nIo5Hv7RO0PZET6LCSZBR3LwBuiKwD\ncDbwfeDhyEnfw3XbR5bhnEVwfuIr9Vz2WQK8C/xnA5uMdW3jI5HttXX3Y7XuT1S4iLSIwkEyUmQv\nYomZ/Rj4JjACWFC7TSRA/kawFOVYdz9QT1evAZ8CVrp7VLjEYXPk+wiCdQqIfEDtfKC+7YqESuEg\nGcPMzgIeAp4ENgC5wL8TLBgzG+hc5ym3E1yZNAHoa2Z9az220d2LgbuA+cA7ZvYHgstFTyYY5Ae5\n+40xlvcqUAY8ZmYTgXbAbcDB+P6VIsmhcJBMspNglawfA/0IDuUsB65w94X1TJsxNPL9oXr6+gYw\nxd23RE5W303wqeNeBIeaVhCcwI6Ju+8zsyuA3wFPEVwl9UuC9ZLr1iUSOn1CWkREouhqJRERiaLD\nSiItFPncQ2OqXLvokma05yDSAmY2gOjPQNT9amjKC5GUpXMOIi1gZm2Bs5potraBy2RFUpbCQURE\nouiwkoiIRFE4iIhIFIWDiIhEUTiIiEgUhYOIiET5/xBSJFzPPv0yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b888898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt = plot_cnt(X,y)\n",
    "ax = plt.gca()\n",
    "ax.add_patch(\n",
    "    patches.Rectangle(\n",
    "        (0, 0),   # (x,y)\n",
    "        width=3.5, height=4.5,  color='green',\n",
    "        fill=False, hatch = '//'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let us try to find the best linear classifier by using linear regression!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "regr.intercept= -0.584200666201\n",
      "regr.coef= [ 0.10007944  0.04197521]\n"
     ]
    }
   ],
   "source": [
    "xnames =['size_unif','marg'] \n",
    "X = np.array(df[xnames])\n",
    "y1=y-0.5 \n",
    "from sklearn import datasets, linear_model\n",
    "regr = linear_model.LinearRegression()\n",
    "regr.fit(X,y1)\n",
    "print('regr.intercept=',regr.intercept_)\n",
    "print('regr.coef=',regr.coef_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize this linear regression function as a separating line by setting y1=0, and find the relation between the two features by setting\n",
    "w0+w1 x1 + w2 x2 = 0 -> x2=(-w0 - w1 x1)/w2\n",
    "We can plot this line together with the previous scatter plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x10b67d9b0>]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAERCAYAAACQIWsgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd81fX1+PHXyd4JhDBC2BBWWCFB\nZBS3tlq0KlZR0a8W3LOto7Z1/KzaVi1aRAUnoCiuihsXMmQlhL0DIQQCBEIWGWS8f398giIJkJvc\n+/nc5J7n45FHkpvP53PODeGe+1nniDEGpZRS6lh+TieglFLK+2hxUEopVYcWB6WUUnVocVBKKVWH\nFgellFJ1aHFQSilVh63FQUReE5H9IrKunp/9SUSMiLSxMyellFJ12b3n8AZwwfEPikgn4Fwg2+Z8\nlFJK1cPW4mCMWQDk1/Oj/wD3AXpHnlJKeQHHzzmIyFhgtzFmtdO5KKWUsgQ4GVxEwoCHgPMauPwk\nYBJAeHj40D59+rgcs6raUFpZRVRIoMvrKqVUc5eenn7AGBN3quXE7t5KItIV+NQYkyQiA4BvgdLa\nHycAe4Bhxpi9J9tOSkqKSUtLczn+3e9k8Pnavbxz03CSO7dyeX2llGrORCTdGJNyquUcPaxkjFlr\njGlrjOlqjOkK5ADJpyoMTfHwb/vTPjqEm2ems6+o3FNhlFKqWbP7UtbZwBKgt4jkiMiNdsYHaBUe\nxPQJKRyuqGLSzHTKK6vtTkEppbye3VcrXWWM6WCMCTTGJBhjXj3u512NMQc8nUfv9pE8+/vBrN5V\nwF8+Wou2LVdKqV9y9IS0k87v3557zknkP99soV+HKP4wurvTKSmlGqGyspKcnBzKyxt2mHhv6V4K\njxRSVVNFkH8QcSFxxATHeDhL+4WEhJCQkEBgYOMuvvHZ4gBwx1k92ZhbxBOfb6R3+0hG9zrlCXyl\nlJfJyckhMjKSrl27IiL1LlNRVcHqfatZsXsFeaV5P/+gEqRc6NaqG6nxqfRu0xs/cfwK/yYzxnDw\n4EFycnLo1q1bo7bR/H8LTeDnJzxzxSAS20Vy+9sZZB047HRKSikXlZeXExsbe8LCsKtwF88ve57P\nt37+y8JQy2DYfmg7765/l1dWvkLJkRJPp+xxIkJsbGyD96bq49PFASA8OIDpE1IQgYkz0igur3Q6\nJaWUi05UGLILs5mxegaHKxv2xm9P8R5ey3iNw0ea/xvFE/1OGsrniwNAp9ZhTB2fzPYDh7nn3dXU\n1OgJaqWau6KKImavnU1ljWtv+PLL8pm9brZL62RlZZGUlOTSOvVJS0vjzjvvbPJ23EGLQ60RPdvw\ntwv78s3GfUz+ZovT6SilmihtTxplVWWNWjenKIfth7a7OaNTS0lJ4fnnn7c9bn20OBzjuhFduSIl\ngee/28bna3OdTkcp1UjVNdWszF3ZpG2s2L3CpeWrqqq47rrrGDhwIJdffjmlpaWkp6czZswYhg4d\nyvnnn09urvW6csYZZ3D//fczbNgwEhMTWbhwIQDz58/noosuAiAvL49zzz2X5ORkbrrpJrp06cKB\nAwfIysqib9++TJw4kf79+3PeeedRVta4IngyWhyOISL8v0uSSO4cwx/nrGbDniKnU1JKNcKmA5ua\nfGJ588HNFFcUN3z5zZuZNGkSa9asISoqihdeeIE77riD999/n/T0dG644QYeeuihn5avqqpi+fLl\nTJ48mUcffbTO9h599FHOOussVq5cye9+9zuys3+eaLB161Zuu+021q9fT0xMDB988EGTnmt9tDgc\nJzjAn5euGUp0aCATZ6SRf/iI0ykppVxU31VJrqoxNRwsO9jg5Tt16sTIkSMBuOaaa/jqq69Yt24d\n5557LoMHD+bxxx8nJyfnp+UvvfRSAIYOHUpWVlad7S1atIgrr7wSgAsuuIBWrX7uBdetWzcGDx58\n0vWbSotDPdpGhfDytUPJK6ng1rfSqayucTolpZQLKqoqbN/O8VcHRUZG0r9/f1atWsWqVatYu3Yt\n8+bN++nnwcHBAPj7+1NVVVVneyfr3HB03ZOt31RaHE5gUKcYnrp0AEu35/P4pxucTkcp5YIg/yDb\nt5Odnc2SJUsAmD17NsOHDycvL++nxyorK1m/fn2Dtzdq1CjmzJkDwLx58zh06JALmTedFoeTuDQ5\ngYmju/Hmkp28s1wnmCrVXLQJa/ooekGIDYtt8PJ9+/blzTffZODAgeTn5/90vuH+++9n0KBBDB48\nmB9//LHB23v44YeZN28eycnJfPHFF3To0IHIyMjGPJVGsX2eg7s0dp6Dq6prDNe/vpyl2w8ye+Jw\nUrq29nhMpVTDbdy4kb59+/7isaqaKp5d8iyllaUnWOvUEmMTGT9gfFPTa7SKigr8/f0JCAhgyZIl\n3HLLLaxatcqlbdT3u2kW8xyaA38/YcpVyXSMCeXmWSvZU+D+S8aUUu4V4BfA4PaDm7SN1PhUN2XT\nONnZ2aSmpjJo0CDuvPNOpk+fbmt8LQ4NEB0WyPQJKZRXVnOTzoBQqlkY1nEYwf7Bp16wHu0j2tOz\ndU83Z+SaXr16kZGRwerVq1mxYgWpqfYWKy0ODdSrXSSTfz+YdXsKeeCDNToDQikvFxMSw7j+4/AX\nf5fWiwqO4qqkq5rcm6i50+LggnP6teOP5ybyv1V7mLbA/lvrlVKu6dm6J+MHjG/wHkRcWBw3DLmB\n6JBoD2fm/Xx6nkNj3HZmTzbmFvPUl5vo3T6SM3q3dTolpdRJ9GjdgztOu4OVuStJ35NOYUVhnWXi\nI+NJjU8lqW0Sgf6NG47T0mhxcJGI8O9xA9l+4DB3zM7g49tG0j0uwum0lFInEREUwa+6/IpRnUeR\nmZ9JQXkBR6qPEBIQQofIDsRHxjudotfRw0qNEBYUwLRrhxLo78fEGWkU6QwIpZoFP/GjV2wvUjum\nMrLzSIbGD3W8MBzbbG/u3Lk89dRTjuZzlBaHRurUOoypVyez82Apd7+zimqdAaGUaqKxY8fywAMP\nOJ0GoMWhSYZ3j+Xhsf35btN+npm32el0lFIOycrKok+fPvzhD38gKSmJq6++mm+++YaRI0fSq1cv\nli9fzvLlyxkxYgRDhgxhxIgRbN5c9zXjjTfe4PbbbwcgMzOT4cOHk5qayt///nciIqzD1/Pnz+eM\nM87g8ssvp0+fPlx99dUeuXrS1uIgIq+JyH4RWXfMY/8WkU0iskZEPhKRGDtzaqprTuvMVcM6M3V+\nJp+s3uN0OkqpU9m7F159FZ58Et57D9w0C2Hbtm3cddddrFmzhk2bNvH222+zaNEinn76aZ544gn6\n9OnDggULyMjI4LHHHuMvf/nLSbd31113cdddd7FixQri43956CsjI4PJkyezYcMGtm/fzuLFi93y\nHI5l957DG8AFxz32NZBkjBkIbAEetDmnJhERHh3bn5Qurfjz+6tZt7vulRBKKS9hDMyZA7t2QUUF\nrF8Px3RKbYpu3boxYMAA/Pz86N+/P2effTYiwoABA8jKyqKwsJBx48aRlJTEPffcc8omfEuWLGHc\nuHEAjB//yzYew4YNIyEhAT8/PwYPHtz8W3YbYxYA+cc9Ns8Yc7Tf7FIgwc6c3CEowI8XrxlKq7Ag\nbpqZzoES97QLVkq5WXEx5Of/8rGdO92y6WPbaPv5+f30vZ+fH1VVVfztb3/jzDPPZN26dXzyySeU\nl5e7JZavtOy+AfjC6SQaIy4ymGnXpnCgpIJbZ63kSJXOgFDK60REQPRxN7h17GhL6MLCQjrWxnrj\njTdOufzw4cN/mvD2zjvveDK1enlNcRCRh4Aq4K2TLDNJRNJEJC0vr+mTntxtQEI0/7p8IMuz8nn0\nk4b3bVdK2cTPDy6/HOLirO979IDzz7cl9H333ceDDz7IyJEjqa4+dX+2yZMn8+yzzzJs2DByc3OJ\nPr6oeZjtLbtFpCvwqTEm6ZjHrgNuBs42xjSox65dLbsb46kvNvHSD5k8fkkS1wzv4nQ6SrVo9bWl\nbpCaGqtYeKnS0lJCQ0MREd555x1mz57Nxx9/7NI2mtKy2/E7pEXkAuB+YExDC4O3+/P5vdm0t4hH\n5q4nsV0kw7rpDAilvI4XFwaA9PR0br/9dowxxMTE8Nprr9ka3+5LWWcDS4DeIpIjIjcCU4BI4GsR\nWSUiL9mZkyf4+wnPXTmEzrFh3DIrnd06A0Ip5aLRo0ezevVq1qxZw4IFC+jZ094W4nZfrXSVMaaD\nMSbQGJNgjHnVGNPTGNPJGDO49uNmO3PylOhQawbEkaoaJs1Io+yIzoBQSjUf3r1f1cz1iIvg+auG\nsCG3iD+/v1pnQCjlIfp/q66m/k60OHjYmX3act/5ffh0TS5T52c6nY5SLU5ISAgHDx7UAnEMYwwH\nDx4kJCSk0dtw/IS0L7h5THc25hbx9LzN9O0QyVl92jmdklItRkJCAjk5OXjj5e1OCgkJISGh8fcU\na3GwgYjwz8sGsv1ACXfNXsVHt42gZ9tIp9NSqkUIDAykW7duTqfR4uhhJZuEBvnz8rUpBAf6MXFG\nOoVlOgNCKeW9tDjYqGNMKC9eM5ScQ6XcOTtDZ0AopbyWFgebpXZtzaNjk/hhSx7/+mqT0+kopVS9\n9JyDA8af1pkNuYW8/MN2+nWI4uLB9jT+UkqphtI9B4c8/Nv+DOvWmvveX8OanAKn01FKqV/Q4uCQ\nQH8/Xrw6mTYRwdw0M539xY3v7a6UUu6mxcFBsRHBTJswlEOlR7hl1koqqrTFhlLKO2hxcFj/+Gie\nHjeI9J2HePjj9XqXp1LKK+gJaS9w0cB4NuUWM+X7bfSPj+La07s6nZJSysfpnoOXuPfcRM7p25ZH\nP9nAksyDTqejlPJxWhy8hJ+f8J/fD6Zrm3BufSudXfktYu6RUqqZ0uLgRSJDrBkQ1TWGiTPSKD1S\n5XRKSikfpcXBy3RrE85/xyezZV8xf3pPZ0AopZyhxcELjUmM48Ff9+XztXt54fttTqejlPJBWhy8\n1B9Gd+N3Qzry9LwtfL1hn9PpKKV8jBYHLyUiPHnpAAYmRHP3Oxls2VfsdEpKKR+ixcGLhQT68/K1\nQwkNCmDijDQKSo84nZJSykdocfByHaJDefnaoeQWlHPH7AyqqmucTkkp5QNsLQ4i8pqI7BeRdcc8\n1lpEvhaRrbWfW9mZU3MwtEsrHr8kiYVbD/DPL3UGhFI+paAAsrOh3N7mnHa3z3gDmALMOOaxB4Bv\njTFPicgDtd/fb3NeXu+K1E5syC1i+sId9GkfxWVDGz84XKlmzRioqQF/f6cz8ayKCvjoI9i82XrO\ngYFw+ulw1lm2hLe1OBhjFohI1+Mevhg4o/brN4H5tMTiUFQEBw9Cly7g17gdtocu7MvmvcU8+NFa\nerSNYHCnmFOvVFMDhw9DZGSjYjbajh2wZQt07w69etkbW3meMbCu9gBAUhKI2BN3zRr49FOorITU\nVPjNb+yJ64RPP4VNxxwpqKyEBQugVSsYMsTj4b3hnEM7Y0wuQO3ntg7n435FRTB1Krz5JnzySaM3\nE+jvx9Srk2kXFcxNM9PYX9SA3czZs+GZZ+Cbbxod12V5eTBrFixZAm+/Dbt32xe7pATeew+2b7cv\nJsC+fTB5Mvzvf/bGLSqCF16wft81Np6P+uEH+OAD6+O77+yJWVICH38MR45YxWn5ctiwwZ7YR+Nv\n2WJPrNJSWL++/p+lpdmSgjcUhwYTkUkikiYiaXl5eU6n03D5+T8fL2ziC2Wr8CCmT0ihuLyKm2al\nn3oGRFaW9XnnzibFdUl+PlTX5mWMtcdkl717rf9Um2w+N5OTYx0btjvu/v1WMc7MtA5D2GXfvvq/\n9qTDh3/+uzqq2MZLvLOy7HuTVVZ24mJfUmJLCt5QHPaJSAeA2s/7T7SgMWaaMSbFGJMSFxdnW4JN\n1rkzJCdDu3Zw7rlN3lyf9lE8M24QGdkF/PWjdSdvsXHppdC/P1xwQZPjNlj37tCxdi5227aQmGhf\n7J494cYb4Zxz7IsJMHiw9TseP97euD16wG9/C1ddBaGh9sUdMQLCw62PUaPsiRkba/09HRUUZD1/\nuyQlwa232hOrVSuIiqr/Z1262JKC2N27p/acw6fGmKTa7/8NHDzmhHRrY8x9p9pOSkqKSbNp98pb\nPfv1Fp7/disP/7Yf/zeym9Pp/FJNjfWuLiKi5Z84VPYpLYUff7QOLSUnQ/v2TmfkOWvWWCekj32N\nDg213vy0adPozYpIujEm5VTL2XpCWkRmY518biMiOcDDwFPAHBG5EcgGxtmZU3N299m92JRbxOOf\nbSSxXSQjezb+D8bt/PwgOtrpLFRLExZm/16hUwYOtC4kWb4cCgshPt7aY2vd2pbwtu85uIvuOVhK\nKqq4dOpi9hdXMPe2UXSODXM6JaWUF2vonoM3nHNQTRARHMD0CSkYAxNnpHG4QmdAKKWaTotDC9Al\nNpwXxiezdX8x985ZRU1N89wbVEp5Dy0OLcSoXm146MJ+fLV+H89/t9XpdJRSzZzd7TOUB90wsisb\nc4uY/M1W+rSP4oKkFnwlh1LKo3TPoQURER6/JInBnWK4d84qNu0tcjolpVQzpcWhhTk6AyIi2JoB\nceiwzoBQSrlOi0ML1C4qhJevHcq+ogpun71SZ0AopVymxaGFGtK5FU/+bgCLtx3kH59vdDodpVQz\noyekW7DLhiawIbeIVxftoG+HKK5I6eR0SkqpZkL3HFq4B3/dh1E92/DXj9axMvuQvcHt7MaqlHIr\nLQ4tXIC/H1PGD6F9dAg3z0xnX0NmQLjD6tXw3/9afWF8xKJ/3Ezm4k+dTsM+331n75wQX7VnD7z0\nElTZ2/1Ai4MPiAkL4pXrUjhcUcWkmemUV55iBoQ7tG0LnTq17K6Zx2mTOoZWnXs7nYZ9unWz2rMr\nz4qOtlrC29zdWBvv+ZB56/cyaWY6lw7pyDNXDELsGu2olPIa2nhP1XFe//bce24iH2bs5tVFO5xO\nRynlxbQ4+Jjbz+zJr5Pa88TnG1m4tRmNWlVK2UqLg4/x8xOeHjeIxHaR3P52BlkHDjudklLKC2lx\n8EHhtTMg/MSaAVFcXul0SkopL6PFwUd1ah3GC1cns/3AYe55d7XOgFBK/YIWBx82okcb/n5RP77Z\nuI/J32xxOh2llBfR9hk+bsLpXdiwp4jnv9tGnw5R/GZAB6dTUkp5Ad1z8HEiwmOX9Ce5cwx/nLOa\nDXt0BoRSSouDAoID/Hnp2qFEhwYycUYa+ToDQimf5zXFQUTuEZH1IrJORGaLSIjTOfmStpEhTJsw\nlAMlFdz6VjqVOgNCKZ/mFcVBRDoCdwIpxpgkwB+40tmsfM/AhBieumwAS7fn8/inG5xORynlIG86\nIR0AhIpIJRAG7HE4H5/0uyEJbMwtZtqC7fTtEMWVwzo7nZJrDhyADRugQwfo1cvpbJRyj8pKCAy0\nNaRLew4iUiMi1Sf4qBKRgyLytYic58p2jTG7gaeBbCAXKDTGzHNlG96uvKSAXasW2B94xQqYMgUy\nMxu8yv0X9GF0rzb87eN1pGXlNy6uMbBlC9TYeHiqrAxee81qJf3WW7B1q32xDx602ip/+aV9McH6\nPX/3HaxaZW/cykqYNQtmzrS+ttPq1fDtt9Zzt1NZGezcaW9MgLffhn/8A5YssTWsq4eV/h+wC8gD\n3gD+CbxZ+30OMBOIA74QkYsaulERaQVcDHQD4oFwEbmmnuUmiUiaiKTl5TWvvkA5qxaQ+cq/7A+8\ndq31bnrTpgav4u8nTLkqmYRWYdw8ayV7Cspcj1tcDHPmQH4ji0tjFBZCaenP3+fm2hd7xw7Yuxcy\nMuyLCVBeDosWwY8/2hv30CHYts1602H3UKclS6znXNaIv8um2LIF/vc/e2PW1Pz8JmeLvfciudSy\nW0QeBM4Dfm2MKT/m8VDgC+Br4EngMyDaGDOigdsdB1xgjLmx9vsJwHBjzK0nWqc5tuyuqa7Cz9/m\nI3l798K6dXDaaRAZ6dKq2/YXc8kLP9KtTTjv3Xw6IYEu9pOvqQE/G09rVVdb72SzsiA8HG64AWJj\n7YldWQmLF0NCAvTsaU/Mo3JyICwMWre2N25amvXuPTXV3riHDkFJiTUvxG52/02D9XvetAnGjHHL\nc25oy25Xi8NO4A5jzNx6fnYxMMUY00lELgPeNMZENHC7pwGvAalAGdZeSZox5r8nWqc5Fofm6JsN\n+5g4M42LB8Xzn98P9v4ZENXV1p5SdDSE6AVvSh3PU/Mc2gInOisSBBx9m3YAaPCriDFmGfA+sBJY\nW5vXNBdzUx5wTr92/Om83vxv1R6mLdjudDqn5u8P7dppYVCqiVwtDmnAIyLyix4LIhIPPFz7c4Au\nuHi1kTHmYWNMH2NMkjHmWmNMhYu5KQ+59YweXDigA099uYn5m/c7nY5SygauFoe7gARgh4h8LyLv\nisj3wHasE8l31i7XE3jbfWkqJ4kI/x43kL7to7hjdgbb80qcTkkp5WEuFQdjzEqsF/5ngRpgQO3n\nZ4BexphVtcv93RjzsJtzVQ4KCwpg2oShBPr7MXFGGkU6A0KpFq3BxUFE/EVkEOBnjPmLMeZsY0y/\n2s8PGWNsvp5N2S2hVRhTr05m58FS7n5nFdU6A0KpFsuVPQeDdU5hiIdyUc3A8O6xPDy2P99t2s8z\n8zY7nY5SykMafNG9MaZGRHYB4R7MRzUD15zWmQ17ipg6P5O+HaL47aB4p1NSSrmZqyekXwbuFpEg\nTySjmgcR4dGx/Unt2oo/v7+adbsLnU5JKeVmrt6uGwn0ALaLyJdYfZCOPfBs9ES0bwgK8GPq1UO5\neMoiJs1IY+4do2gTEex0WkopN3H1DulTdVEzxhgXeyw0jt4h7R3W7S7kshd/ZFBCDLP+cBpBAV7R\nBV4pdQIeuUPaGON3ig9bCoPyHkkdo/nX5QNZnpXPo5+sdzodpZSb6Ns81WQXD+7IzWN68NaybGYt\nPaal8eHD9idTUgLz50N2tv2xt22DggL741ZUQFWV/XGVPQ4fhs2b7W1/jxYH2+SsWcziJ0/YZNZz\nDhyA11+HI56dC/3n83tzZu84Hpm7nuU78mH9evj3v2HlSo/GreOzz6ziMGuW1YTPLmvXWjFfe82+\nmGD9u06ebHWjtduiRbDAgRklq1ZZ/86+Ys4cmD0bFi60NazLxaF2pkKGiJTWN/THE0m2BOGx7YlM\nTLI/cGgodO0KAZ5tFe7vJzx31RA6x4Zxy6x0cgiBuDiIifFo3DqOtiUPD7e3tXJkpNX0r1Ur+2KC\nFbNDB+vDbm3bWh92a90a4n3o8ulWrUDE9v9Lrp6QngBMxxrw8wesNtuBwFisgT9vGWMe9UCedegJ\nae+UmVfCJVMW06l1GB/cMoLQIJtPQ9XUWIeU2ra1ZhzYqaLCGuVod79/1bIZY/1tuanTsKdadt+N\nNcznltrvpxpjrgO6Y81h0BYaPq5HXATPXzWEjXuL+PP7q3HlzYdb+PlZe0p2FwaA4GAtDMr9RBxp\nQe/qX3IvYAFWs70arBkOGGMOAf/A6tqqfNyZfdpy3/l9+HRNLlPnN3x2tVLKe7haHMqwGu8ZYC/W\nHsNRJVhtu5Xi5jHdGTsonqfnbebbjfucTkcp5SJXi8NarJbdAAuBv4jI6SKSCjwCNHyKvWrRRIR/\nXjaQ/vFR3PXOKrbtL3Y6JaWUC1wtDtOAo5dj/A2IABYBy4BE4I/uS001d6FB/rx8bQohgX5MnJFO\nYZnOgFCquXD1Dul3jTFP1n69DegPnA9cAvQ0xsx3e4aqWesYE8qL1wwl51Apd87O0BkQSjUTjbq0\nQkQ6icgI4DSsE9MlwEAROcudyamWIbVrax4dm8QPW/L411d65FGp5sClO6NEpDvwFjDs6EP83JX1\n6NfaX0nVMf60zmzMLeLlH7bTt30Ulwzp6HRKSqmTcPW22VeAzlj3O2wCPNuTQbUof/9tPzbvK+b+\nD9bQPS6cgQk23z2tlGowVw8rpQJ3GmP+a4z52hjzw/EfjU1ERGJE5H0R2SQiG0Xk9MZuS3mnQH8/\nXrw6mTYRwdw0M539xeVOp6SUOgFXi0MOnttbeA740hjTBxgEbPRQHOWg2Ihgpk0YSkFpJbfMWklF\nlbbjUsobuVocngDuFxG3zpEWkSjgV8CrAMaYI8YYB3ofK7epqYH8/Ho7o/aPj+bpcYNI33mIhz9e\nb3+LDaXUKbl0zsEYM1NE+gBZIrIUOFR3EXNdI/LojtW473URGQSkA3cZYxwYCOAhlZWQl2d/N8lP\nPoHVq+Hcc+G00+yLO3s2bN0KXbrA9ddb/WGOceHADmzM7cmU77fRLz6KCad3dV/s3buhfXurY6ld\nCgrg44+t53vGGfbFLSuDDz+E6Gi48MI6v+cWp7wc3ngDiorg6quhow9c2PDhh7BhA/z61zB0qG1h\nXdpzEJHrgQeBGCAZGF3PR2ME1G7vRWPMEOAw8EA98SeJSJqIpOXl5TUylEM2b7b6/dtt9WprEMya\nNfbFrK62CgPAzp3Wf+h63HtuIuf0bcujn2xgSaabejZWV1svHjt2uGd7DbVlixXzxx/tjbtrl/W7\nTkuzCkVLt3ev9VFaav3OW7qaGmtWSFWV9dlGrrbs3gmkATe687CPiLQHlhpjutZ+Pxp4wBhz4YnW\naZYtu8vL7e+uuGyZVRjGjIHERPvizptnvWANHAgXXXTCxYrLK/nd1B85WFLB3NtH0am1G7qpOvF7\nLi+3hgx16gT9+9sXt7oavv/e2nNITbUvrlOqq2HuXCguhrFj7Z8X4oTFi2HjRjjrLOje/dTLn0JD\nW3a7WhxKgIuNMd82JbkTbHsh8AdjzGYReQQIN8b8+UTLN8vioOq148BhLp6yiPiYUD64ZQThwZ4d\nTKSUL/PUPIdFQN/GpXRKdwBvicgaYDDWyW/lA7q1Cee/45PZsq+YP73nwAwIpVQdrhaHu4CJInK1\niMSKiN/xH41NxBizyhiTYowZaIy5pHZGhPIRYxLjePDXffli3V6mfLfN6XSU8nmu7r8fvfdgxgl+\nbhqxTaUA+MPobmzILeKZr7fQu30k5/Vv73RKSvksV1/IH+PnXkpKuZWI8OSlA8jMK+Ged1fx0W0j\nSWwX6XRaSvkkl05IexM9Id2kPb84AAAaFklEQVRy7S0s57dTFhEW5M/Ht40kJizI6ZSUajE8dUJa\nKY9rHx3CS9cMJbegnDtmZ1BVXeN0Skr5HC0OyisN7dKKxy9JYuHWAzz1hc6AUMpuevJYea0rUjux\nIbeIVxbtoG+HKC4bmuB0Skr5DN1zUF7toQv7MqJHLA9+tJZVu7QXo1J20eKgvFqgvx8vjE+mXVQw\nk2aksa9IZ0AoZQctDsrrtQoPYvqEFEoqqrhpZjrllToDQilP0+LQ0h05AllZVndHO2VlWQ3Strnn\nbuc+7aN4ZtwgVu0q4K//W3fiFhvGnLQTbItTVgZffgkLFtj/b+wEY2DhQvjiC6szqy/49lt4/XVY\nv97WsFoc7JKTAx98YH/cDz+0Wlj/0OgJrq6rqbHmOaxcCe+8YxUoN/j1gA7ceXYv3k/P4fXFWfUv\ntGSJ9R/p3XfdEtMlc+fa3yr8m29g6VL47jt727KD1fF36VJ7Y65da71YLlsGX39tb2wnbNtmFcOd\nO63/y5WVtoXW4mCXoCCrrbLdjrautrOFtQgEBlpfBwS4dQDN3Wf34rx+7fjH5xtZtPVA3QVCQ63P\ndrfsBoiKguBge2MGHXOD4NHfuV3CwyHMDS3WXXHs8w3ygZsjj/03dfP/pVPRO6Rbupoaa2qW3X3v\n8/KsHvSJidZUNjcqqaji0qmL2VdUwdzbR9Il9riptYWFEBFh7yQ4p1RVwYoV1gv1wIFOZ2OPtWuh\npARSUuwviE5IT4fsbEhOtiYNNpFH5jl4Ey0Ovm3nwcOMnbKYdlHBfHjrSCJ0BoRSDaLtM1SL1iU2\nnBfGJ5OZd5g/zllFTU3zfJOjlLfS4qCarVG92vDQb/ry1fp9PPftVqfTUapF0X1x1az938iubMgt\n4rlvt9K3QyQXJHVwOiWlWgTdc1DNmojw+CVJDO4Uw71zVrNpb5HTKSnVImhxUM1eSKA/L187lIjg\nACbOSOPQYffcV6GUL9PioFqEdlEhTJuQwr6iCm57eyWVOgNCqSbR4qBajMGdYnjydwP4MfMg//hs\n46lXUEqdkJ6QVi3KZUMT2JBbxKuLdtCvQxRXpHZyOiWlmiXdc1AtzoO/7sOonm346//Wkb7zkNPp\nKNUseVVxEBF/EckQkU+dzkU1XwH+fkwZP4QOMSHcPCudvYU+0qFVKTfyquIA3AW0zIPFJSVWZ1Zf\naKvsBWLCrBkQpRVVTJqZpjMgVPNkjNVp+OOPYcsWW0N7TXEQkQTgQuAVp3NxuwMHKH76CXY98zA1\nc+bYGtps2EDRq1MhN9fWuKWVpSxJ/5iSIyW2xmX9eg4/+Rg1n8wlsV0k//n9YNbkFPLgh2tPPAPC\nDUxNDevmvkLuJgf6fa1cCZmZ9sc9fNj6sFtFhdVc0W5VVZCfb2/MBQvgq68gI8Nqg79zp22hvaY4\nAJOB+4ATvrUWkUkikiYiaXl5efZl1lQ7dpC9fyuZhzIpXLPc1tD5337CyoVzOLLsR1vjbti6hIr/\n/oe1a7+1NW7NihWkb5nPoYVfQ1UV5/Vvz73nJvJRxm5eXeS5WQt5O9Zx4MNZZH5g83ubsjJrjsQX\nX9gbF2DePGvQkN0WL3ZmNsqaNdZsFDvt2vXz18bA7t22hfaKq5VE5CJgvzEmXUTOONFyxphpwDSw\nurLalF7T9exJ1479aZW/h5iUUbaGjv3N5aTGdiBo5K9sjTsgcRRr7/sb/XqdbmtcvxEjSC04QEi/\ngVb/e+D2M3uyMbeIJz7fSK92kYxJjHN73LY9BpI//iZie9ncNjs0FMaNs2ZJ2O03v7FesOw2ejQM\nH25/3MGDrRb0durW7edpin5+bmnZ3VBe0bJbRJ4ErgWqgBAgCvjQGHPNidZpdi27y8uhuBjatLF1\nYIeyHK6o4rIXf2RPQRkf3z6Kbm3CT72SUt4gIwP274fevaFr1yZvrtnOc6jdc/iTMeaiky3X7IqD\nctyu/FLGTllEbEQwH906gsgQHxgUo9RxdJ6DUsfp1DqMF65OZseBw9zzrs6AUOpkvK44GGPmn2qv\nQanGGtGjDX+/qB/fbNzPs1/be2mgUs2JV5yQVspOE07vwsbcIqZ8v40+HSK5aGC80ykp5XW8bs9B\nKU8TER69uD9Du7Tiz++tYf0eB66ZV8rLaXFQPik4wJ8Xr0kmOjSQSTPSOVhS4XRKSnkVLQ7KZ7WN\nDGHahKEcKKng1rd0BoRSx9LioHzawIQY/nnZQJbtyOexTzY4nY5SXkNPSCufd8mQjmzMLeLlBdvp\n2yGK8ad1djolpRynew5KAfdd0IcxiXE8PHcdK7Jsbq6mlBfS4mCDPcV7eHfdu7yc9jLf7fiOyupK\np1NSx/H3E56/cggJrcK4ZVY6uwvKnE5JKUdpcfCwwvJCZi5/BXn/fRJmzWXLt3OYu3mubfGXZS3m\no+9fZNOBTbbFBKyGbHv32j+/IjPTam28aJHLq0aHBTJ9wlDKK2u4aWYaZUd0BoTyXVocPGzNvjW0\nXbudtll5hBccpvfizWzelUF5leenk+0q3MXON5+j1ZvvsnT2v22JedTuD99k4YNXkz37JdtiUl2N\nmT2bTYs/pvSLuT93s3RBz7aRPHflYNbvKeL+D9Y0fAZEcTGbH5zI/g9nuhyzqRZlL7K/+Dtp82ZY\nuND+uDk5zrQoX74cnnzS9pksWhwcIA619LG7yaKhefYuOrtvO/50Xm/mrt7Dywu2N3zFGuNIC2tv\na55pC6eesxNxHXquXteVtaGaS1fWgvICXl78PN0XrCWs4DC7kjoRN/I8Lut3mS3xl2YtYs+ONfRL\nOpM+cX1tiQn8fFipXTurD71dtm2DFSsgIcHq+99IxhjumJ3BZ2tzee26VM7s09aNSSrlnGbbsruh\nmktxAOuE9IKdCygsL6RXbC9Gdx5NoL+2i/Z2pUequPzFJezKL+V/t4+kR1yE0ykp1WRaHJRyg90F\nZYz97yKiQwP56LaRRIdqUVfNm85zUMoNOsaEMvXqZLLzS7n7nQyqdQaE8hFaHJQ6hdO6x/LI2P58\nvzmPf3+12el0lLKFts9QqgGuGd6FDblFvPRDJn07RHLx4I5Op6SUR+meg1IN9Mhv+zOsa2vue38N\na3N0BoRq2bQ4KNVAQQF+TL0mmdjwICbNTCOvWGdAqJZLi4NSLmgTEcy0CSkcKj3CLbPSOVKlMyBU\ny6TFQSkXJXWM5l+XDyJt5yEenrvON+9QVi2enpBWqhHGDopnY24RL87PpF98NNcO7+J0Skq5lVcU\nBxHpBMwA2gM1wDRjzHPOZuU+mw5sYlnOMkqOlNAlpgujO48mOiTa6bRUE/3pvN5s3lvMo3PX06tt\nBMO7xzqdklJu4y2HlaqAPxpj+gLDgdtEpJ/DOblF+p503ln3Drv2b+Vw7k7Sdq/g1YxXKa0stSX+\nrsJdLMtZxoHSA7bEO1aNadnH4/39hMlXDqZzbBi3vrWSXfn2/JsqZQevKA7GmFxjzMrar4uBjUCz\nv5C8xtTww84fCCkuY9iHyxj20XL6LN5MUUUR6XvSPR5/W/42Xst4jS+2fcH09Onkl9k34eyTzZ/w\n2A+P8f6G922LeVRRXg7GpjkSUSGBTJ+QQmV1DZPeXE5p+RFb4irlaV5RHI4lIl2BIcAyZzNpurLK\nMooqimi9O5+gcmv6W7vMfQDsLdnr8fiZ+Zk/tc2uqK4guzDb4zEBqmuqSc+1it+6/ets20sCqK48\nwsoHr2dn2re2xewRF8HzVw1h074S/vzKAj1BrVoEryoOIhIBfADcbYwpqufnk0QkTUTS8vLy7E/Q\nRaGBoYQHhlPQoRVVgf4AHOhkHZeOC4/zePzO0Z1/+jrAL4COkfbsjPn7+dM/rj8APVv3JDQg1Ja4\nAP6BQST99b90HnqmbTEBzuzdlvvHdOGznAqmzs+0NbZSnuA1XVlFJBD4FPjKGPPsqZZvLl1Zl+xa\nwleZXxFUWkFISTnFbaIIDQ7nlpRbiAyO9Hj8rQe3srt4N4mxicRHxns83lHGGMqqyggNCEVEbIvr\nJGMMd7+7irmr9zD92hTO6dfO6ZSUqqNZdWUV69XjVWBjQwpDc1FZXUnHqI50i+nGXjnMhpBiqsUw\nvONwjlTbc2y6V2wvzuh6hq2FAUBECAsM85nCANZz/udlA+kfH8Xd765i675ip1NSqtG8Ys9BREYB\nC4G1WJeyAvzFGPP5idbx5j2HfSX7WL57OWv3rz1pEYgLiyMlPoXB7QcTHBBsY4bKk/YUlDF2yiIi\nggP4+LZRRIfpDAjlPXTYjwMqqyv5dse3LMtZ5tL85KjgKMb2HkvP1j09mJ2yU1pWPldNX8rw7rG8\nfn0qAf5esZOuVPM6rNQSHCo7xMvpL7M0Z6lLhQGgqKKIWWtmMS9znl7p0kKkdG3NYxcnsXDrAf6l\nMyBUM+QVd0g3dwXlBby+6nWKKupcYOWSH3f9SGV1JRcmXuimzJSTrhrWmY25RUxbsJ0+7SO5NDnB\n6ZSUajDdc2iiGlPDu+vebXJhOGrFnhW23CCn7PG3i/pxWrfWPPDhWlbvKnA6HaUaTItDEy3KXkRu\nSa5btzkvcx6F5TpMpiUI9Pdj6tXJxEUEM2lmGvuLyp1OSakG0eLQBIePHGbBzgVu325FdQXfZ33v\n9u0qZ8RGBDN9QgpFZVXcPCudiqpqp1NS6pS0ODRBxt4MqmqqPLJtu9tOKM/qFx/F0+MGsTK7gL//\nb71eeKC8nhaHJli1d5XHtl1VU8X6/eubvJ3iimKyC7Mpr7L3cEZpZSnZhdmUHCmxNa43u3BgB+44\nqyfvpu3izR+znE5HqZPSq5UaqbyqnIOlBz0aI6coh9SOqY1at8bU8MnmT1i9bzU1poZAv0B+1eVX\njO4y2s1Z1vXN9m9YmrOUqpoq/MWflPgULuh5gU/dLX0i95yTyMbcIv7fZxtJbBfJiJ5tnE5JqXrp\nnkMj5Rbnunw/g8sxmnCie3H2YjL2Zvw0U6GyxrpBb+vBre5Kr15r9q1hUfainw63VZtqlu1eRtoe\n77ph0Sl+fsJ/fj+Y7m3CufVtnQGhvJcWh0Y6XHnYpeU7rdvFaR8sI7ik4Yd3Dh9xLcaxVu9b7dLj\n7rJ6r7V9/8pqei3dSkCF1ap8zb41Ho17VN7hPCYvnezVlwNH1s6AqKkxTJyRxuGKxp+3qqqp4qMZ\nf+HreS+5MUPlTQrKC/hy25e2n6fS4mCToLIjBJVV4F/d8CE0Tdkzqa6p/4qYEz3uLtXG2r7U1BBS\nUo5f7fP11In74x2pPkJxRbHLxdtuXduEM2V8Mlv2FfOn91ZTU9O4f+saU0NlwUEqig+5OUPlLSqr\nKykoL7B9sqKec2gkV2cUZKb2YEdyN2pc6LHTlDkI/eL6sXjX4jqP943r2+htNjRuVkEWVcGBrD1n\nwC8et0PHqI48OPpBAvy8/0/7V4lx/OU3fXn8s43897tt3HVOL5e3EeQfxGW3T8VP9H1eSxUXHseV\nSVfaHlf/ohqpfUR7l9dxpTAAdIjs4HKMo8Z0HUPXmK6/eCy5QzID2g6ofwU3SYlPIalt0i8eS4xN\n5PROp3s07rGaQ2E46sZR3bh0SEf+880WvlrfuOmA/n7+erJfuV3z+V/kZcKDwokJiaGg3HMtEZoy\ngyHIP4jrB1/PrsJd5Jfl0zGqI23CPH9ljJ/4cXm/yxndeTR7S/YSFx5n+yyJ5kREeOLSAWTmlXDv\nu6v48NaR9G7v+SFQSp2K7jk0gSffhfuJX5134I3RKboTg9oPsqUwHKtdRDsGtR+khaEBQgL9efna\nFMKCA5g4I42CUnsGQSl1MlocmiAlPgXBM7vzibGJRAVHeWTbyvu0jw7hpWuGsrewnNvfzqDKhQsX\nlPIELQ5NEB0S3eib1E7GX/w5q9tZbt+u8m5Du7Ti8d8lsWjbAZ74fJPT6Sgfp8Whic7pfg6tQlq5\ndZtjuo6hbXhbt25TNQ9XpHTi+hFdeW3xDt5L2+V0OsqHaXFooiD/IH6f9HtCAkLcsr0+bfowqvMo\nt2xLNU9/vbAvI3rE8tBH68jI1vsXlDO0OLhB+4j2TBg0gbDAsCZtp2+bvlze73K9Zt3HBfj78cL4\nZNpFB3PTzHT26QwI5QB9FXKT+Mh4bk29ld6xvV1eN8g/iAt7XcgV/a9oVtfoK89pFR7E9AkplFRU\ncdPMdMordQaEspcWBzeKCIrgqgFXMX7AeHq06nHKK5lCA0I5PeF0bku9jdSOqXojk/qFPu2jePaK\nwazaVcBDH63TGRDKVl7zNlVELgCeA/yBV4wxTzmcUqMlxiaSGJvIobJDZBdmk1uS+1NvlCD/INqF\nt6NDZAe6RHch0D/Q6XSVF7sgqT13n9OLyd9spV98FDeO6uZ0SspHeEVxEBF/4AXgXCAHWCEic40x\nG5zNrGlahbaiVWgrBjHI6VRUM3bnWb3YmFvEPz7bQO92kYzqpTMglOd5y2GlYcA2Y8x2Y8wR4B3g\nYodzUsor+PkJz1wxmF5tI7nt7ZXsPOjdHWdVy+AtxaEjcOxF3Tm1jymlgIjgAKZPSEEEJs5Io6QJ\nMyCUagivOKwE9Z65rXP2TUQmAZNqvy0Rkc2NjNcGONDIdZsjX3u+0MKfc+S9dR5q0c+3Hvp8G69L\nQxbyluKQA3Q65vsEYM/xCxljpgHTmhpMRNKMMSlN3U5z4WvPF3zvOevzbdmceL7eclhpBdBLRLqJ\nSBBwJTDX4ZyUUspnecWegzGmSkRuB77CupT1NWPMeofTUkopn+UVxQHAGPM58LlN4Zp8aKqZ8bXn\nC773nPX5tmy2P1/Ruy6VUkodz1vOOSillPIiPlccROQCEdksIttE5AGn8/EkEekkIt+LyEYRWS8i\ndzmdkx1ExF9EMkTkU6dz8TQRiRGR90VkU+2/8+lO5+RpInJP7d/zOhGZLSLu6ZfvJUTkNRHZLyLr\njnmstYh8LSJbaz+7d4hMPXyqOBzTpuPXQD/gKhHp52xWHlUF/NEY0xcYDtzWwp/vUXcBG51OwibP\nAV8aY/oAg2jhz1tEOgJ3AinGmCSsC1iudDYrt3sDuOC4xx4AvjXG9AK+rf3eo3yqOOBjbTqMMbnG\nmJW1XxdjvXC06DvPRSQBuBB4xelcPE1EooBfAa8CGGOOGGMKnM3KFgFAqIgEAGHUc09Uc2aMWQDk\nH/fwxcCbtV+/CVzi6Tx8rTj4bJsOEekKDAGWOZuJx00G7gNqnE7EBt2BPOD12sNor4hIuNNJeZIx\nZjfwNJAN5AKFxph5zmZli3bGmFyw3vQBHp8j7GvFoUFtOloaEYkAPgDuNsYUOZ2Pp4jIRcB+Y0y6\n07nYJABIBl40xgwBDmPD4QYn1R5rvxjoBsQD4SJyjbNZtUy+Vhwa1KajJRGRQKzC8JYx5kOn8/Gw\nkcBYEcnCOmR4lojMcjYlj8oBcowxR/cG38cqFi3ZOcAOY0yeMaYS+BAY4XBOdtgnIh0Aaj/v93RA\nXysOPtWmQ6zRcq8CG40xzzqdj6cZYx40xiQYY7pi/dt+Z4xpse8qjTF7gV0icnQ27dlAs56B0gDZ\nwHARCav9+z6bFn4SvtZc4Lrar68DPvZ0QK+5Q9oOPtimYyRwLbBWRFbVPvaX2rvRVctwB/BW7Zud\n7cD/OZyPRxljlonI+8BKrKvxMmhhd0uLyGzgDKCNiOQADwNPAXNE5EasAjnO43noHdJKKaWO52uH\nlZRSSjWAFgellFJ1aHFQSilVhxYHpZRSdWhxUEopVYcWB+UzROQMETEicobTubiDiGSJyBvHPfZb\nEVkrIuW1zzXGofRUM6eXsiqfUduorh+woSW0ERGRIUCRMSaz9vsArIZtPwJPAEeAFcaYaueyVM2V\nFgelWggR6QJkATcaY15zOB3VzOlhJdWiiEiiiHxUOyylXESyReQ9EQk4/rCSiDxS+319H9cfs81u\nIvKWiOSJSIWIrBKR37mY1yMiUuedmIi8UdsL6uj3XWvj3yQij4lIrogUiMgnte3Ij133p8NKIvII\nVmEAeLV2G/NdyVGpY/lU+wzlEz4FCoBbgANYLdl/Q/1vhF4BvjzusT9h9crfAtY0Paw25/uBe7Ba\nZP8e+EBELjHGeKo314NYh4duwGrP/AzwFjDmBMu/AqwD3gMeBz4Dmv2hM+UcLQ6qxRCRNkAv4OLj\nXrTfrv35L5Y3xuRgdTY9uv444FKs1uY/1j78CFar9zHGmIO1j31VWzQew3ONG3caY8Yfk1sc8G8R\niTfG1OkkbIzJOaZ/VqYxZqmH8lI+Qg8rqZbkIFbzuadEZKKI9GroiiKSgjVha6ox5vljfnQB8DlQ\nWHtoKqD2xO9XwKDak9ye8Nlx36+t/dzZQ/GU+gUtDqrFMNbVFecCacCTwBYR2S4it5xsvdpj+XOB\n+Vjzp4/VFpgAVB738e/an8e6K//jHD8msqL2c4iH4in1C3pYSbUoxpjtwITaXv+DgNuBqbUnfcuO\nX752rOZcrPMTv6/nss+DwELgnycI2dBhUeW18YJq55cf5aniolSTaHFQLVLtXsQqEbkXuBFIwhr2\n9JPaAjILa9zkMGNMcT2b+hI4HVhvjKlTXFyws/ZzEtYsAmpvUBsB1BdXKUdpcVAthogMBJ4D3gW2\nYQ10uh5rKMx3QORxq9yPdWXSXUC8iMQf87NMY0we8HdgObBARKZgXS7aCutFvrsx5oYGpvcFUAhM\nF5GHgWDgPqDEtWeplD20OKiWZC/WlKx7seaDl2OdyL3IGJNeT9uMPrWfn6tnW/8HvGGMya49Wf0I\n1l3HcViHmtZhncBuEGNMgYhcBPwHmIN1ldRjWDORj89LKcfpHdJKKaXq0KuVlFJK1aGHlZRqotr7\nHk6m2uguumpmdM9BqSYQka7UvQfi+I8TtbxQymvpOQelmkBEgoCBp1hs8wkuk1XKa2lxUEopVYce\nVlJKKVWHFgellFJ1aHFQSilVhxYHpZRSdWhxUEopVcf/B+OoTV0BIo1qAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10baaa128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "w0=regr.intercept_\n",
    "w1=regr.coef_[0]\n",
    "w2=regr.coef_[1]\n",
    "plt = plot_cnt(X,y)\n",
    "x1=np.zeros([11,1])\n",
    "x2=np.zeros([11,1])\n",
    "for i in range(11):\n",
    "    x1[i]=i;\n",
    "    x2[i]=(- w0-w1*i)/w2\n",
    "    \n",
    "plt.plot(x1, x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1a12c66208>]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAERCAYAAACQIWsgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VOXVwPHfyZCwhh3ZISj7bhIR\npNQoNZHWuoA7FN+qoFYr2vYFlyLiAlVbRavWBGoFVNQKvloqbmgUTaomgAiyCBgBBQkgOyEknPeP\nmVCSQEjCzH0mM+f7+czn5pnlnnNnICfz3JnziKpijDHGHC3GdQLGGGPCjxUHY4wx5VhxMMYYU44V\nB2OMMeVYcTDGGFOOFQdjjDHleFocRORZEdkqIsuPcdsfRERFpLmXORljjCnP63cOzwHnl71SRNoD\n5wEbPM7HGGPMMXhaHFT1I2DHMW56DBgP2DfyjDEmDDg/5yAiFwLfqeoXrnMxxhjjV8tlcBGpB9wN\npFby/mOBsQD169dP6t69e5VjFhUrBw4VEV8ntsqPNcaYmi43N3ebqrY40f3E695KIpIAzFfV3iLS\nB1gI7A/c3A74Hhigqlsq2k9ycrLm5ORUOf5tLy3hzS+3MGfsQJI6Nqny440xpiYTkVxVTT7R/ZxO\nK6nql6p6iqomqGoCsAlIPFFhOBmTftmL1o3rcMPsXDbvOhCqMMYYU6N5/VHWOUA20E1ENonIdV7G\nB2hSP44Zo5MpOFTM2Fm5HCgs9joFY4wJe15/WukqVW2tqrGq2k5V/17m9gRV3RbqPLq0jOfxK/uz\n/PtdjJ+7DGtbbowxpTk9Ie3S0B4tGZ/WnYfeWkX3VvHcfE5n1ykZY6rh0KFDbNq0iYKCAtephJU6\nderQrl07YmOr9+GbqC0OADeefSqrt+zmkbdX0+WUBqT2auU6JWNMFW3atIn4+HgSEhIQEdfphAVV\nZfv27WzatIlOnTpVax/Ov+fgkojwpxF96deuEbe/vJRVW3a7TskYU0UFBQU0a9bMCsNRRIRmzZqd\n1LupqC4OAHVifaT/Kpn6tWtx/cwcduwrdJ2SMaaKrDCUd7LPSdQXB4BWjeqQMTqZrXsOctPzuRwq\nPuw6JWNMlMjMzOSCCy4A4I033uBPf/qT44z8rDgE9G/fmIdH9OXTb3Yw+V8rXKdjjIlCF154IXfc\ncYfrNAArDqVcfHpbbjz7NJ7/zwZm/+db1+kYY2qIvLw8unfvzvXXX0/v3r0ZOXIk7733HoMHD6ZL\nly589tlnfPbZZ5x11lmcfvrpnHXWWaxevbrcfp577jluueUWANatW8fAgQM544wzuOeee2jQoAHg\nf6eRkpLCpZdeSvfu3Rk5cmRIPo5vxaGM/03rxrndT2HyGyvIWhfyr1wYY1zIzoapU/3bIFm7di3j\nxo1j2bJlrFq1ihdffJGPP/6YP//5z0yZMoXu3bvz0UcfsWTJEu677z7uuuuuCvc3btw4xo0bx+ef\nf06bNm1K3bZkyRKmTZvGV199xfr16/nkk0+CdhwlrDiU4YsRHr+yPwnN6/ObFxazYfv+Ez/IGFNz\nZGfD0KEwcaJ/G6QC0alTJ/r06UNMTAy9evVi6NChiAh9+vQhLy+PXbt2cdlll9G7d29uv/12Vqyo\nePo6Ozubyy67DICrr7661G0DBgygXbt2xMTE0L9/f/Ly8oJyDEez4nAM8XVimTE6GVW4ftbn7D1Y\n5DolY0ywZGZCYSEUF/u3mZlB2W3t2rWP/BwTE3NkHBMTQ1FRERMnTuScc85h+fLl/Otf/zqpj5ke\nHcvn81FUFPzfUVYcjiOheX2eHpnIuvx93PbSUg4fthYbxkSElBSIiwOfz79NSfEk7K5du2jbti3g\nP7dwIgMHDmTu3LkAvPTSS6FM7ZisOFRgcOfm3HNBT95b+QN/ebf8ySNjTA00aBAsXAj33+/fDhrk\nSdjx48dz5513MnjwYIqLT9zwc9q0aTz66KMMGDCAzZs306hRIw+y/C/P13MIluqu51BVqspdr33J\nnM828viV/bmof9uQxzTGVN7KlSvp0aOH6zSCbv/+/dStWxcR4aWXXmLOnDm8/vrrVdrHsZ6byq7n\nENW9lSpDRJh8YW/Wbd3H+FeX0al5ffq2a+w6LWNMhMvNzeWWW25BVWncuDHPPvusp/FtWqkS4mrF\n8LdRiTRvUJuxs3LZutu6PxpjQmvIkCF88cUXLFu2jI8++ojOnb3tHG3FoZKaNajN9NHJ7C44xNjZ\nuRQcskWCjDGRy4pDFfRs05BHL+/H0o07uWvel7ZIkDEmYllxqKLze7fm9p91Zd6S75i+aL3rdIwx\nJiSsOFTDb8/tzM/7tGLqglV8sGqr63SMMSborDhUQ0yM8OfL+tGjVUNunbOEtVv3uk7JGONQXl4e\nvXv3Pun95OTkcOuttwYho5NnxaGa6sXVYvo1ydSOjWHMrBx27T/kOiVjTA2XnJzME0884ToNwIrD\nSWnbuC7PjEpi04/7uWXOYopskSBjolZRURHXXHMNffv25dJLL2X//v3k5uZy9tlnk5SURFpaGps3\nbwYgJSWFCRMmMGDAALp27cqiRYuA0gv/5Ofnc95555GYmMgNN9xAx44d2bZtG3l5efTo0YMxY8bQ\nq1cvUlNTOXDgQNCPx9PiICLPishWEVl+1HWPiMgqEVkmIq+JSI36hllyQlMevLgPi77exoNvrnSd\njjGmErI3ZjN10VSyNwavZffq1asZO3Ysy5Yto2HDhjz11FP89re/5dVXXyU3N5drr72Wu++++8j9\ni4qK+Oyzz5g2bRqTJ08ut7/Jkydz7rnnsnjxYi655BI2bNhw5Lavv/6am2++mRUrVtC4ceMjPZiC\nyetvSD8HPAnMOuq6d4E7VbVIRB4C7gQmeJzXSbn8jPas3LKbf3ySR/dW8VxxRgfXKRljjiN7YzZD\nZw2lsLiQOF8cC0cvZFD7k++v1L59ewYPHgzAqFGjmDJlCsuXL+e8884DoLi4mNatWx+5//DhwwFI\nSko6Zsvtjz/+mNdeew2A888/nyZNmhy5rVOnTvTv37/Cx58sT4uDqn4kIgllrnvnqOF/gEu9zClY\n7v55D9Zu3csf/285p7ZowBkJTV2nZIw5hsy8TAqLCynWYgqLC8nMywxKcRCRUuP4+Hh69epF9nHW\niyhpu328ltsVfY+qbMvuGj+tVAnXAgtcJ1EdtXwxPHlVIu2a1OPG2bl8tzP4L5Yx5uSlJKQQ54vD\nJz7ifHGkJKQEZb8bNmw4UgjmzJnDwIEDyc/PP3LdoUOHTrjAz9F+8pOf8MorrwDwzjvv8OOPPwYl\nz8oKm+IgIncDRcALFdxnrIjkiEhOfn6+d8lVUqN6sUwfnUxh0WHGzMxhf6EtEmRMuBnUfhALRy/k\n/nPuD9qUEkCPHj2YOXMmffv2ZceOHUfON0yYMIF+/frRv39/srKyKr2/SZMm8c4775CYmMiCBQto\n3bo18fHxQcm1Mjxv2R2YVpqvqr2Puu4a4EZgqKpWal1Or1p2V8cHq7dy7XOfM6x3K568KpGYGDnx\ng4wx1RKpLbsPHjyIz+ejVq1aZGdnc9NNN7F06dIq7aNGt+wWkfPxn4A+u7KFIdyd0+0U7hzWnSlv\nruKvLdcy7mddXKdkjKlhNmzYwOWXX87hw4eJi4tj+vTpnsb3tDiIyBwgBWguIpuASfg/nVQbeDdw\nQuc/qnqjl3mFwpghp7Jqyx4ee28NXVs2YFif1id+kDHGBHTp0oUlS5Y4i+/1p5WuOsbVf/cyB6+I\nCFMu6cM32/bxu1e+oGOz+vRs09B1WsYYUylhc0I6EtWJ9ZE+KolGdWMZMyuHbXsPuk7JmIhk7fPL\nO9nnxIpDiJ3SsA4Zo5PYtvcgNz2fS2GRtdgwJpjq1KnD9u3brUAcRVXZvn07derUqfY+nJ+QjgZ9\n2zXmkcv6ceucJdzz+nKmDu9T7gszxpjqadeuHZs2bSIcP97uUp06dWjXrl21H2/FwSMX9mvD6i27\neeqDdXRvFc//DO7kOiVjIkJsbCydOtn/p2CzaSUP/f68bvysR0vu//dKPv56m+t0jDHmuKw4eCgm\nRph2ZX9Oa1Gfm19czDfb9rlOyRhjjsmKg8ca1K7FjNFnIAJjZuWwu8AWCTLGhB8rDg50aFaPp0cm\nkrdtH7e9tJTiw/YpC2NMeLHi4MhZpzVn0oW9eH/VVh5+e5XrdIwxphT7tJJDvxrYkVWbd5P+4Xq6\nt4rnktOr/7EzY4wJJnvn4Ni9F/Zi4KlNmTD3S5Zs8LZfuzHGHI8VB8difTE8PTKJlg1rc8PsXLbs\nKnCdkjHGWHEIB03rxzFj9BnsO1jE2Nk5FBwqdp2SMSbKWXEIE91axfPYFf1ZtmkXE+Yusz4xxhin\nrDiEkdRerfhDaldeX/o9z3y43nU6xpgoZsUhzNx8Tmcu6Nuah99exXtf/eA6HWNMlLLiEGZEhEcu\n7UfvNo0Y99IS1vywx3VKxpgoZMUhDNWN85ExOom6cbW4fmYOP+4rdJ2SMSbKWHEIU60b1SX9V0ls\n2VXAb15YzKFiWyTIGOMdKw5hLKljE6YM70P2+u08MP8r1+kYY6KItc8Ic5cmtWP1lt1MX/QN3Vo1\n5OozO7hOyRgTBeydQw1wx7AenN21Bfe8vpxP1293nY4xJgp4WhxE5FkR2Soiy4+6rqmIvCsiXwe2\nTbzMqSbwxQhPXHU6HZrV46YXFrNxx37XKRljIpzX7xyeA84vc90dwEJV7QIsDIxNGY3qxjJjdDJF\nxYcZMyuHfQeLXKdkjIlgnhYHVf0I2FHm6ouAmYGfZwIXe5mTl7I3ZjN10VSyN2ZX6/GntmjAk1cn\nsuaHPdz+8lIOh/siQRkZkJbm33qtdWsQ8W+9dOaZEBvr33pt1Cho1sy/9dKECdCli3/rNVfHnJ0N\nU6f6t14T+e8llFTV0wuQACw/aryzzO0/VmY/SUlJWpNkbcjSug/UVd9kn9Z9oK5mbciq9r5mLFqv\nHSfM17+8vSqIGQZZeroq/PeSnu5d7FatSsdu1cqbuAMGlI47YIA3cVVVR44sHXvkSG/ijh9fOu74\n8d7EVXV3zFlZqnXrqvp8/m1W9f8vV9nRx1tyqfIuyNFK/I6tUSekRWSsiOSISE5+fr7rdKokMy+T\nwuJCirWYwuJCMvMyq72vawcncFlSO554fy3zl30fvCSDae7cisehtGVLxeNQWby44nEoLVhQ8ThU\n5s2reBxKro45MxMKC6G42L/NzPQmrsfCoTj8ICKtAQLbrce7o6pmqGqyqia3aNHCswSDISUhhThf\nHD7xEeeLIyUhpdr7EhEeuKQ3SR2b8Id/fsHy73YFL9FgGTGi4nEotWpV8ThUEhMrHofSsGEVj0Nl\n+PCKx6Hk6phTUiAuDnw+/zYlxZu4XqvM24tgXig/rfQIcEfg5zuAhyuzn5o2raTqn1qa8tGUk5pS\nOtrW3QU6aMp7OnDKe7p1d0FQ9hlU6emqqaneTimVKJla8mpKqcSAAaq1ank7pVRi5EjVpk29m14p\nMX68aufO3k4plXB1zFlZqlOmeDulVOIkppT8D6/ctJL47+sNEZkDpADNgR+AScD/Aa8AHYANwGWq\nWvakdTnJycmak5MTumRriOXf7eLSZ7Lo2bohc8YOpHYtn+uUjDFhTERyVTX5RPfz+tNKV6lqa1WN\nVdV2qvp3Vd2uqkNVtUtge8LCYP6rd9tG/OWy/izesJO7X1tuiwQZY4IiHM45mJP0i76tuXVoF17N\n3cSzn+S5TscYEwGsOESI24Z2Ia1XSx7891d8uKZmfZLLGBN+rDhEiJgY4dHL+9O1ZTy3vLiY9fl7\nXadkjKnBrDhEkPq1azF9dDKxvhiun5nDrgOHXKdkjKmhrDhEmPZN6/G3kYls2LGf385ZQnG4t9gw\nxoQlKw4R6MxTm3H/xb35aE0+U99c6TodY0wNZIv9RKirBnRg1ebdzPj4G7q1iuey5PauUzLG1CD2\nziGC/fGCnpx1WjPufm05ud/+6DodY0wNYsUhgsX6Ynh6ZCKtG9fhhtm5fL/zgOuUjDE1hBWHCNe4\nXhwzRidTcKiYsbNzOFBY7F1wlz3vXcY23onG19mjY7ZzDlGgS8t4Hr+yP9fPyuF/X/2Cv151OhLq\nhUKys2HoUH9L47g4WLgQBg0KbcxwiG28E42vs4fHbO8cosTQHi0Zn9ad+cs289QHa0Mf0GXP+yjp\ntx/1ovF19vCYrThEkRvPPpWL+7fhz++s4Z0VIV4Ax2XP+2jptx/tovF19vCYPW3ZHUzWsrt6Cg4V\nc0V6Nl9v3cu835xF91YNQxcsO9v/l01Kivdv913GNt6Jxtf5JI+5si27rThEoS27CrjwyY+JqxXD\nG7f8hKb141ynZIzxSFiu52DCQ6tGdcgYnczWPQe56flcCosOu07JGBNmrDhEqf7tG/PwiL58+s0O\nJv9rhet0jDFhxj7KGsUuPr0tq7bs4ZkP19G9VTy/GpTgOiVjTJiwdw5R7n/TunFu91O4919fkbV2\nm+t0jDFhwopDlPPFCI9f2Z9OzevzmxcXs2H7ftcpGWPCgBUHQ3ydWGaMTkYVrp/1OXsKbJEgY6Kd\nFQcDQELz+jw9MpF1+fu4/eWlHLZFgoyJamFTHETkdhFZISLLRWSOiNRxnVO0Gdy5Ofdc0JP3Vm7l\nz++sdp2OMcahsCgOItIWuBVIVtXegA+40m1W0Wn0oI5cNaA9T2eu4/Wl37lOxxjjSFgUh4BaQF0R\nqQXUA753nE9UEhEmX9ibAQlNGf/qMpZt2ln9nUVjy+60NKhXz7/12qhR0KyZf+sll69zRob/uc7I\n8D62Kw0bgoh/G0qqWukLcBgoPs6lCNgOvAukVmW/gX2PA/YC+cALJ7p/UlKSmtDZtqdAz5q6UM98\n8D39YdeBqu8gK0u1bl1Vn8+/zcoKfpLhFjs1VRX+e0lN9SauqurIkaVjjxzpTVyXr3N6euljTk/3\nLrYr8fGljzk+vsq7AHK0Er+Tq/rO4X5gY+AX+HPAQ8DMwHgTMBtoASwQkQsqu1MRaQJcBHQC2gD1\nRaTcnz8iMlZEckQkJz8/v4qpm6po1qA200cns7vgEGNn51JwqIqLBEVjy+5Fiyoeh9KCBRWPQ8Xl\n6zx3bsXjSLRnT8XjIKpqcSgAvgESVPU6Vb1LVa/F/0s9D3+RSATeAe6qwn5/BnyjqvmqegiYB5xV\n9k6qmqGqyaqa3KJFiyqmbqqqZ5uGPHp5P5Zu3Mld874seYdXOdHYsnvIkIrHoTRsWMXjUHH5Oo8Y\nUfE4EsXHVzwOpsq8vSi5AN8CFx7ntouAjYGfRwB7q7DfM4EV+M81CP53I7+t6DE2reSdae+u0Y4T\n5mv6h2ur9sCsLNUpU7ydanAdOzXVP73i5ZRSiZEjVZs29W5KqYTL1zk93f9cR8OUUomSqaVqTCmp\nVn5aqUotu0XkADBKVcu9fxORy4CZqlpPRM4G3lTV+lXY92TgCvznLpYA16vqwePd31p2e0dVueXF\nJby5fDPPXnMG53Q/xXVKxphqClXL7hzgXhFpXSZYG2BS4HaAjlTx00aqOklVu6tqb1X9VUWFwXhL\nRHjksr70aNWQW+csYe3W0M1zGmPCQ1WLwzigHfCNiHwgIi+LyAfAevwnkm8N3K8z8GLw0jSu1Yur\nxfRrkqkdG8P1M3PYtd9abBgTyapUHFR1Mf5f/I/i/1hrn8D2L0AXVV0auN89qjopyLkax9o2rssz\no5L4bucBbn5xMUXFtkiQMZGq0sVBRHwi0g+IUf+nlIaqas/A9m5V3R7CPE2YSE5oyoMX9+Hjtdt4\n8M2VrtMxxoRIVd45KP5zCqeHKBdTQ1x+RnuuHdyJf3ySx8ufb3CdjjEmBCpdHFT1MP4vwFX6E0gm\nct318+4M6dKcP/7fcj7P2+E6HWNMkFX1hHQ6cJuIxIUiGVNz1PLF8ORVibRrUo8bZ+fy3c4DrlMy\nxgRRVdeQjgdOA9aLyFvAZvzTTSXUTkRHj0b1Ypk+OplLnvqE62fmMPemQdSLs2XJjYkEVf0S3Ik+\nnqKq6ju5lCrHvgQXPj5YvZXrnvuctF6teOrqRGJixHVKxpjjCMmX4FQ15gQXTwqDCS/ndDuFO4f1\nYMHyLTzx/teu0zHGBEE4redgarDrh3RieGJbpr33NQu+3Ow6HXdrDLhc28AVl8c8YQJ06eLfesnl\nMZ95JsTG+rehVJkGTOF4scZ74edAYZFe/NTH2v2PC3T5dzvdJeJqjQGXaxu44vKYx48vvbbB+PHe\nxHV5zAMGlD7mAQOqvAtCtJ5DyZoKS0Rkv4gUl72EoH6ZGqJOrI/0UUk0qhvL2Fm5bNvrqD2WqzUG\nXK5t4IrLY543r+JxqLg85sWLKx4HUZWKg4iMBv4KfA7UAf4BPA/sBtYB9wU7QVOznNKwDhmjk9i2\n9yA3PZ9LYZGDFhuu1hhwubaBKy6Pefjwiseh4vKYExMrHgdRVT+ttBh4A/+KcIeAZFVdHFjJLROY\nrqpPhiLRsuzTSuHtjS++59Y5S7jyjPZMHd4HEY8/wZSd7f+LLiUFBg2K/LguuTzmCRP87xiGD4eH\nHvIurstjPvNM/zuGxET49NMqP7yyn1aqanHYg39Rnw/wF4efqOp/ArddDjyoql2qnG01WHEIf4+8\nvYqnPljHpF/25NeDO7lOxxhD6NZzOIC/8Z4CW4BTj7ptL/623cYA8PvzunFez5Y88O+VfPz1Ntfp\nGGOqoKrF4Uv8LbsBFgF3icggETkDuBdYFcTcTA0XEyM8dkV/OrdowM0vLuabbftcp2SMqaSqFocM\noEng54lAA+Bj4FOgK/D74KVmIkGD2rWYcU0yMQJjZuWwu8AWCTKmJqjqN6RfVtWpgZ/XAr2ANOBi\noLOqZgY9Q1PjtW9aj6dHJpG3bR/j5iyh+HDlz3MZY9yo1jekRaS9iJwFnIl/Jbi9QF8ROTeYyZnI\nMei0Zky6sBcfrM7n4bdt9tGYcFelFpoicirwAjCg5Cr+25W15Gfrr2SO6VcDO7J6y27SP1xPt5bx\nDE9s5zolY8xxVLW/8gygA3Ab/pPPhUHPyES0Sb/sxdqte7lj3pd0al6f0zs0OfGDjDGeq+q00hnA\nrar6V1V9V1U/LHupbiIi0lhEXhWRVSKyUkSi5BtE0SXWF8PTI5No2bA2N8zOZcuuAtcpGWOOoarF\nYROhe7fwOPCWqnYH+gG2en2Ealo/jhmjz2DfwSLGzs6h4JC15DIm3FS1OEwBJohIUNeRFpGGwE+B\nvwOoaqGq7gxmjHCQ8coE0u7rQsYrHrcXdiR7YzZTF00le2P5tsbdWsXz2BX9+fK7XUyYu4yqfFO/\nUjIyIC3Nv/VSWhrUq+ffes1V+2pXz7Xr2K549TpXpnXr0RfgQSAf+Bcwq8xlZlX3F9hnf+Az4Dlg\nCf5zG/UrekxNa9md/vJ4ZRJHLukve9Re2JGsDVla94G66pvs07oP1NWsDcdua/zk+19rxwnz9ekP\n1gYveHp66bbG6enB23dFUlNLx01N9Sauqrv21a6ea9exXQnC60woWnaLyP8AdwKNgURgyDEu1VEr\nsL+/qerpwD7gjmPEHysiOSKSk5+fX81QbsxdFWgnLGXGESozL5PC4kKKtZjC4kIy8zKPeb/fpJzG\nL/u14eG3V/HeVz8EJ/jcuRWPQ2XRoorHoeSqfbWr59p1bFc8fJ2rOq00GXgNaKGqbVW1U5nLqSfa\nwXFsAjapakmLwVfxF4tSVDVDVZNVNblFixbVDOXGiO6BdsJaZhyhUhJSiPPF4RMfcb44UhJSjnk/\nEeHhEX3p3aYR415awpof9px88BEjKh6HypAhFY9DyVX7alfPtevYrnj5Olfm7UXJBf+X3YZW5TFV\n2PcioFvg53uBRyq6f02bVlL1Ty2lTu4c8VNKJbI2ZOmUj6Ycd0rpaN/v3K/JD7yrQx56X3fsPXjy\nwdPT/dM6Xk81pKb6VwfzckqpxPjxqp07ezelVMLVc+06tisn+TpTyWmlqrbsfguYryFYs0FE+uM/\n1xAHrAd+rao/Hu/+1rI78ize8CNXpv+HpI5NmHXdAGJ9tsS5McEWqpbd44AxIjJSRJqJSEzZS/XS\nBVVdqv4po76qenFFhcFEpsQOTZgyvA/Z67fzwPyvXKdjTFSr6jekS757MOs4t2s19mnMEZcmtWP1\nlt1MX/QN3Vo15OozO7hOyZioVNVf5Pfx315KxoTEHcN6sOaHvdzz+nJOa1GfM09t5jolY6JOlc45\nhBM75xDZdh04xCVPf8LO/Yd4/ebBtG9az3VKxkSEUJ1zMMYTjerGMmN0MkXFhxkzK4d9B4tcp2RM\nVLHiYMLWqS0a8OTViaz5YQ+/e2Uph22RIGM8Y8XBhLWfdm3B3b/oydsrfmDae2tcp2NM1LBPFpmw\nd+3gBFZv2c0T76+la6t4LujbxnVKxkQ8e+dgwp6IcP/FvUnq2IQ//PMLln+3y3VKxkQ8Kw6mRqhd\ny8czo5JoWi+OMbNyyN9z0HVKxkQ0Kw7RIDsbpk71b70M++QEpl7bhewng9N3vkV8bTJGJ7Nz/yFu\nfD6Xg0XHXyQo47FRpN3WjIzHRgUldqU5eq4Bd+s5uIrrMHb2ggymPpBG9gIH60iE63oO4XKpiY33\nnMjK8jeC8/n826wTN8ELSti/jte6d6O+e9C6d6NZfw1eM7j5X3yvHSfM1z+8slQPHz5c7vb0R0eW\nXjvj0ZFBi10hR8+1qrpbz8FVXIexs95ML/1v+00Pm/6F63oOpgbKzITCQigu9m8zM70Ju3gehT4o\njoHCGP84WH7RtzW3Du3CP3M38ewneeVun/vtAv8PUmYcao6ea8Ddeg6u4jqMnZk7t/S/7VwP15EI\n4/UcTE2TkgJxceDz+bcpKd6ETRxOXDH4iiHusH8cTLcN7UJar5Y8+O+v+HBN6YWfRnQc5v9By4xD\nzdFzDbhbz8FVXIexU5JGlP63neThOhLhup5DOF1sWqkKsrJUp0zxdppD/VNLU37dOahTSkfbW3BI\n0x77UHtPekvXbd1T6rb0R0dq6rim3k0plXD0XKuqu/UcXMV1GDvrzXSdcn+qt1NKJcJxPYdwYr2V\nDMDGHfu56KlPaFwvltd+M5gLE3yUAAAU20lEQVRGdWNdp2RMWLPeSiYqtG9aj2dGJbFh+35unbOE\nYmuxYUxQWHEwNd6ATk25/+LefLgmnz8tWHniBxhjTsjaZ5iIcNWADqza/N9Fgi5Nauc6JWNqNHvn\nYCLGxAt6MrhzM+6a9yW539oqs8acDCsOJmLU8sXw1NWJtG5chxtm57J51wHXKRlTY1lxMBGlcb04\nZoxOpuBQMWNm5XCg8PgtNowxx2fFwUScLi3jeeKq/qz4fjf/++oX1NSPaxvjkhUHE5HO7d6S8Wnd\nmb9sM09nrnOdjjE1TlgVBxHxicgSEZnvOhdT89149qlc3L8Nj7y9mndWbHGdjjE1SlgVB2AcELEf\nVM/IzSBtdhoZud62+XXVXjg7rSdTz/aRndbT07gA2den8afz63PJe3fTr10jbn95Kau27A59YJft\nq121Cx81Cpo182+91rOnv5dVT+//jTmTkQFpaf5tKFWmx4YXF6AdsBA4F5h/ovvXtN5K6Tnpyr0c\nuaTneNOTxVV74azUHqXjpvbwJK6qatZ1qaVizx9ziZ7xwLs6+E8Ldfveg6EL7LJ9tat24SNHlj7m\nkR72surRo3TsHt79G3MmPb30MadX/f8zNbBl9zRgPHD4eHcQkbEikiMiOfn5+ce7W1ia+9XcCseh\n4qq9cGbB6tJxC1Z7Ehcgc+OiUrGXffsWGaOT2brnIDc9n8uh4uP+Ezs5LttXu2oXvmBBxeNQWr26\n4nEkmju34nEQhUVxEJELgK2qmlvR/VQ1Q1WTVTW5RYsWHmUXHCN6jqhwHCqu2gun1OlWOm6dbp7E\nBUhpP6R07PZD6N++MQ+P6Mun3+zg3jdWhCawy/bVrtqFDxtW8TiUunWreByJRoyoeBxMlXl7EeoL\nMBXYBOQBW4D9wPMVPaamTSup+qeWUmelejalVMJVe+Gs1B465acxnk4pHYl9XapOSa2rWdellrp+\n6psrteOE+TorOy80gV22r3bVLnzkSNWmTb2dUirRo4dqTEx0TCmVSE9XTU2t1pSSag1u2S0iKcAf\nVPWCiu5nLbtNdRQfVsbMyuGjNfnMum4AZ53W3HVKxnjKWnYbcwy+GOHxK/uT0Lw+v3lhMRu273ed\nkjFhKeyKg6pmnuhdgzEnI75OLDNGJ6MK18/6nL0Hi1ynZEzYCbviYIwXEprX5+mRiazL38dtLy3l\nsC0SZEwpVhxM1BrcuTn3XNCT91b+wF/ejYKPQRpTBbbYj4lqowd1ZNWW3Tz1wTq6toznov5tXadk\nTFiwdw4mqokIky/szYCEpox/dRnLNu10nZIxYcGKg4l6cbVi+NuoRJo3qM3YWbls3V3gOiVjnLPi\nYAzQrEFtpo9OZnfBIcbOzqXgkC0SZKKbFQdjAnq2acijl/dj6cad3DXvS1skyEQ1Kw7GHOX83q25\n/WddmbfkO6YvWu86HWOcseLgIZksRy5eSpudRr0H65E2O83TuBl3pZF2Qz0y7vI2LkDGKxNIu68L\nGa9UfV2FW4d25hd9WjN1wSo+WLW1ag9OS4N69fxbr7mK7dX6Asfiag0Ll8fs1RoWlWnAFI6XmtZ4\nj3tRJh11uRdP4qbOSi21jkTqrNQTPygI0u9MLXW86Xd6E1dVNf3l8aVjv1z1Jnj7Dh7SYdM+0t73\nvKVf/7Cncg9KTS3daz/Vu2N2FjsI6wtUm6s1LFwecxDWsKAGrucQHbx908CiDYsqHIfK3O2BOFJm\n7EXsVfNKx15V9XUV6sXVYvo1ydSOjWHMrBx27T904gctWlTxOJRcxfZwfYFyXK1h4fKYPVzDwoqD\n1zw+xzmkw5AKx6EyolkgjpYZexG7+/DSsbtXb12Fto3r8syoJDb9uJ9b5iym6ESLBA0ZUvE4lFzF\n9nJ9gbJcrWHh8pi9XMOiMm8vwvFS06aVVLXU9I6XUmelat0H6no2pVQi/c5UTR1b19MppSOxXx6v\nqZM7V2tKqayXP9ugHSfM13vfWH7iO6em+qc4vJxSch37JNcXOCmu1rBwecwnuYYFNXU9h8qy9RyM\nlyb/awX/+CSPh0b04YozOrhOx5hqs/UcjAmiu3/egyFdmvPH/1tOTt4O1+kYE3JWHIyphFq+GJ68\nKpF2Tepx4/O5fLfzgOuUjAkpKw7GVFKjerFMH53MwUOHGTMzh/2FtkiQiVxWHIypgs6nNOCJq09n\n5Zbd/OGfX9giQSZiWXEwporO6XYKdw7rzptfbuGv7691nY4xIWGL/RhTDWOGnMqqLXt47L01dG3Z\ngGF9WrtOyZigsncOxlSDiDDlkj6c3qExv3vlC776frfrlIwJKisOxlRTnVgf6aOSaFQ3ljGzcti2\n96DrlIwJmrAoDiLSXkQ+EJGVIrJCRMa5zsmYyjilYR0yRiexbe9BfvP8YgqLTtBiw5gaIiyKA1AE\n/F5VewADgZtFJMT9aI0Jjr7tGvPwpX35LG8H97y+3BYJMhEhLIqDqm5W1cWBn/cAK4G2brMKPlfr\nOSRMSyBmcgwJ0xI8jXvmxNbE3iOcOdH7k7XZd45i6sXNyL5zlCfxLurflpvPOY2XPt/IzGHXwYSq\nryNx0kaNgmbN/FsvuVpTAfzPc5cubp5vVzx6ncOiOBxNRBKA04FP3WYSXGULglcFImFaAt/u+hZF\n+XbXt54ViDMntuYz3xaKYuAz3xZPC0T2naMY6nuBif12MNT3gmcF4vfv/4Offf0f7u97MR+//La3\nv7BGjYIXXoAdO/xbrwpEdjYMHQoTJ/q3XhaICRPg4Ydh7Vr/NhoKhIevc1gVBxFpAMwFblPVch//\nEJGxIpIjIjn5+fneJ1gDbdi1ocJxqCyWLf4fpMzYA5krF1Dog+IYKIzxj70QM28e0+b/hdO2b+Lm\ni+4g7+2PPIkLwIIFFY9DxdWaCgDz5lU8jkQevs5hUxxEJBZ/YXhBVY/5Kqtqhqomq2pyixYtvE2w\nhurQqEOF41BJ1Fb+H7TM2AMpPYYRVwy+Yog77B97YvhwGhQeYMbc+xCU69NuZ3dBJRYJCoZhwyoe\nh4qrNRUAhg+veByJPHydw6Jlt4gIMBPYoaq3VeYxNbFl99FTSTrJu+c9YVoCG3ZtoEOjDuTdludZ\n3DMntmaxbCFRW/Hp/Zs9iwv+qaXMlQtI6TGMQVOf9y7whAkwbx5Zw3/NaF9/ftq1BdNHJ+OL8WAa\ncdQo/1+Sw4bB8x4ec3a2/x1DSgoMGuRdXDjyfDN8ODz0kLexXTnJ17myLbvDpTj8BFgEfAmUfBbw\nLlV983iPqYnFwUSX2f/5lon/t5wbzj6VO4f1cJ2OMUDli0NYtM9Q1Y/xfHVlY0LrVwM7smrzbtI/\nXE/3VvFccno71ykZU2lhc87BmEh074W9GHhqUybM/ZKlG3e6TseYSrPiYEwIxfpieHpkEqfE12bs\nrBy27CpwnZIxlWLFwZgQa1o/jhnXJLPvYBE3zM6h4FCx65SMOSErDsZ4oHurhjx2RX++2LSLCXOX\nWYsNE/asOBjjkdRerfhDaldeX/o9z3y43nU6xlTIioMxHrr5nM5c0Lc1D7+9ioUrf3CdjjHHZcXB\nGA+JCI9c2o/ebRox7qWlrPlhj+uUjDkmKw7GeKxunI+M0UnUifVx/cwcftxX6DolY8qx4uAhmSzI\nJO9bdsvEQNyJ3sb1BeL6PI4LkDEigbTRMWSMSPA2cCXbV7duVJf0XyWxZVcBN7+4mEPFQVgkyFHr\n7OwFGUx9II3sBRmexnUd25m0NKhXz78NobBon1EdNa19hkyWI03o/Fd4019JJgr4jrqiGPT+0Mf1\nTRQOHxU3phiKPYgL/sJwQ59vj4zTv+zI2Ll5oQ9c0r66sNDfhG7hwhP2Gno1dxN/+OcXXDOoI5Mv\n6u1p7GDIXpDB0E9uoNAHccWwcHA6g4aNDXlc17GdSUuDd9757zg1Fd5+u0q7qGz7DHvn4JWSPwyl\nzDjUSl5hKTMOscNl4h728F/a3PobSsU+Mg61arSvvjSpHWOGdGJm9re8+OlJ5OmodXZm7tzS7dFz\n53oS13VsZxYtqngcRFYcvFLyTGuZcaiVFCEtMw6xmDJxYzxcWnnEvg6lYh8Zh1o121ffMawHZ3dt\nwT2vL+fT9ds9jX2yUpJGlG6PnjTCk7iuYzszZEjF4yCyaSUPyWTx/3KO8bZlt0wUfzE67M2UUgnf\nROFwjL8weDWlVCJjRAJz629gxL4O3kwplahm++pdBw5xydOfsHP/IV6/eTDtm9bzLPbJyl6QQWbu\nXFKSRng+reMytjNpaf53DEOGVHlKCWpYy+7qqInFwZiKrM/fy0VPfULbxnWZe9NZ1K8dFk2TTYSx\ncw7G1DCntmjAU1cnsuaHPfzulaUcPlwz/3AzkcGKgzFh5KddW3D3L3ry9oofmLbwa9fpmChm71uN\nCTPXDk5g1ebdPLHwa7q1jOcXfVu7TslEIXvnYEyYEREeuKQ3SR2b8Pt/LmX5d7tcp2SikBUHY8JQ\n7Vo+nhmVRNN6cYydlUP+noOuUzJRxoqDMWGqRXxtMkYns2N/ITc+n8vBIlskyHjHioMxYax320b8\n5bL+5H77I398bbktEmQ8Y8XBmDD3i76tufXczvwzdxPPfpLnOh0TJcKmOIjI+SKyWkTWisgdrvMx\nJpzc9rOupPVqyYP//oqP1uS7TsdEgbAoDiLiA54ChgE9gatEpKfbrIwJHzExwqOX96dry3hueXEx\n6/P3uk7JRLiwKA7AAGCtqq5X1ULgJeAixzkZE1bq167F9NHJ1PLFcP2sHHYdOOQ6JRPBwqU4tAU2\nHjXeFLjOGHOU9k3r8beRiWzYvp9b5yyh2FpsmBAJl29IH2upsHL/6kVkLFDSenGviKyuZrzmwLZq\nPramsmOOMOuAWdeVuzqij/k47JirpmNl7hQuxWET0P6ocTvg+7J3UtUM4KTXAxSRnMp0JYwkdszR\nwY45OnhxzOEyrfQ50EVEOolIHHAl8IbjnIwxJmqFxTsHVS0SkVuAt/GvePysqq5wnJYxxkStsCgO\nAKr6JvCmR+FOemqqBrJjjg52zNEh5MdcY1eCM8YYEzrhcs7BGGNMGIm64hBtbTpEpL2IfCAiK0Vk\nhYiMc52TF0TEJyJLRGS+61y8ICKNReRVEVkVeK0Huc4p1ETk9sC/6eUiMkdE6rjOKdhE5FkR2Soi\ny4+6rqmIvCsiXwe2TUIRO6qKQ5S26SgCfq+qPYCBwM1RcMwA44CVrpPw0OPAW6raHehHhB+7iLQF\nbgWSVbU3/g+yXOk2q5B4Dji/zHV3AAtVtQuwMDAOuqgqDkRhmw5V3ayqiwM/78H/SyOiv30uIu2A\nXwAzXOfiBRFpCPwU+DuAqhaq6k63WXmiFlBXRGoB9TjGd6NqOlX9CNhR5uqLgJmBn2cCF4cidrQV\nh6hu0yEiCcDpwKduMwm5acB44LDrRDxyKpAP/CMwlTZDROq7TiqUVPU74M/ABmAzsEtV33GblWda\nqupm8P/xB5wSiiDRVhwq1aYjEolIA2AucJuq7nadT6iIyAXAVlXNdZ2Lh2oBicDfVPV0YB8hmmoI\nF4F59ouATkAboL6IjHKbVWSJtuJQqTYdkUZEYvEXhhdUdZ7rfEJsMHChiOThnzY8V0Sed5tSyG0C\nNqlqyTvCV/EXi0j2M+AbVc1X1UPAPOAsxzl55QcRaQ0Q2G4NRZBoKw5R16ZDRAT/XPRKVX3UdT6h\npqp3qmo7VU3A//q+r6oR/Relqm4BNopIt8BVQ4GvHKbkhQ3AQBGpF/g3PpQIPwl/lDeAawI/XwO8\nHoogYfMNaS9EaZuOwcCvgC9FZGngursC30g3keO3wAuBP3rWA792nE9IqeqnIvIqsBj/J/KWEIHf\nlBaROUAK0FxENgGTgD8Br4jIdfiL5GUhiW3fkDbGGFNWtE0rGWOMqQQrDsYYY8qx4mCMMaYcKw7G\nGGPKseJgjDGmHCsOJmqISIqIqIikuM4lGEQkT0SeK3PdL0XkSxEpCBxrY0fpmRrOPspqokagQV1P\n4KtIaCEiIqcDu1V1XWBcC3+TtixgClAIfK6qxe6yNDWVFQdjIoSIdATygOtU9VnH6ZgazqaVTEQR\nka4i8lpggZQCEdkgIv8UkVplp5VE5N7A+FiX/zlqn51E5AURyReRgyKyVEQuqWJe94pIub/EROS5\nQB+oknFCIP4NInKfiGwWkZ0i8q9AK/KjH3tkWklE7sVfGAD+HthHZlVyNOZoUdU+w0SF+cBO4CZg\nG/6W7D/n2H8IzQDeKnPdH/D3x18D/pX08Lc43wrcjr819hXAXBG5WFVD1ZvrTvzTQ9fib8n8F+AF\n4Ozj3H8GsBz4J/AA8G+gxk+dGXesOJiIISLNgS7ARWV+ab8YuL3U/VV1E/6OpiWPvwwYjr+teVbg\n6nvxt3o/W1W3B657O1A07iN0jRu/VdWrj8qtBfCIiLRR1XKdhFV101G9s9ap6n9ClJeJEjatZCLJ\ndvxN5/4kImNEpEtlHygiyfhX1XpaVZ846qbzgTeBXYGpqVqBE79vA/0CJ7lD4d9lxl8Gth1CFM+Y\nUqw4mIih/k9XnAfkAFOBNSKyXkRuquhxgbn8N4BM/GtPH+0UYDRwqMzlkcDtzYKVfxlll4Y8GNjW\nCVE8Y0qxaSUTUVR1PTA60OO/H3AL8HTgpO+BsvcPLKf5Bv7zE1cc42Of24FFwEPHCVnZxaIKAvHi\nAuuXlwhVcTHmpFhxMBEp8C5iqYj8DrgO6I1/sacjAgXkefzLTA5Q1T3H2NVbwCBghaqWKy5V8G1g\n2xv/GgQEvqB2FnCsuMY4ZcXBRAwR6Qs8DrwMrMW/oNP/4F8M5n0gvsxDJuD/ZNI4oI2ItDnqtnWq\nmg/cA3wGfCQiT+L/uGgT/L/kT1XVayuZ3gJgFzBdRCYBtYHxwN6qHaUx3rDiYCLJFvwrY/0O//rg\nBfhP5F6gqrnHaJvRPbB9/Bj7+jXwnKpuCJysvhf/t45b4J9qWo7/BHalqOpOEbkAeAx4Bf+npO7D\nvxZy2byMcc6+IW2MMaYc+7SSMcaYcmxayZiTFPjeQ0WK1d6imxrG3jkYcxJEJIHy34Eoezleywtj\nwpadczDmJIhIHND3BHdbfZyPyRoTtqw4GGOMKcemlYwxxpRjxcEYY0w5VhyMMcaUY8XBGGNMOVYc\njDHGlPP/OXXUyqqIg9AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b7d47f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create the scatter plot\n",
    "plt.plot(X[Imal,0],X[Imal,1],'r.')\n",
    "plt.plot(X[Iben,0],X[Iben,1],'g.')\n",
    "plt.xlabel(xnames[0], fontsize=16)\n",
    "plt.ylabel(xnames[1], fontsize=16)\n",
    "plt.ylim(0,14)\n",
    "plt.legend(['malign','benign'],loc='upper right')\n",
    "x1=np.zeros([11,1])\n",
    "x2=np.zeros([11,1])\n",
    "for i in range(11):\n",
    "    x1[i]=i;\n",
    "    x2[i]=(- w0-w1*i)/w2\n",
    "    \n",
    "plt.plot(x1, x2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training data using two features = 0.922401\n"
     ]
    }
   ],
   "source": [
    "yhat=regr.predict(X)\n",
    "yhati= (yhat >=0).astype(int)\n",
    "acc = np.mean(yhati == y)\n",
    "print(\"Accuracy on training data using two features = %f\" % acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training data using 10 features = 0.960469\n"
     ]
    }
   ],
   "source": [
    "xnames = ['thick','size_unif','shape_unif','marg','cell_size','bare',\n",
    "         'chrom','normal','mit']\n",
    "X = np.array(df[xnames])\n",
    "\n",
    "Xs = preprocessing.scale(X)\n",
    "regr.fit(Xs,y1)\n",
    "yhat=regr.predict(Xs)\n",
    "yhati= (yhat >=0).astype(int)\n",
    "acc = np.mean(yhati == y)\n",
    "print(\"Accuracy on training data using 10 features = %f\" % acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>slope</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>thick</td>\n",
       "      <td>0.089390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>size_unif</td>\n",
       "      <td>0.066909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shape_unif</td>\n",
       "      <td>0.046706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>marg</td>\n",
       "      <td>0.023596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cell_size</td>\n",
       "      <td>0.022381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bare</td>\n",
       "      <td>0.165260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>chrom</td>\n",
       "      <td>0.046940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>normal</td>\n",
       "      <td>0.056522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>mit</td>\n",
       "      <td>0.001695</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      feature     slope\n",
       "0       thick  0.089390\n",
       "1   size_unif  0.066909\n",
       "2  shape_unif  0.046706\n",
       "3        marg  0.023596\n",
       "4   cell_size  0.022381\n",
       "5        bare  0.165260\n",
       "6       chrom  0.046940\n",
       "7      normal  0.056522\n",
       "8         mit  0.001695"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'feature': xnames, 'slope': np.squeeze(regr.coef_)}\n",
    "dfslope = pd.DataFrame(data=data)\n",
    "dfslope"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Functions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This demo illustrate an important classifier known as *logistic regression*.  In logistic regression, we do not just output an estimate of the class label, `yhat`.  Instead, we ouput a *probability*, an estimate of how likely the sample is one class or the other.  That is our output is a number from 0 to 1 representing the likelihood:\n",
    "$$\n",
    "    P(y = 1|x)\n",
    "$$\n",
    "which is our estimate of the probability that the sample is one class (in this case, a malignant sample) based on the features `x`.  This is sometimes called a *soft classifier*.  \n",
    "\n",
    "In logistic regression, we assume that likelihood is of the form\n",
    "$$\n",
    "    P(y=1|x) = f(z),  \\quad z = w_0 + w_1x_1 + \\cdots + w_kx_k,  \n",
    "$$\n",
    "where $w_0,\\ldots,w_k$ are the classifier weights and $f(z)$ is the so-called *logistic* function:\n",
    "$$\n",
    "    f(z) = \\frac{1}{1+e^{-z}}.\n",
    "$$\n",
    "\n",
    "To understand the logistic function, suppose $x$ is a scalar and samples $y$ are drawn with $P(y=1|x) = f(w_1x)$ for some scaling constant $w_1$.  We plot these samples for different $w_1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXmclNWZ779PNQ3d7AiICsgiO6iA\nBmKMEaMGTCZqVpcsZpnh3kycyR1zk9EkkzEZP9kzk5n5ZOMmcUzMZgwxaDBoIm3ELSB7s8giS9s0\nSwPddEMvVXXuH6equ2h67/fUqTr1fD+f/pyqt06976+eqvr1U88573nFGIOiKIoSLjHfAhRFURS3\nqNEriqIEjhq9oihK4KjRK4qiBI4avaIoSuCo0SuKogSOGr2iKErgBGv0IvJvIrJFROIicn8f9nOn\niOwXkXoReUxEzuukr0n1q0v9/ai3x81looitiFwoIitEpDIVt4ld9J8oIqtF5LSI7BCRG3pz3Fwn\noti+Q0TWiMhJEakSkf8nIkM66R9sbLPxWRWRASLyExGpTcX7ni7290+pfjWp5w3oja6eEKzRA7uB\nzwJ/6O0ORGQ28EPgQ8AY4DTwvS6edrkxZnDq7297e+wcp8+xBZLAH4H3dLP/L4ENwEjg88CjIjK6\nD8fPVaKI7TDgAeAiYCYwDvhmJ/1Djm02Pqv3A1OBCcB1wGdFZEl7HUVkMXAvcD0wEZgMfKkP2rqH\nMSav/oCPAo9n3N8NPJJx/yAwN+P+w8D9vTzWV4BfZNy/BGgChnTQ3wBTfMcoH2KbsY9+qbhN7KTP\nNKAxM+7Ac8D/9h2zXI5txr7eDWwJKba59FkFXgfelnH/34BfdbCPXwBfybh/PVDlOl75mNE/C1wj\nIjERuRAoBq4GEJHJwGBgc2c7EJE3p37WdvT35lTX2cCm9POMMXuwRj+tk93/JfWzbHlX5YgcJJux\n7Qmzgb3GmFMZ2zaltucLPmP7FqC8g8fyNbY58VkVkRHYX06bMjZ3Fr/Z7fQdIyIjuzpWX+jncucu\nMMbsFZFTwFys4a4C5orIDOAq4DljTLKLfawBhnfjcIOBmjbbaoCO6p3XAi8BA7E/nZ8QkbnGmHg3\njuWdLMe2J3T0PoyN+DjO8BVbEbkRuAtY2EGXvIxtDn1WB6fazBh25hFt452+PQSo7qOWDsk7o0/x\nLLAImJK6fRJrslel7kdFHTC0zbahwKl2+mKM+UvqZpOIfAqoxdZIt0SoyTXZim1P6NH7kMNkNbYi\n8kZsqeC9xphXO+iWz7HNhc9qXaodCjRk3O4ofm3jnb7tNN75WLqB1jf4mtTtZ7Fv8LV04w0WkWsy\nZsa093dNqms5cHnG8yYDA4COvjRtMYB090XlCNmKbU8oBya3mTlyOR2XI3KVrMVWROYBK4CPGWP+\n3Mlu8zm23j+rxpgTwCEyfILO41feTt/Dxhhn2TyQf4OxqQGMadj/gLtT94cCx7EZdFFqWzFQgs1o\nHkjdLurhcWan9nkNMAg7oNPRIMts7M/IIuzPs+8AO4Fi3/HKxdim9lOSiqsBpgMlnfR9CfhW6jnv\nwmZvo33HKxdjC8wBDgO3dbN/XsY2Vz6rwNew/1hGADOwxr+kg/0sAaqAWan+zwBfcx4r329WH97k\nQ8CDGffXAU9m3P+f1JuS+feRXhznTuAAUA/8Hjgv47Engc+lbr81Zez1wBHgMWCq7zjleGzb7sNk\nPPYD4AcZ9ycCZcCZVJxv8B2nXI0t8CB2SmBdxl95iLHNkc/qAOAn2H8wh4F7Mh67OBX/izO23ZPq\nV5t6rwa4jpOkDqwoiqIESr7W6BVFUZRuokavKIoSOGr0iqIogaNGryiKEjjeTpgaPny4mTJliq/D\nn0N9fT2DBg3yLeMsXnnllWPGmB4vLpVrsYXci29vYwu5F99ciy2E89kNJbbejH7MmDGsW7fO1+HP\noaysjEWLFvmWcRYisr83z8u12ELuxbe3sYXci2+uxRbC+eyGElst3SiKogSOGr2iKErgqNEriqIE\njhq9oihK4KjRK4qiBI4avaIoSuCo0SuKogSOGr2iKErgqNEriqIEjhq9oihK4KjRK4qiBI4avaIo\nSuCo0SuKogROl6tXishPgL8Bjhhj5rTzuAD/CbwdOI298O76qIUmqz8IQGzkwz1/Xnw79Jt59gOZ\n25pfgfjdJA9/GsxpkIGtj8W3220AJIAiKL6C2MiHSR6+wmoa84o9TvNfbTcZknpOwt4GMKdsW7zA\nPjezf2qfVkt6W/6RGeuW15gmcT3J6h/Z2/Httk3HtfgK+x6k4978it0uA1P9TmUcpYiW94FE6vkL\nWp+f2T8d+4BJHr4C4h+zsU3FNTbmFbs9/VlOfxYpsk+SgcTGvNL6/JbHsX0yn9PyWSZjOy3vVcv7\n3PI9Se8nEBKvtfpCOh79Zrb6x1n+kIEMzPjctsb9HFr2cYrW2Ld5XuY+2/pYN+lORv8/wJJOHr8J\nmJr6Wwp8v1dKFEVRFCd0mdEbY/4iIhM76XIL8FNjjAFeEpHhInKhMeZQFAJbssJUptvdzL41yzh1\n1vPP4qxtprWvOdVJZp2A5r+SrJrWeqyM2y3Pb+926pjn9E/tM19pL9bJqpmcld2ZN3b8GtPb28a9\nbewgY5+t+040rKOmdgh19QM5faaUxqaxNDf3I54oIh7vRzIpJE0MY4RkMp3bRP6jM6ucnYmbs+J2\n1ufrrBgmMAZOnIhx5NUl1NUPpLFpKs3xfiQSRRgj9immJ0o+grWRS9ts39WTneQcrZ/pj4I5RXNz\nPzaWX0pN7ZBUfNKvt+3r7g1R7KNzorjwyFjgYMb9itS2c4xeRJZis35Gjx5NWVlZ13tPXG9b88bU\nTlJXeynq4rmJ68FcTXd/StadGcNftv5jt/pmg6amGPBJ3zJyini8iBfXzeO5l65kY/lM9uybQEXl\nBVSfGI4xPR1u+qkTjbmIMfCHPy3iJ794H2tevpLqEyOycNSHsnCM7BCPF7H4tgd59sWFvqWk6Hls\nozB6aWdbuzmBMWYZsAxg+vTppntXbrF9el6jz3heN2r0f9l6N2+59EFvNfqKozfydNkc1rxwHus2\nXsr2XZeQL0affk86rdHLIFtLhx7X6I8fj/EfP/woP/r5+zlybBQAl0zcz7RLXmPh/I2cf34JI4bs\nZPCQZkpLGinpX0NxcTP9+vWnX78EsViSmBhEDLFYEhHDoludh8UpZ9XYERvbdmr0jU3D+NAn7+e3\nTyxh7IVV3HLTn5k17QAXTLmHoUOh+PQnKe5XR1GsGREQEZAShDNAMlUzbrAHlZKzbxdNIDbsX0nW\nfAkS+1OPJQF4yy3ZjUfUtPjM1gf5+fL38+yLC/nSZ77DtVdvIVZ8sX29RRMyXncbpCTDN2Kt29rS\nso/Ttl97z8vcZ9EE3vKOnr+eKIy+AhifcX8cUBnBfoPnWPUIHn70Fn6x/GZe2WzHuUeOOMGC+Zt4\n59ue4av/5VlgDvDo49fzic/ex4mTw3jn257hzvc8wQ3XrGHE8Hp0MLZrln76C/z2iSV89fPf5J/+\n108pLo6nBmPvASB5+KU+DMY2ERsJyepd4Q7GAq/umUC/fs187v98n1jRYOjXmEpomiIYjG3q4WBs\nU+9ehDGmyz9gIrC1g8feATyJzezfCPy1O/ucNm2aySVWr16dtWMdPGjMJz5hzIABxoAxV15pzNe+\nZszmzcYkk639gHWmG7Fs+5drsTWmd/H96ldtfBYutLGJkt7G1uRgfDuK7R//aOP3xS9mV48x4Xx2\nV69ebe66y5jx430raaU3se3O9MpfYusgo0SkAvhXoDj1T+IHwErs1Mrd2OmVH+3dv5zwaW6Gb34T\nHngA4nG46y741KdgzjmTVpVly+C+++DOO+HBB6F/f9+K8o8HHoBJk+Bzn/OtJL95/XW46CLfKvpG\nd2bd3NHF44Z8KSZ7ZO9euO02WLcO3vMe+Na3YOJE36pyk82b4e67YckSeOgh6BdFgbHA2LYN1qyB\nb3wDBgzwrSa/qayE6dN9q+gb+hXKAs89B7feCskkPPqoNXqlfZJJ+Nu/hREj4OGH1eR7y/LlIAIf\n/rBvJflPZSVcd51vFX1Dv0aOefppuPlmmDAB/vAHuOQS34pym1//GtauhZ/+FEaO9K0mf3nqKZg/\nH8aM8a0kv2loiHHyZP6XbnStG4e88ALccov92bdmjZp8VxgDX/mKHbP4wAd8q8lf6urgxRfhxht9\nK8l/qqvt4FC+G71m9I7Yv9+a/NixNqsfNcq3otznT3+CrVttXT6mKUiv2bjRDva/+c2+leQ/DQ12\niuOQPJ+pq18nBzQ1wfveZ2fZ/OEPMHq0b0X5wY9/DOedZwetld6zYYNt5871qyME4nF7PmhxsWch\nfUQzegd86Uu2zvzb38K0tsvaKO1SWwuPPQZLl+oskb6ycaNNLvK93JALJBI2F873SQGa0UfMxo3w\n9a/DRz4C7363bzX5w8qV0Nio2XwUbNpks3lpb3ESpUckEmFk9Gr0EZJMwt//va3H//u/+1aTXyxf\nDhdcAFdd5VtJfmMM7NoFM2b4VhIGWrpRzuHXv7azHR580M4DV7pHImEHrN/9bh2E7SvV1bYMNnmy\nbyVhEIrR69cqIpqb4QtfgMsv15NUesq6dXDypE4HjIK9e22rRh8N6dJNvtfo81x+7vCzn9kv2eOP\na1baU/78Z9tef71fHSGgRh8tmtErLSQS8NWvwhVXwDt6sVZ0ofP88zBrlk5DjYI9e2yrRh8N6Vk3\n+W70mtFHwIoVsHs3PPKIznToKcmkHdfQ9X+i4cAB+w9zYDtLnys9J5TSjWb0EfBf/2XXsnnXu3wr\nyT927oQTJ+BNb/KtJAwOHYILL/StIhy0dKMAdjnYsjL4xCfy/7++D9ats+2CBX51hIIafbSo0SuA\nPW2/uBg+qpdb6RXr10Npqc77jgo1+mhRo1dobrbL6d58M5x/vm81+cn69XZKalFR132Vzkkm4fBh\nNfoo0Rq9wqpVcOyYXe5A6TnG2CUj5s3zrSQMqqvtqpVq9NERyqwbNfo+8POf24tjLF7sW0l+UlFh\nz+K89FLfSsLg0CHbqtFHh5ZuCpzTp+20yve+N/8/BL4oL7ft7Nl+dYRCVZVtL7jAr46Q0NJNgfPk\nk9bs3/9+30ryl7TRz5rlV0coHDtmWz3xLDo0oy9wfvc7W7Z5y1t8K8lfysvtILZefSsaqqttq9fa\njY5EQhDJ/8kCavS9IH3lqHe+M/9/0vlk506YOdO3inBIG/3w4X51hEQiIUF8x9Xoe8GaNXa1xZtv\n9q0kv3n1VZg61beKcDh+3Jp8CMaUK8Tjsbwv24Aafa9YuRL699dldfvCyZO2pqxGHx3V1Vq2iZp4\nXNToC5WVK+Haa2HwYN9K8pddu2yr19SNjuPH7cXVlehIJNToC5KKCru+zZIlvpXkN2mj14w+OjSj\nj554XGv0BcnTT9tWyzZ9Q9dNj57qas3oo6agMnoRWSIiO0Vkt4jc287jF4vIahHZICKbReTt0UvN\nDZ5+2p6QMmeObyX5zWuv2TM4S0t9KwmH48c1o4+aghmMFZEi4LvATcAs4A4RaXuKyxeAR4wx84Db\nge9FLTQXMAaeeQbe+la9wEhf2bcPJk70rSIcEgk7wK0ZfbQU0vTKBcBuY8xeY0wT8CvgljZ9DDA0\ndXsYUBmdxNxhxw67OuB11/lWkv+89hpMmuRbRTjU1Nh2xAi/OkIjlFk33flfNRY4mHG/AljYps/9\nwFMi8g/AIOCG9nYkIkuBpQCjR4+mrKysh3LdUVdX16We3//+ImAapaUvU1Z2Jiu6QiQeh4MH1eij\npLbWtsOG+dURGqHU6Ltj9O0VKUyb+3cA/2OM+baIXAX8TETmGGOSZz3JmGXAMoDp06ebRYsW9UKy\nG8rKyuhKzw9/CBddBHfeuVBLN32gosKWGrR0Ex3pjH7o0M77KT2jkEo3FcD4jPvjOLc083HgEQBj\nzItACRDUCibGwHPP2bVt1OT7xoEDtp0wwa+OkNCM3g2hlG66Y/RrgakiMklE+mMHW1e06XMAuB5A\nRGZijf5olEJ9s28fvP46XHONbyX5T9rox4/vvJ/SfTSjd0PBzLoxxsSBu4FVwHbs7JpyEfmyiKRX\ne/k08Hcisgn4JfARY0zb8k5e88ILtr36ar86QuBgasRHjT460kavGX20FFKNHmPMSmBlm21fzLi9\nDQjaAp9/HoYM0fnzUXDwoJ0GOGiQbyXhkC7daEYfLXpmbIHxwguwcGH+r0udCxw8qNl81GhG74ZQ\nMno1+m5QXw9btsBVV/lWEgZq9NFTW2uXJy4p8a0kLNToC4hXXoFk0mb0St+pqIBx43yrCIuaGpvN\n64ywaNHSTQHx8su2XbDAr44QaGiwi2+NHetbSVjU1mp93gUFM+tGgb/+1Z7FqRdd7juHDtlWjT5a\n0hm9Ei1auikg1q6FN7zBt4owqEydanfRRX51hIZm9G4opDNjC5qjR2H/fjX6qFCjd8OpU2r0LlCj\nLxDWrbPtFVf41REKr79uWzX6aKmr00tbusAYiAXgkgG8BLesX2/b+fP96giFykoYMEDXTY8aNXo3\nJJOiRl8IrF8PU6boQFdUVFbaK0vpNMBoUaN3QzKpGX1BsH69ZvNRUlVljV6JDmPU6F1hjARxNrwa\nfSccP25XrVSjj46qKnvNXSU6mppiJJNq9C7QjL4A2LTJtvPm+dUREprRR8+ZMzblVKOPHmO0Rh88\nGzbYdu5cvzpCoanJnhWrGX20qNG7I5FQow+ejRvtNMDzz/etJAyOHLGtGn20qNG7w5gwVqxVo++E\njRvh8st9qwiHqirbqtFHixq9O3R6ZeA0NcH27Wr0UaJG74a00Q8Z4llIgOgJU4GzYwfE42r0UZI2\n+jFj/OoIDc3o3WCMDsYGT3rGzWWX+dUREocP21bHPKLl9Gk1ehckk7bVGn3AbN5sT9WfNs23knA4\nfNieYaxXQYoWzejdkDZ6zegDZssWmDmTIFauyxWOHNFs3gVq9G5Qoy8AtmyBSy/1rSIsDh/W+rwL\nzpwpQgRKS30rCQs1+sCprraLb6nRR4tm9G5oaChi4EBdKC5qEgnbqtEHytattlWjjxbN6N3Q0FDE\noEG+VYSHDsYGjhp99MTjQnW1ZvQuaGiIqdE7QEs3gbN1KwwfrldBipKaGnuFZTX66EmXbpRoUaMP\nnK1bYfZsrXlGyYkT1ui1dBM9WrpxQ8HV6EVkiYjsFJHdInJvB33eLyLbRKRcRH4RrczsYQyUl1uj\nV6KjpqY/AKNHexYSIFq6cUNINfouZ4mLSBHwXeBGoAJYKyIrjDHbMvpMBe4DrjbGnBCRvP2BfugQ\nnDgBc+b4VhIW6YxejT56NKN3Q6GVbhYAu40xe40xTcCvgFva9Pk74LvGmBMAxpgj0crMHttS/740\no4+Wkye1Ru8KrdG7ISSj7855n2OBgxn3K4CFbfpMAxCR54Ei4H5jzB/b7khElgJLAUaPHk1ZWVkv\nJLuhrq6OsrIyHntsLDCVkyefp6ys2besYKip6U9REYwY4VtJeGjpxg2FZvTtDUmadvYzFVgEjAOe\nE5E5xpiTZz3JmGXAMoDp06ebRYsW9VSvM8rKyli0aBG/+AWMHAnvetfVOhgbISdOFDNqVBhfmlxD\nSzduKLTB2ApgfMb9cUBlO31+b4xpNsa8BuzEGn/esW0bzJqlM26ipqamWOvzjtDSjRtCGoztjtGv\nBaaKyCQR6Q/cDqxo0+cx4DoAERmFLeXsjVJoNtAZN+44caK/Gr0D4nFobtbSjQtCKt10+RKMMXHg\nbmAVsB14xBhTLiJfFpGbU91WAdUisg1YDXzGGFPtSrQrqqrg5Emb0SvRohm9G06ftq0affSEZPTd\nWoTXGLMSWNlm2xczbhvgntRf3qIzbtxx8qQavQvq622rRh89hVajLxjSRj9zpl8dodHcDHV1avQu\nSBu91uijp9Bq9AXDtm12jRu9eHW0VKeKeGr00aOlG3eEVLoJ4CVEx/btOuPGBUeP2laNPnq0dOMO\nNfpASU+tVKJFjd4davTu0Bp9gNTUFHP0qNbnXZA2+lGj/OoIkXTpRmv00aMZfYDs32+/KWr00XPs\nmG01o48eNXp36GBsgKSNXks30ZPO6EeO9KsjRHQw1h2a0QfIgQMDGTgQxo/vuq/SM44ehSFDmunX\nrbM2lJ6g0yvdoUYfIPv2DWLGjDDe1Fzj2DEYPlxXAnWBlm7coYOxAXLgwECtzzvi6FEYNkyN3gVp\noy8p8asjRLRGHxh1dXDkSIkavSOOHoXhw5t8ywiS+nooKUkEkXXmGlq6CYwdO2yrRu+GY8c0o3fF\n6dMwYEDCt4wgUaMPjO3bbatGHz3GqNG75PRpKClJ+pYRJFqjD4zt26GoKMmUKb6VhEdNjV0zXQdj\n3ZAu3SjRozX6wNi+HcaOPUNxsW8l4ZGeQ68ZvRts6UYzehdo6SYwtm+HCRNO+5YRJOmzYjWjd4Mt\n3WhG7wI1+oBoaoLdu9XoXdGa0eusGxdo6cYdavQBsWuXHXS5+OJ631KCJG30mtG7QUs37tDB2IBI\nz7jRjN4NWqN3i5Zu3KGDsQGxfbu90Mj48Wr0Ljh2zJ6er1MA3aDTK92hpZuA2LYNJk6E0lL9srjg\n6FFdntgl9fV6wpQr1OgDQq8q5RY1encYoxm9S7RGHwiJBOzcqWfEuuToUb2ylCsaG23WqTV6N2iN\nPhBee81+WdTo3XHsmGb0rkivRa+lGzdo6SYQtm2z7ezZfnWEjJZu3JE2eh1fcoMafSCkjV4zejec\nOWPNSEs3bmhdi14zehdojT4Qtm2DceNg6FDfSsIkPYf+/PP96giVdEavRu+GgsvoRWSJiOwUkd0i\ncm8n/d4rIkZEroxOojvKy3XGjUuOHLGtlm7c0Gr0WrpxQUENxopIEfBd4CZgFnCHiJxjjyIyBPhH\n4OWoRbogmbQnS6nRu0MzerdoRu+WQsvoFwC7jTF7jTFNwK+AW9rp92/AN4CGCPU5Y98+W0PWgVh3\naEbvFq3RuyUko+/XjT5jgYMZ9yuAhZkdRGQeMN4Y84SI/N+OdiQiS4GlAKNHj6asrKzHgqPihRdG\nApfS2LiesrJa6urqvOoJkcyM/uDBzvsqPUczereENBjbHaOXdraZlgdFYsB/AB/pakfGmGXAMoDp\n06ebRYsWdUukC156ybYf/OB8hg2DsrIyfOoJkSNHYMAAGDzYt5Iw0Rq9WwqqRo/N4Mdn3B8HVGbc\nHwLMAcpEZB/wRmBFrg/IlpfD2LEwbJhvJeGSnkMv7aUKSp/R0o1bQirddOclrAWmisgkEekP3A6s\nSD9ojKkxxowyxkw0xkwEXgJuNsasc6I4IrZuhUsv9a0ibI4c0YFYl2hG75aCMnpjTBy4G1gFbAce\nMcaUi8iXReRm1wJdkEjYGTdz5vhWEjZHjuhArEvq66F/fygqMl13VnpModXoMcasBFa22fbFDvou\n6rsst+zZY9e4UaN3y5EjOn3VJfX1MGiQbxXhUlAZfYhs3WpbnVrpDmPg8GEt3bjk9Gk1epcU2mBs\ncGzZYgcINdt0x6lT9lfTmDG+lYSLZvRu0Yw+z9myBaZMsZe4U9xw+LBtNaN3R329foZdokaf52zZ\nojNuXJM+K1YzenfU1+s5Ci5JD8aGMD244Iz+zBnYvVuN3jWa0bunrg6GDPGtIlySSYjFwpjRVHBG\nX15u38DLLvOtJGw0o3dPXZ1m9C5Ro89jNm2yrRq9W9IZvV50xB1q9G5JJsMo20ABGv3mzXamwuTJ\nvpWEzeHDMHKkPaFHccOpU2r0LkkkNKPPWzZtsidKhTCSnstUVcEFF/hWES7GaEbvGi3d5CnGWKOf\nO9e3kvBRo3dLY6PNONXo3WGN3reKaAjkZXSP/fvh5EmYN8+3kvBRo3dLXZ1t1ejdYWv0mtHnHRs3\n2lYzercYA4cOwYUX+lYSLmr07tGMPk/ZsMG+cTqH3i21tdDQoBm9S9JGr/Po3ZFIaEafl6xfDzNm\n6Gnjrqmqsq0avTs0o3ePZvR5yvr1MH++bxXho0bvHjV69+ismzykqgoqK9Xos0Fl6kKTF13kV0fI\nqNG7R0+YykPWr7etGr171Ojdo0bvHj1hKg9Zt87+d1ajd8/rr9txkKFDfSsJFzV692jpJg9Zu9YO\nxOosBfdUVsLYseH87M1FTp2yrRq9O2xG71tFNATyMjrHGGv0b3iDbyWFQWWllm1cU1NjL3GnV5hy\nR3NzOBdeLwijP3jQLrKlRp8dXn9djd41tbW2NKa/mtwRj6vR5xUvv2zbhQv96igEjNGMPhvU1OgY\niGuam6Ffv6RvGZFQMEY/YABcfrlvJeFTXW3Pih03zreSsKmthWHDfKsIG2v0mtHnDS+/bGfb6Nro\n7jl40Lbjx/vVETqa0btHSzd5RFOTHYh905t8KykM1Oizg2b07tHB2Dxiwwa7drcafXZIG/3FF/vV\nETqa0btHSzd5xPPP2/aqq/zqKBQOHoTiYjj/fN9KwqamRjN618TjBTYYKyJLRGSniOwWkXvbefwe\nEdkmIptF5M8iMiF6qb3juefgkkt0bfRscfCgHYgN5USTXCU9vVJxR0GVbkSkCPgucBMwC7hDRGa1\n6bYBuNIYcxnwKPCNqIX2hmTSGv011/hWUjgcOKD1edc0Nto/zejdUmilmwXAbmPMXmNME/Ar4JbM\nDsaY1caY06m7LwE5Mblu+3Y73U+NPnu89hpMmuRbRdjU1tpWM3q3hJTR9+tGn7HAwYz7FUBnpx59\nHHiyvQdEZCmwFGD06NGUlZV1T2Uv+d3vxgJTKS19ibKyhk771tXVOdcTOo2N9mQpNXq31NTYVjN6\nt9gafeEYfXsnWbf76kXkg8CVwLXtPW6MWQYsA5g+fbpZtGhR91T2kv/+b5gwAe64441d9i0rK8O1\nntA5cMCeGTtxom8lYXPypG3V6N0SUkbfndJNBZBZdR0HVLbtJCI3AJ8HbjbGNEYjr/ckElBWBtdd\n51tJ4fDaa7bVjN4tx4/bduRIvzpCp9CWQFgLTBWRSSLSH7gdWJHZQUTmAT/EmvyR6GX2nA0b7Bfi\nxht9Kykc1OizQ3W1bc87z6+jxa3fAAANYUlEQVSO0CmoM2ONMXHgbmAVsB14xBhTLiJfFpGbU92+\nCQwGfiMiG0VkRQe7yxpPP23bG27wq6OQ2LPHLjOhC5q5RTP67BBS6aY7NXqMMSuBlW22fTHjds7Z\n6R//CHPn6ok72WTXLnvOQlGRbyVhoxl9dii06ZV5x8mT9ozYt7/dt5LCYtcumDrVt4rwOX7cTq3s\n1600TektIc26CdLon3rKDsaq0WePZNKWbtTo3VNdrWWbbFBog7F5x4oV9ougFxrJHhUVdh36adN8\nKwmf6mot27gmmbR/odTogzP65mZ44gl45zv1p2022bHDtmr07jl+XDN61zQ321aNPkd55hl75uCt\nt/pWUliUl9t29my/OgoBLd24Jx63rdboc5Tf/AaGDIHFi30rKSzKy2H0aPunuOXYMTV616QzejX6\nHKSpCZYvh5tvhpIS32oKi/JyzeazQUODnVV2wQW+lYSNlm5ymCefhBMn4AMf8K2ksEgm1eizRVWV\nbfX6Cm5Jl26KinTWTc7x0EP2BCld9iC77N0Lp07BvHm+lYTPoUO2VaN3i5ZucpTDh+Hxx+HDH9bZ\nNtlm/XrbqtG7J230Wrpxi5ZucpQHH7Q/tz72Md9KCo8NG+x1YrV04x7N6LODzrrJQRIJ+MEP7JLE\nM2f6VlN4rF0Lc+bAgAG+lYTPoUP2erw6u8ktWrrJQZYvh/374R/+wbeSwiMeh5degje9ybeSwuDQ\nIRgzRheOc01r6UYHY3MCY+Ab34ApU+y0SiW7bNkC9fVw9dW+lRQGBw7AuJy4InPYaI0+x1i1Ctat\ng89+VrMcHzz3nG01o88Oe/fapaAVt2iNPodIJuELX7DXhb3rLt9qCpM//QkmT7bvgeKWeNyWKNXo\n3RNajT6vJyL+8pfwyit2/nz//r7VFB7Nzfa6vHfe6VtJYXDggJ14MHmybyXho6WbHKG21pZrrrwS\nPvhB32oKkzVr7IlSb3ubbyWFwd69tlWjd09opZu8zejvu8/OQFi+3E43U7LP8uV2TSFdQC47vPqq\nbadM8aujENBZNznAqlXwve/Bpz6lFxfxRSJhjX7xYhg0yLeawmDTJhgxAsaO9a0kfEKr0eed0R88\naEs1s2fDV77iW03h8tRTUFkJH/qQbyWFw4YNdpkJEd9Kwqd1UTM1+qxTW2uvHNXUBI8+CqWlvhUV\nLj/8IYwaZd8PxT3xuD1nYe5c30oKA83oPVFXZ02lvBweeQRmzPCtqHDZscNel/cTn9DZTtli0ya7\nFv0VV/hWUhicPm3bAQO0Rp81jhyB66+3szweflgH/3zzL/8CAwfC3Xf7VlI4PP20bd/6Vr86CoX0\n4nEjRjT5FRIROW/0L7xgp1Bu3gy//S3cdptvRYXNU0/ZstlnPmPX/leyw6pVcPnlujxxtqistJ9v\nLd04pqYG7rkHrrnGri+/Zo1e8Ns3VVXw0Y/astk//7NvNYXDgQPw7LO6llM2ef11uOgi3yqiI+fm\n0VdX24G+b3/bXhZw6VK7aNnQob6VFTZHj8KSJfZ6pU8+qdfkzSbf/75tP/5xvzoKicpKNfrIqa21\nJYHf/AZ+/3tobISbboIHHoD5832rU557zl65q6oKHnsMLrvMt6LCYfdu+M53bMlS1xPKHpWVYXlP\nt4xeRJYA/wkUAT8yxnytzeMDgJ8CVwDVwG3GmH2d7fN4dYyLZSU1TOUUkzEUUVpcz4zRW7nsonWM\nPnOE+fO/BMCnr/tXAPZs3MeZugaSidaR8EHDBp6z70vmTgRg87PbALjs2ll8e3XrvvZs3Mclcyey\ndc0OAOa8eQbz7pgOizqPQ+Zz0/trrw/Q4ePZoPF0E7eOuIvHTjzErSPuor7mNIOGDeSxEw8BtGwD\nG790vPZs3NeyPWGKOT3oDRwteTt7q6dTwhEuK/pPnvpaPxYvbn1tN8beB9gYb12zg9LBZ6f6Z+oa\nWuK7+EY7wDLnzTO8xqevNJ5u5MbY+4gVxVo+i4OGDWyJHUCsKMaq5l8DZ8c78/HSwSXnPGfOm2ew\nZ+M+AL619iEWXHaMRGMJx1d+gVtHnOBMXQOlg0vOes/e8S/X8viXVud1TDNZXHwbyUSy5Xu7uLj1\nc5P+/gFsXbODZCJJrKi1Ap0Z0/T2ttvS+8n0kvT7caaugaSJUZX4Oc/8+Ldc+4bmLn0hH+jS6EWk\nCPgucCNQAawVkRXGmG0Z3T4OnDDGTBGR24GvA50Omx6rHkSM6xnKHqYM+APnF2/lDfObicXCmM7k\nk0RSaEwO4fBhaEgOo9H0J5YcxIEDdj52XWIM9SZOkmLOxIfBsYk0NJdyoGEOtWYEdYznFJNI1ven\ntKmOS2K/5uLk4xRLM6DzWhPJGM1mMDEjJI0drGtKltJsWg0nZoTjx0k9NqjlMdPyeBGxZAmNpj+G\nGEn6YUwpB05MZF/jBKrjM5g9GyQ5kIWD/p3S2IlsvkRvJBJCsxlE0hjONJdy/Dg0G3vq9ZnmUpqS\ngzjTbE+gSfeLmdYzyIqSJS2xTm9vuy29n2ZT1PL+xYxQlCyhyRRzzMwFYpTKMWBYll65W8SYzkeV\nReQq4H5jzOLU/fsAjDFfzeizKtXnRRHpB1QBo00nOx97wSVm5uErEenY2NPZettsqLdkZmBtef83\nb2LHE/uBc7PxdCafqSOdCWf+UoCzf0W0t6+eICKvGGOu7PnzrjSwrlfHLKaWQVQwlD2cx2bOYysx\niZ/Vp7M4dsT7v3kTj3zmybP24TOz721s7XN7H9/uUkwtY3iRiSynRI532jcdW98xzcTHZzdKBnGA\nBdzHHd+6nkfvXZUzcYXexbY7pZuxwMGM+xVA2xVmWvoYY+IiUgOMBI61EbgUWAowatQobvtW5xPi\n0z+9emoqHSK0plRtGDFumC3fAGVlZWc9Nu+O6cy6ZdJZOmJFMQYM7N/SN/3cGX9jC6mlQ0ra3Vc2\nGDG8nvNP/hhIYnMag5BsaYU4MeLEaKKIJvpxmn7UMYCTFEkY84ZdMmJYPaNrftKHPdgPoUDq/Ugg\nxCmikWJOUcJRSjmKSBhT+3pC32PbVwzF1DOKDUF9F7pj9O2trNH2E9idPhhjlgHLACZPmGzSGV46\nc0/X3tK0zZZd1+g/uLSDhdUXcc5zz/nvntEnU7sPhg5uYAZ/6WGNvoQ9G+uor2n9cKcfy6yFts1s\nulujLx1S0vKPO5eyo94wdMgZLq590mGNvg4o5bETD7V85tK0V6NHzv6M5zOjxySZWL/Ka42+lRil\nQ0pa3sd8pjvz6CuA8Rn3xwGVHfVJlW6GAZ3/3lQURVGygzGm0z9s1r8XmAT0BzYBs9v0+STwg9Tt\n24FHutrvtGnTTC6xevVq3xLOAVhnuohje3+5Fltjci++vY2tycH45lpsjQnnsxtKbLss3Rhbc78b\nWIWdXvkTY0y5iHw5dcAVwI+Bn4nIbmwmf3uE/4sURVGUPtCtefTGmJXAyjbbvphxuwF4X7TSFEVR\nlCjI2bVuFEVRlGhQo1cURQkcNXpFUZTAUaNXFEUJHDV6RVGUwFGjVxRFCRw1ekVRlMBRo1cURQkc\nNXpFUZTAUaNXFEUJHDV6RVGUwFGjVxRFCZwuLyXo7MAip4CdXg7ePqNoc0WsHGC6MWZIT5+Ug7GF\n3Itvr2ILORnfXIsthPPZDSK23Vq90hE7TS+v2ekCEVmXS3rAaurlU3MqtpB78e1DbCHH4ptrsYVw\nPruhxFZLN4qiKIGjRq8oihI4Po1+mcdjt0eu6YHeawrptbiiL3pCei2uCOWzm2t6oBeavA3GKoqi\nKNlBSzeKoiiBo0avKIoSOFk3ehF5n4iUi0hSRK5s89h9IrJbRHaKyOIsalqSOuZuEbk3W8dto+En\nInJERLZmbDtPRJ4WkV2pdkQX+9DYtq8hyNimju01vlHENvWcnIuv79imNEQSX4wxWf0DZgLTgTLg\nyozts4BNwABgErAHKMqCnqLUsSYD/VMaZnmIy1uA+cDWjG3fAO5N3b4X+LrGVmObS/GNIra5GN9c\niG2U8c16Rm+M2W6Mae/Mt1uAXxljGo0xrwG7gQVZkLQA2G2M2WuMaQJ+ldKSVYwxfwGOt9l8C/BQ\n6vZDwK1d7ENj2w6BxhZyIL5RxDa1n1yLr/fYQnTxzaUa/VjgYMb9itS2UI/bHcYYYw4BpNrze7kf\nje255HtsfR+7M6KKLehntz16HF8nSyCIyJ+AC9p56PPGmN939LR2tmVj7qev4/YKja078iy2vo/d\nY/IsvnkV265wYvTGmBt68bQKYHzG/XFAZTSKcvK43eGwiFxojDkkIhcCRzS2kZHvsfV97M44J7ag\nvhAh7ca3M3KpdLMCuF1EBojIJGAq8NcsHHctMFVEJolIf+D2lJZcYAVwV+r2XUBHWU939qOxPZt8\njy3kbnyjim16X/rZPZuex9fDKPK7sP8tG4HDwKqMxz6PHeneCdyURU1vB15NHfvz2Y5JSsMvgUNA\ncyo+HwdGAn8GdqXa8zS2Gttcim8Usc3V+PqObZTx1SUQFEVRAieXSjeKoiiKA9ToFUVRAkeNXlEU\nJXDU6BVFUQJHjV5RFCVw1OgVRVECR41eURQlcP4/Q/aGqb3bFI8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a12c6fe80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nx = 100\n",
    "xm = 10\n",
    "w1s = np.array([0.5,1,2,10])\n",
    "nplot = w1s.size\n",
    "iplot = 0\n",
    "for w1 in w1s:\n",
    "    iplot += 1\n",
    "    x  = np.random.uniform(-xm,xm,nx)\n",
    "    plt.subplot(1,nplot,iplot)\n",
    "    py = 1/(1+np.exp(-w1*x))\n",
    "    yp = np.array(np.random.rand(nx) < py)\n",
    "    xp = np.linspace(-xm,xm,100)\n",
    "    pyp = 1/(1+np.exp(-w1*xp))\n",
    "\n",
    "    plt.scatter(x,yp,c=yp,edgecolors='none',marker='+')\n",
    "    plt.plot(xp,pyp,'b-')\n",
    "    plt.axis([-xm,xm,-0.1,1.1])\n",
    "    plt.grid()\n",
    "    if (iplot > 1):\n",
    "        plt.yticks([])\n",
    "    plt.xticks([-10,0,10])\n",
    "    plt.title('w1={0:.1f}'.format(w1))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that $f(w_1x)$ represents the probability that $y=1$.  The function $f(w_1x) > 0.5$ for $x>0$ meaning the samples are more likely to be $y=1$.  Similarly, for $x<0$, the samples are more likely to $y=0$.  The scaling $w_1$ determines how fast that transition is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting the Logistic Model\n",
    "\n",
    "In a logistic model, we must fit the weights $w_0,w_1,\\ldots,w_k$ in the model \n",
    "$$\n",
    "    P(y=1|x) = f(z),  \\quad z = w_0 + w_1x_1 + \\cdots + w_kx_k,  \n",
    "$$\n",
    "The `sklearn` module has very good routines for fitting this model.  First we get the training data and samples.  We will standardize the `X` values to make more sense of the slope values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xnames = ['thick','size_unif','shape_unif','marg','cell_size','bare',\n",
    "         'chrom','normal','mit']\n",
    "X = np.array(df[xnames])\n",
    "Xs = preprocessing.scale(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create a logistic regression object.  The parameter `C` states the level of regularization and will be discussed in class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "logreg = linear_model.LogisticRegression(C=1e5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=100000.0, class_weight=None, dual=False,\n",
       "          fit_intercept=True, intercept_scaling=1, max_iter=100,\n",
       "          multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.fit(Xs, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can next plot the accuracy on the training data.  We see we get an accuracy better than simple linear classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training data = 0.969253\n"
     ]
    }
   ],
   "source": [
    "yhat = logreg.predict(Xs)\n",
    "acc = np.mean(yhat == y)\n",
    "print(\"Accuracy on training data = %f\" % acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also useful to print the weights for each feature.  We can use the `pandas` package to make a table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>slope</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>thick</td>\n",
       "      <td>1.508834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>size_unif</td>\n",
       "      <td>-0.015979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>shape_unif</td>\n",
       "      <td>0.957072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>marg</td>\n",
       "      <td>0.947234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>cell_size</td>\n",
       "      <td>0.214964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bare</td>\n",
       "      <td>1.395001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>chrom</td>\n",
       "      <td>1.095654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>normal</td>\n",
       "      <td>0.650696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>mit</td>\n",
       "      <td>0.925912</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      feature     slope\n",
       "0       thick  1.508834\n",
       "1   size_unif -0.015979\n",
       "2  shape_unif  0.957072\n",
       "3        marg  0.947234\n",
       "4   cell_size  0.214964\n",
       "5        bare  1.395001\n",
       "6       chrom  1.095654\n",
       "7      normal  0.650696\n",
       "8         mit  0.925912"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W=logreg.coef_\n",
    "data = {'feature': xnames, 'slope': np.squeeze(W)}\n",
    "dfslope = pd.DataFrame(data=data)\n",
    "dfslope"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross-Validation\n",
    "\n",
    "The above code measured the accuracy on the training data.  However, we should measure the accuracy on test data.  We perform 4-fold cross-validation and print average precision, recall, f1-score and accuracy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision = 0.9610, SE=0.0118\n",
      "Recall =    0.9615, SE=0.0144\n",
      "f1 =        0.9608, SE=0.0112\n",
      "Accuracy =  0.9679, SE=0.0110\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "nfold = 10\n",
    "kf = KFold(n_splits=nfold)\n",
    "prec = []\n",
    "rec = []\n",
    "f1 = []\n",
    "acc = []\n",
    "for train, test in kf.split(Xs):            \n",
    "    # Get training and test data\n",
    "    Xtr = Xs[train,:]\n",
    "    ytr = y[train]\n",
    "    Xts = Xs[test,:]\n",
    "    yts = y[test]\n",
    "    \n",
    "    # Fit a model\n",
    "    logreg.fit(Xtr, ytr)\n",
    "    yhat = logreg.predict(Xts)\n",
    "    \n",
    "    # Measure performance\n",
    "    preci,reci,f1i,_= precision_recall_fscore_support(yts,yhat,average='binary') \n",
    "    prec.append(preci)\n",
    "    rec.append(reci)\n",
    "    f1.append(f1i)\n",
    "    acci = np.mean(yhat == yts)\n",
    "    acc.append(acci)\n",
    "\n",
    "# Take average values of the metrics\n",
    "precm = np.mean(prec)\n",
    "recm = np.mean(rec)\n",
    "f1m = np.mean(f1)\n",
    "accm= np.mean(acc)\n",
    "\n",
    "# Compute the standard errors\n",
    "prec_se = np.std(prec)/np.sqrt(nfold-1)\n",
    "rec_se = np.std(rec)/np.sqrt(nfold-1)\n",
    "f1_se = np.std(f1)/np.sqrt(nfold-1)\n",
    "acc_se = np.std(acc)/np.sqrt(nfold-1)\n",
    "\n",
    "print('Precision = {0:.4f}, SE={1:.4f}'.format(precm,prec_se))\n",
    "print('Recall =    {0:.4f}, SE={1:.4f}'.format(recm, rec_se))\n",
    "print('f1 =        {0:.4f}, SE={1:.4f}'.format(f1m, f1_se))\n",
    "print('Accuracy =  {0:.4f}, SE={1:.4f}'.format(accm, acc_se))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROC curve\n",
    "\n",
    "As discussed above, the logistic classifier outputs a *soft* classification $P(y=1|x)$.  One simple idea is to select the class label $\\hat{y}=1$ whenever $P(y=1|x) > 0.5$.  However, one can also set $\\hat{y}=1$ whenever $P(y=1|x) > t$ for some threshold $t$.  Using higher thresholds values selects $\\hat{y}=0$ more often which results in fewer *false alarms*, but also many *missed detections* (reduces sensitivity).  Similarly, lower thresholds values selects $\\hat{y}=1$ more often which results in fewer *missed detections* (increases *sensitivity*) but also more *false alarms*.  The ROC curve displays this  tradeoff by graphing the *true positive rate* vs. *false positive rate* obtained with different $t$.  \n",
    "\n",
    "The ROC curve can be plotted in the `sklearn` package with the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 0.1)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEKCAYAAADuEgmxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAExFJREFUeJzt3X2wXHV9x/H3l8QYikZQYicmRGIb\n7/RqEZ9IGVvnthYN1CFtcSzYTsVWg1OxU61WqA5iZjq2jg/TjhRNW+rDDARKx5J2MlKlbqVOilEJ\n0YReeg2W3CQO8hBikCGEfPvHbtxl2XvzS3bPPuS+XzN32HPOb3e/+2VvPvd3ztmzkZlIklTipEEX\nIEkaHYaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpWGWhERHXRcT9EfG9GbZHRPxNRExFxLaIeEVVtUiS\neqPKmcbngNWzbD8fWNn4WQtcW2EtkqQeqCw0MvPrwEOzDFkDfCHr/hs4NSKWVFWPJKl78wf43EuB\nXS3L0411e9sHRsRa6rMRFi5c+Mrly5f3pcBhd/jwYU46ycNSYC9aDboXP3z0MAefhAXzBlbCTyUQ\ngy5iSOzfPfVAZi7u9nEGGRqd/l92vKZJZq4H1gOMjY3l5ORklXWNjFqtxsTExKDL6Knr77iPW7bu\nPub77du3j1NPPbWCikbPoHuxY+9+xpcs4sbLzh1YDUeciL8jxysi/q8XjzPIP82mgTNalpcBewZU\ni4bELVt3s2Pv/kGXoS6ML1nEmrOXDroMVWSQM42NwOURsQFYBTySmU/bNaW553j+Sq3/RTn4v2yH\ngb1QlSoLjYi4AZgATo+IaeDDwDMAMvMzwCbgAmAK+AnwtqpqkST1RmWhkZmXHGV7Au+q6vlVneM9\n7lDiyP5wScPJ0010zKo87uD+cGm4DfKYhkbYsJwdI6m/DA11NNsuKHchSXOXu6fU0Wy7oNyFJM1d\nzjTmqKMdzB6mD2hJGh7ONOaoox3MdjYhqRNnGicoZxKSquBM4wTlTEJSFZxpnMCcSUjqNWcakqRi\nhoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKK+eG+IXa0S4Hs2/cY105u7rjNy5dLqoIz\njSHWzTfkeZkQSVVwpjHkZrsUSK1WY2LCy4RI6h9nGpKkYoaGJKmYoSFJKmZoSJKKeSB8wGY7rdbT\nZiUNG2caAzbbabWeNitp2DjTqJjf1S3pROJMo2J+V7ekE4kzjT5wJiHpROFMQ5JUzJlGgaMdl5iN\nZ0BJOpE40yjghQMlqc6ZRiGPS0iSMw1J0jEwNCRJxSoNjYhYHRGTETEVEVd02P7CiLgtIrZFRC0i\nllVZjySpO5WFRkTMA64BzgfGgUsiYrxt2MeBL2TmWcA64KNV1SNJ6l6VM41zgKnM3JmZB4ENwJq2\nMePAbY3bX+uwXZI0RKo8e2opsKtleRpY1TbmLuAi4K+B3wKeHRHPy8wHWwdFxFpgLcDixYup1WpV\n1dzRvn2PAfT9eY/mwIEDQ1fToNiLJnvRZC96r8rQiA7rsm35fcCnI+JS4OvAbuDQ0+6UuR5YDzA2\nNpYTExM9LfRorp3cDDB038dd/47wiUGXMRTsRZO9aLIXvVdlaEwDZ7QsLwP2tA7IzD3AbwNExLOA\nizLzkQprkiR1ocpjGluAlRGxIiIWABcDG1sHRMTpEXGkhiuB6yqsR5LUpcpCIzMPAZcDtwJ3Azdl\n5vaIWBcRFzaGTQCTEXEP8LPAX1RVjySpe5VeRiQzNwGb2tZd1XL7ZuDmKmuQJPWOnwiXJBUzNCRJ\nxQwNSVIxQ0OSVMzQkCQVMzQkScUMDUlSMUNDklTM0JAkFav0E+Gj5Po77uOWrbs7btuxdz/jSxb1\nuSJJGj7ONBpu2bqbHXv3d9w2vmQRa85e2ueKJGn4ONNoMb5kETdeNlzfmSFJw8SZhiSpmKEhSSpm\naEiSihkakqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpmKEhSSpm\naEiSihkakqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKlZpaETE6oiYjIipiLiiw/blEfG1\niLgzIrZFxAVV1iNJ6k5loRER84BrgPOBceCSiBhvG/Yh4KbMfDlwMfC3VdUjSepelTONc4CpzNyZ\nmQeBDcCatjEJLGrcfg6wp8J6JEldml/hYy8FdrUsTwOr2sZcDfx7RLwbOAX49U4PFBFrgbUAixcv\nplarHXMxtV1PsHnPoRm33/fjwyx/9knH9diDcuDAgZGqt0r2osleNNmL3qsyNKLDumxbvgT4XGZ+\nIiLOBb4YES/NzMNPuVPmemA9wNjYWE5MTBxzMdd+djN7HtvP+JJFHbefeiqsOXspE6uWH/NjD0qt\nVuN4enEishdN9qLJXvRelaExDZzRsryMp+9++kNgNUBmbo6IhcDpwP1VFDS+ZBE3XnZuFQ8tSXNC\nlcc0tgArI2JFRCygfqB7Y9uY+4DXAUTELwALgR9VWJMkqQuVhUZmHgIuB24F7qZ+ltT2iFgXERc2\nhv0p8I6IuAu4Abg0M9t3YUmShkSVu6fIzE3AprZ1V7Xc3gG8psoaJEm94yfCJUnFDA1JUjFDQ5JU\nzNCQJBUzNCRJxSo9e6rfrr/jPm7Zurvjth17Z/40uCSpzAk107hl62527N3fcdv4kkWsOXtpnyuS\npBPLCTXTAC8VIklVOuaZRkTMi4jfraIYSdJwmzE0ImJRRFwZEZ+OiNdH3buBncCb+1eiJGlYzLZ7\n6ovAw8Bm4O3A+4EFwJrM3NqH2iRJQ2a20HhRZv4iQET8PfAAsDwzf9yXyiRJQ2e2YxpPHLmRmU8C\n9xoYkjS3zTbTeFlE7Kf5DXwntyxnZvqhB0maY2YMjcyc189CJEnDb8bQaHz16juBnwe2Adc1vlhJ\nkjRHzXZM4/PAq4DvAhcAn+hLRZKkoTXbMY3xlrOn/gH4Zn9KkiQNq9Kzp9wtJUmadaZxduNsKaif\nMeXZU5I0x80WGndl5sv7VokkaejNtnsq+1aFJGkkzDbTeH5EvHemjZn5yQrqkSQNsdlCYx7wLJqf\nCJckzXGzhcbezFzXt0okSUNvtmMazjAkSU8xW2i8rm9VSJJGwoyhkZkP9bMQSdLwO+bvCJckzV2G\nhiSpmKEhSSpmaEiSihkakqRihoYkqZihIUkqVmloRMTqiJiMiKmIuKLD9k9FxNbGzz0Rsa/KeiRJ\n3Znt2lNdiYh5wDXAecA0sCUiNmbmjiNjMvM9LePfDfj9HZI0xKqcaZwDTGXmzsw8CGwA1swy/hLg\nhgrrkSR1qbKZBrAU2NWyPA2s6jQwIl4IrAD+Y4bta4G1AIsXL6ZWq3V8wn37HgOYcfuJ5sCBA3Pm\ntR6NvWiyF032oveqDI1OV8md6dsALwZuzswnO23MzPXAeoCxsbGcmJjo+CDXTm4GYGLi3GMsdTTV\najVm6sVcYy+a7EWTvei9KndPTQNntCwvA/bMMPZi3DUlSUOvytDYAqyMiBURsYB6MGxsHxQRY8Bp\nwOYKa5Ek9UBloZGZh4DLgVuBu4GbMnN7RKyLiAtbhl4CbMjMmXZdSZKGRJXHNMjMTcCmtnVXtS1f\nXWUNkqTe8RPhkqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpmKEh\nSSpmaEiSihkakqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpmKEh\nSSpmaEiSihkakqRihoYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJKmZoSJKKGRqSpGKGhiSpWKWh\nERGrI2IyIqYi4ooZxrw5InZExPaIuL7KeiRJ3Zlf1QNHxDzgGuA8YBrYEhEbM3NHy5iVwJXAazLz\n4Yh4flX1SJK6V+VM4xxgKjN3ZuZBYAOwpm3MO4BrMvNhgMy8v8J6JEldqmymASwFdrUsTwOr2sa8\nGCAivgHMA67OzC+3P1BErAXWAixevJhardbxCfftewxgxu0nmgMHDsyZ13o09qLJXjTZi96rMjSi\nw7rs8PwrgQlgGXB7RLw0M/c95U6Z64H1AGNjYzkxMdHxCa+d3AzAxMS5XZQ9Omq1GjP1Yq6xF032\nosle9F6Vu6emgTNalpcBezqMuSUzn8jMe4FJ6iEiSRpCVYbGFmBlRKyIiAXAxcDGtjH/AvwqQESc\nTn131c4Ka5IkdaGy0MjMQ8DlwK3A3cBNmbk9ItZFxIWNYbcCD0bEDuBrwPsz88GqapIkdafKYxpk\n5iZgU9u6q1puJ/Dexo8kacj5iXBJUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUM\nDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVIxQ0OSVMzQkCQVMzQkScUM\nDUlSMUNDklTM0JAkFTM0JEnFDA1JUjFDQ5JUzNCQJBUzNCRJxQwNSVKx+YMu4Fj98NHD/M5nN3fc\ntmPvfsaXLOpzRZI0d4zcTOPgkzNvG1+yiDVnL+1fMZI0x4zcTGPBPLjxsnMHXYYkzUkjN9OQJA2O\noSFJKmZoSJKKVRoaEbE6IiYjYioiruiw/dKI+FFEbG38vL3KeiRJ3ansQHhEzAOuAc4DpoEtEbEx\nM3e0Db0xMy+vqg5JUu9UOdM4B5jKzJ2ZeRDYAKyp8PkkSRWr8pTbpcCuluVpYFWHcRdFxGuBe4D3\nZOau9gERsRZY21h8PCK+1+tiR9TpwAODLmJI2Isme9FkL5rGevEgVYZGdFiXbcv/CtyQmY9HxDuB\nzwO/9rQ7Za4H1gNExLcy81W9LnYU2Ysme9FkL5rsRVNEfKsXj1Pl7qlp4IyW5WXAntYBmflgZj7e\nWPw74JUV1iNJ6lKVobEFWBkRKyJiAXAxsLF1QEQsaVm8ELi7wnokSV2qbPdUZh6KiMuBW4F5wHWZ\nuT0i1gHfysyNwB9HxIXAIeAh4NKCh15fVc0jyF402Ysme9FkL5p60ovIbD/MIElSZ34iXJJUzNCQ\nJBUbqtAouOzIMyPixsb2OyLizJZtVzbWT0bEG/pZdxWOtxcRcV5EfDsivtv479NOYR413bwvGtuX\nR8SBiHhfv2quSpe/I2dFxOaI2N54fyzsZ+291sXvyDMi4vONHtwdEVf2u/ZeKujDayPiOxFxKCLe\n1LbtrRHxv42ftxY9YWYOxQ/1g+XfB14ELADuAsbbxvwR8JnG7YupX4IEYLwx/pnAisbjzBv0axpQ\nL14OvKBx+6XA7kG/nkH1omX7PwP/BLxv0K9ngO+L+cA24GWN5efN4d+RtwAbGrd/BvgBcOagX1OF\nfTgTOAv4AvCmlvXPBXY2/nta4/ZpR3vOYZpplFx2ZA31DwAC3Ay8LiKisX5DZj6emfcCU43HG1XH\n3YvMvDMzj3weZjuwMCKe2Zeqq9HN+4KI+E3qvwzb+1RvlbrpxeuBbZl5F/z0M1KzfA/m0OumFwmc\nEhHzgZOBg8D+/pTdc0ftQ2b+IDO3AYfb7vsG4CuZ+VBmPgx8BVh9tCccptDodNmR9u9u/emYzDwE\nPEL9L6aS+46SbnrR6iLgzmx+gHIUHXcvIuIU4APAR/pQZz908754MZARcWtjV8Wf9aHeKnXTi5uB\nR4G9wH3AxzPzoaoLrkg3//Yd132H6eteSy47MtOYkvuOkm56Ud8Y8RLgr6j/hTnKuunFR4BPZeaB\nxsRj1HXTi/nALwOvBn4C3BYR387M23pbYt9004tzgCeBF1DfLXN7RHw1M3f2tsS+6ObfvuO67zDN\nNI562ZHWMY2p5XOofyiw5L6jpJteEBHLgC8Bv5+Z36+82mp104tVwMci4gfAnwB/3vjA6ajq9nfk\nPzPzgcz8CbAJeEXlFVenm168BfhyZj6RmfcD3wBG9fpU3fzbd3z3HfSBnJaDMvOp73teQfOAzkva\nxryLpx7Yuqlx+yU89UD4Tkb7IF83vTi1Mf6iQb+OQfeibczVjP6B8G7eF6cB36F+4Hc+8FXgNwb9\nmgbUiw8A/0j9L+1TgB3AWYN+TVX1oWXs53j6gfB7G++N0xq3n3vU5xz0i257URdQv0T694EPNtat\nAy5s3F5I/SyYKeCbwIta7vvBxv0mgfMH/VoG1QvgQ9T3125t+Xn+oF/PoN4XLY8x8qHRbS+A36N+\nQsD3gI8N+rUMqhfAsxrrtzcC4/2Dfi0V9+HV1GcVjwIPAttb7vsHjf5MAW8reT4vIyJJKjZMxzQk\nSUPO0JAkFTM0JEnFDA1JUjFDQ5JUzNCQCkXEkxGxteXnzIiYiIhHIuLOxhVTP9wY27r+fyLi44Ou\nX+qFYbqMiDTsHsvMs1tXNC63fXtmvrFxrautEfFvjc1H1p8M3BkRX8rMb/S3ZKm3nGlIPZKZjwLf\nBn6ubf1j1D9kOcoX0ZQAQ0M6Fie37Jr6UvvGiHge8Eu0XYY9Ik4DVgJf70+ZUnXcPSWVe9ruqYZf\niYg7qX9fwV9m5vaImGis3waMNdb/sI+1SpUwNKTu3Z6Zb5xpfUS8GPivxjGNrf0uTuold09JFcvM\ne4CPUr+6qjTSDA2pPz4DvDYiVgy6EKkbXuVWklTMmYYkqZihIUkqZmhIkooZGpKkYoaGJKmYoSFJ\nKmZoSJKK/T8ptdZv0r9tSQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b8b58d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "yprob = logreg.predict_proba(Xs)\n",
    "fpr, tpr, thresholds = metrics.roc_curve(y,yprob[:,1])\n",
    "\n",
    "plt.plot(fpr,tpr)\n",
    "plt.grid()\n",
    "plt.xlabel('FPR')\n",
    "plt.ylabel('TPR')\n",
    "plt.ylim([0.5,1])\n",
    "plt.xlim([0,0.1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us see what are the relation between the threshold and the TPR and FPR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5,0,'Threshold')"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xt8XWWd7/HPb9+S7CTNtS3QJE2A\ntrRc2kqEMghWC4oMFzmKwhxmzvhCEY8Mjpc5B2cUFXWOtxmVmR4RHV4weEZAHLVqEYciFpFCixQo\nLW0DlDZAb2matrnv5Dl/rJ1L06TZSfbea6+d7/v12q+999ore/1Wk/7Ws571rN9jzjlERCS/hPwO\nQERE0k/JXUQkDym5i4jkISV3EZE8pOQuIpKHlNxFRPKQkruISB5SchcRyUNK7iIieSji14arq6td\nfX29X5sXEQmkZ555Zr9zbuZ46/mW3Ovr69mwYYNfmxcRCSQzey2V9dQtIyKSh5TcRUTykJK7iEge\nUnIXEclDSu4iInlo3ORuZneZ2V4z2zTG52Zmt5tZk5k9b2ZvSX+YIiIyEam03O8GLjnO5+8B5iUf\nNwDfm3pYIiIyFeOOc3fOrTWz+uOsciXw786br2+dmZWb2YnOuTfTFONRtjz1MG0v/IZ1NR+mPzTx\nYfonlhdx7Tl1GYhMRCR3pOMmpjnArmHvm5PLjknuZnYDXuueurrJJdi2bU+wrPkurn/5bXRQOKGf\nHZgu9sFnmqkuiU1q+5NRHIswb3YpC04oYf7sUuaUF2FmWdu+iEw/6Ujuo2WpUWfdds7dCdwJ0NjY\nOKmZuZedegK8DC/e+k4oqpjQz+7Y384n7t9Ie3eC9u7EZDY/KW2dvfzns68Pvi8piDBvdgkLZpcy\nf3YpC04oZd7sEsqLMnfAiYSMUEgHFJHpIh3JvRmoHfa+BngjDd87ulDYe+7vn/CP1lcX84uPn5/m\ngFLT1tlL097DbN19hG17DrN192F+u3kP963fNf4Pp0F5PMrFC2dz6Zkn8menVlEQCWdluyLij3Qk\n91XATWZ2H3Au0Jap/nbPQOtzUg1/35QVRTl7biVnz608avn+I91s232YbXsO097Tl7Htb99zmN9s\n2s1PnmmmtCDCioWzWHZyFZHw1EbDRsPGjMIoM4oilBVFk6+jFEZ18BDx07jJ3cx+DCwHqs2sGfgC\nEAVwzt0BrAYuBZqADuBDmQo2GZD37IKV3MdSXVJA9akF/Nmp1RnfVneijz82tbD6hTf5ry17+PnG\nzJ1gxSKhZLL3kn5ZkZf0y4qiFERCRMMhyuNRKuIxKotjVBTHvNfxGKWFEXUhiUxRKqNlrh3ncwd8\nPG0RpSw/kns2FUTCvOO0WbzjtFn09vWzu61ryt/Z09fPoc5eDnUlaOvspa2z13vf2cuhroH3CfYf\n6eGV/e20dfbS3dtPb18/if7Rf4fhkFERj1KeTPaFsfBRF3YiIfM+K45SUeytU1HsHSQqk+/LiqI6\nQMi05lvJ30nLs5a7X6LhELWVcd+275zjSHeC1vZeWjt6ONDRQ2t7DwfaezjY0XvU+7bO3qN+NtHX\nz0u7D3OgvYfO3tG7skIG5fEYFfEoVcUFVBRHvTOEgTOFeIzKEu9AMHDmUBwLaxST5I3gJfdRB+dI\n0JgZpYVRSguj1FVN/iDT2dN31IGgtSP53N5Dy7D3O/Z38KedB2lt7xnzjCEWDiUPAgXeWcHwA8Gw\nM4OhrqSoLkxLzgpgch+glrtAUSzMnFgRc8qLUlrfOcehroR3MBhxUGhpH3jvnU1sfuMQBzq8M4mx\nlBREvAPCQNfQiC6ioQODd7Aoj8cIq7tIsiB4yV3dMjIFZjZ4gbee4pR+JtHXz8HO3hFnB70caO8e\nPBAcSH62fc8RWjt66Bhj5JMZlBeNuFYwrIuoYtiBYOAAUVIQUXeRTFjwkntAh0JKcEXCIW9UU0lB\nyj/T1ds3rIuol5b27uTZQu9RZw27DnTw3K6DtHb00Ns3+t90NGyDyb60cGqJvrQgwnmnVPH2+TM5\ndVaJDhp5LHjJXS13CYDCaJgTy4o4sSz17qKBC8wHOnqGzgpGdB8d6hq7iygVr7a0s+alvXzl11s4\nsayQC+fN5ML5M3nbqdWUxaNT+m7JLcFL7mq5Sx5K1wXmVLx+sJO12/axdts+Vm96k/s37CJksLi2\nfDDZL6kt17WBgAtectdppMiUzElWRr32nDoSff1s3HWQtdv28fvt+7n90e18d812yoqivO3Uas6q\nKTtukj/thBmcd0qVDgQ5KHjJPZQ8dezr8TcOkTwQCYdorK+ksb6ST71rAa3tPfyhab/Xst++j1+/\nMH4lkZmlBSyfP5P66mJqK+PUVhRRVxmnsjimPn0fBS+5R5N9mL2d/sYhkocqimNcvvgkLl980uB1\ngLH09Tv++HILP3/2dR59aS8t7Uc3uIpjYS/ZV8aprYhTV1lEXZX3uqYiTlFM9whkUvCSuy6oimTF\nwHWA47n0zBO59MwTAWjvTtDc2snOAx3sOtDBzgMdNLd28FpLO3/Yvv+Yu4kLIuMXrQuZ8ednnchn\n33MaVRMYrSSBTO7JPwg38ZK/IpI5xQURFpzgzU8wknOO/Ud62NXqJf5dBzo4nMKcCgfbe/npn5r5\nr817+N+XnMY1b61VzaAUBS+5h5MTWqjPXSQwzIyZpQXMLC3gLXUTm2Tnwxc08Lmfb+Lvf/YCP3lm\nF1957xmcflJZhiLNH1Mr5u2Hwck6Mlf7XERyx7zZpdx3wzL+6erF7Gzp4PJ/+QO3/XJzVmdTC6Lg\nJfeBbhmNcxeZNsyM951dw6OfXs4159Rx1xOv8u7vrGXttn1+h5azgpfcB25iUp+7yLRTFo/yj1ed\nyQMfPY9YOMRf3fU0l//LH/juI9vZ9HobTgMtBgWvz33wgqp+iSLT1TkNlaz+xAXc++RrPLTpTb6z\nZhvffmQbJ5YVsmLhLFYsnM15J1dN6+keA5jc1XIXEa9+z0cuPJmPXHgy+4908+hLe1mzZQ8/feZ1\nfrRuJ/FYmAvmVbNi4WzeedqsCRV+ywcBTO7qcxeRo1WXFPCBxlo+0FhLV28fT77Swpote1izZS8P\nv7gHM1haW86KhbO5eNFs5k2DipjBS+7qcxeR4yiMhnnHglm8Y8Esvnyl48U3DrFmy17WvLSHbz68\nlW8+vJXayiJWnOYl+rfWVxJL4YaqoAleclefu4ikyMw4Y04ZZ8wp4xMXzWPPoS4v0W/Zw4+f3snd\nf9xBaUGECxfM5KKFs1hSW0FDdWqTuOS6ACZ3tdxFZHJmzyjkL86t4y/OraOzp48/NO1nzZY9PLJl\nL79+3iuS9rk/X8iHLzjZ50inLoDJXS13EZm6oliYixd5XTP9/Y4XXm/j9jXb+cqvt9DX7/jo20/x\nO8QpCWBHU35fBBGR7AuFjMW15Xz/L8/m8sUn8X8eeomVv2vyO6wpCV7LXUQkQyLhEN/+wGIiIeOb\nD28l0ef4xEXz/A5rUpTcRUSGiYRDfOvqxYRDxrcf2UZffz+fvHh+4IZOKrmLiIwQDhnfeN9ZRELG\n7Y82keh3/N27FwQqwQcvuYeTIavkr4hkUChk/ONVZxIKGf/3sZfp63fc8p7TApPgg5fco8kxqL0d\n/sYhInkvFDK++t4ziISM7699hUS/43N/vjAQCT54yT08MEF2r79xiMi0YGZ86YrTCYeMf/vDq/T1\nO75w+aKcT/ApDYU0s0vMbKuZNZnZLaN8XmdmvzOzZ83seTO7NP2hDm7Me9ZNTCKSJWbGrZct4iMX\nNHD3H3fw+V9sor8/t++1GbflbmZhYCVwMdAMrDezVc65zcNW+xzwgHPue2a2CFgN1GcgXjTOXUT8\nYGb8/aULiYRDfC/ZB//V956Zs3O6ptItcw7Q5Jx7BcDM7gOuBIYndwfMSL4uA95IZ5Cjy+2jpojk\nHzPjf717AZGQ8S+PNpHoc3ztfWcRzsEEn0pynwPsGva+GTh3xDpfBH5rZn8DFAMXpSW60QyWH1C3\njIhkn5nx6XctIBwyvvPIdnYe6OCO686mojjmd2hHSaXPfbRD0shm87XA3c65GuBS4F4zO+a7zewG\nM9tgZhv27Zvk3IcDF1T7NTmuiPjnby+azxcuX8Qzr7Vy9fef5I2DnX6HdJRUknszUDvsfQ3Hdrtc\nDzwA4Jx7EigEqkd+kXPuTudco3OucebMmZOLOJw8Omq0jIj47EPnN/CjD5/LnrYu3v+9P9K094jf\nIQ1KJbmvB+aZWYOZxYBrgFUj1tkJrAAws4V4yT0z05IPjJbp78vI14uITMSyk6u476PL6OlzXH3H\nH9m466DfIQEpJHfnXAK4CXgY2II3KuZFM7vNzK5IrvZp4CNm9hzwY+CvXaamIbfkhLdOyV1EcsPp\nJ5Xx04+dR0lhhL/4wTrWbstM23YiUhrn7pxb7Zyb75w7xTn31eSyW51zq5KvNzvnznfOLXbOLXHO\n/TZjEef4jQMiMj3NrSrmpzf+GXWVcf7qrqe5c+3LvsYTwHruIiK5adaMQu7/6HkA/Co5s5NflNxF\nRNKorCjKdcvqeK3F3/pXSu4iImk2t7KYts5e2jr8G9Wn5C4ikmZ1VXEAXjvQ7lsMSu4iImlWV+kl\n950H/OuaUXIXEUmzgeTuZ7+7kruISJoVF0SoLilgp5K7iEh+mVsVV5/7pKgqpIjksLmVcbXcJyRc\n4D0nNEG2iOSu2so4bx7qojvhT6mUACb3qFfTPZFb5TVFRIabWxXHOWhu9SdXBS+5m0EoqqqQIpLT\n5ibHuvvVNRO85C4iEgB1lcUAvNbiz0VVJXcRkQyoLokRj4V5zacbmYKZ3M00WkZEcpqZUefjiJlg\nJvdwAfRptIyI5La6yrha7hMSiUGi2+8oRESOa25VnF0HOujvz8zEdMcTzOSulruIBEBdVTHdiX72\nHs5+YzSYyd1CkKEpWkVE0mXuYAGx7I+YCWZyFxEJgMHqkD70uwc4uavlLiK5bU5FEeGQ+TJiJpjJ\n3fwOQERkfNFwiJPKC9VynxD1uYtIAMytLPZlRqaAJnfdxCQiwVBXFWenLqimKFYMvf7VSRYRSdXc\nyjitHb0c6urN6naDmdyjRUruIhIIg5NlZ/miajCTe2EZdLX5HYWIyLjqqvyZLDugyb0cOg/6HYWI\nyLjmVnmlf7N9UTWYyb2gBHr8m3hWRCRVJQURqopj7MzyZNnBTO4YuolJRIKiriqubpmUWEhDIUUk\nMOoqczS5m9klZrbVzJrM7JYx1vmAmW02sxfN7D/SG+bIjalwmIgEx9zKOG+2ddKTyF6jNDLeCmYW\nBlYCFwPNwHozW+Wc2zxsnXnAZ4HznXOtZjYrUwF7G1TLXUSCo6YiTr+DPYe6qE0Ojcy0VFru5wBN\nzrlXnHM9wH3AlSPW+Qiw0jnXCuCc25veMEdQcheRAJlZWgDAviPZq+ueSnKfA+wa9r45uWy4+cB8\nM3vCzNaZ2SWjfZGZ3WBmG8xsw759+yYXMSi5i0igVBbHAGhtz94kQ6kk99FqMI7s8I4A84DlwLXA\nD82s/Jgfcu5O51yjc65x5syZE411SDiimZhEJDAq4snk3pG9EgSpJPdmoHbY+xrgjVHW+YVzrtc5\n9yqwFS/ZZ0ZhOSS6oLcrY5sQEUmX8uIokHst9/XAPDNrMLMYcA2wasQ6PwfeAWBm1XjdNK+kM9Cj\nhL2jIP2JjG1CRCRdSgsiREJGa0f2kvu4o2Wccwkzuwl4GAgDdznnXjSz24ANzrlVyc/eZWabgT7g\n75xzLZkMPBld5jchOaG3t5fm5ma6uvLrbK2wsJCamhqi0ajfoUgGmRnl8WhWu2XGTe4AzrnVwOoR\ny24d9toBn0o+Ms80FdN009zcTGlpKfX19Vie/P6dc7S0tNDc3ExDQ4Pf4UiGlcdjHMxiyz2Yd6jK\ntNPV1UVVVVXeJHbwWnNVVVV5dzYio6uMx7LaLRPs5K67VKeVfErsA/Jxn2R05fEoB3OtWyb36D+E\nZFdLSwsrVqwAYPfu3YTDYQaG8z733HMsXryYRCLBwoULueeee4jH44TDYc4880wSiQQNDQ3ce++9\nlJcfM0JYpomKeIznmrNXqjzYLXddUJUsqaqqYuPGjWzcuJEbb7yRT37yk4Pvi4uL2bhxI5s2bSIW\ni3HHHXcAUFRUNLi8srKSlStX+rwX4qfy4iit7b24LPU4BDO561RWctQFF1xAU1PTMcvPO+88Xn/9\ndR8iklxREY/R09dPR09fVrYX7G4ZlSCYlr70yxfZ/MahtH7nopNm8IXLT5/SdyQSCR566CEuueTo\n6ht9fX2sWbOG66+/fkrfL8FWEU/eyNTRQ3FB5lNvMFvuMW/aKrqP+BuHCNDZ2cmSJUtobGykrq5u\nMIkPLK+qquLAgQNcfPHFPkcqfipPliDI1kXVYLbco8mSmb2d/sYhvphqCzvdBvrWx1re1tbGZZdd\nxsqVK7n55pt9iFBywVB9mewMhwxoy30guWd3ZhORySgrK+P222/nW9/6Fr292RsKJ7llqFsmO38D\nwUzuoeQJh8vOhQmRqVq6dCmLFy/mvvvu8zsU8UlFsuzvc7uyMxwymN0yAzQSUnzwxS9+8aj3R46M\nfu1n5PJf/vKXmQpJAqC8yGu5p3swwFiC2XLXTUwiEjCRcIgz5szI2kjugCZ3EZHgKSmIkOjTTUxj\nCyd7kzQbk4gESDQcorc/O/fnBDO5F1V4z50H/I1DRGQCIiFTy/24NM5dRAIoEg7R26eW+9ji1d5z\n+35/4xARmYBo2Ej0q+U+tqIKsDC07/U7EplGwuEwS5YsGXzs2LGDxx57jLKyMpYuXcrChQv50pe+\nBHDU8tNOO43PfOYzPkcvuSCaxZZ7MMe5h0JQVA6d2auNLDJamYEdO3ZwwQUX8Ktf/Yr29naWLFnC\nZZddBjC4vLOzk6VLl3LVVVdx/vnn+xG65IhIKKQ+93FF49DT7ncUIoOKi4s5++yzefnll49aXlRU\nxJIlS1TyV4iGTS33cUXjkNAF1WnpoVtg9wvp/c4TzoT3fO24qwxUeQRoaGjgZz/72VGft7S0sG7d\nOj7/+c+zb9++weWtra1s376dCy+8ML0xS+BEstjnHtzkHgpDv2rLSPaMVf3x8ccfZ+nSpYRCIW65\n5RZOP/10HnvsMR5//HHOOusstm7dyi233MIJJ5zgQ9SSSyIh9bmPz8KarGO6GqeFnW0DfetjLd+2\nbRtve9vbuOqqqwZb/jI9RcMa5z6+UBj6E35HITKu+fPn89nPfpavf/3rfociPouEQyR0h+o41C0j\nAXLjjTeydu1aXn31Vb9DER9FQ0Zvn8vKJNnB7ZYJRaBfEx9I9oxW2nf58uUsX7583OVFRUUaLSNE\nw157OtHviIYzWx4yuC33wnLoavM7ChGRlEUGknsW+t2Dm9zjVdDe4ncUIiIpG2itZ6MyZHCTe7QQ\n+rr9jkJEJGWh5Ewd/VkY6x7c5C7TTjYuQmVbPu6TjC2U7GbPxn1MKSV3M7vEzLaaWZOZ3XKc9d5v\nZs7MGtMXoggUFhbS0tKSV8nQOUdLSwuFhYV+hyJZYsmWe06MljGzMLASuBhoBtab2Srn3OYR65UC\nNwNPZSJQmd5qampobm4+6rb+fFBYWEhNTY3fYUiWDMyfmo0mSipDIc8BmpxzrwCY2X3AlcDmEet9\nGfgGkJ3apqGIptmbRqLRKA0NDX6HITIlA4Mfs3ECmkq3zBxg17D3zcllg8xsKVDrnDv2HuxMKaqA\nrkO6kUlEgmOgWyYLbfdUkvtoI+0HIzOzEPBt4NPjfpHZDWa2wcw2TPn0uqjSC0Nj3UUkIAaTaY60\n3JuB2mHva4A3hr0vBc4AHjOzHcAyYNVoF1Wdc3c65xqdc40zZ86cfNQA8UrvuUOTZItIMGSzzz2V\n5L4emGdmDWYWA64BVg186Jxrc85VO+fqnXP1wDrgCufchoxEPCBW7D33dmR0MyIi6WIMjJbJ/LbG\nTe7OuQRwE/AwsAV4wDn3opndZmZXZDrAMYWi3nOf6suISDAMtdxzYCgkgHNuNbB6xLJbx1h3+dTD\nSkF4ILlrxIyIBEOujZbJTeGY96wSBCISELnW556biiq85w4VDxORYBjqc8+NoZC5qSw51L5NNbJF\nJCAGWu7qljmOwjKIlcIhJXcRCYbMTs9xtOAmd4DSE+Dwbr+jEBFJyVDhsMxvK9jJPVKgoZAiEhiD\no2VypPxA7ooWQW+731GIiKTE1OeeosIyr3iYiEgAaChkqkJRdcuISGBoKGSqZpwEbbvGX09EJAeo\n5Z6qeBV0HYQszCQuIpIu6nMfT0GJ96yLqiISAKEszqEa7OQeSyb37iP+xiEikgJ1y6SqoNR77lFy\nF5Hcl1P13HNacbX33D7FKftERLIgm/Xcg53c4wPJfb+/cYiIpED13FOlqfZEJEB0h2qqBpJ7j0bL\niEgQJPvc1S0zDiV3EQkQtdxTFY17z0ruIhIAqueeqlDYS/AaCikiAaB67hNRVgOtO/yOQkRkXKrn\nPhEV9dD6mt9RiIiMS33uE1FWC4ea/Y5CRGRcKj8wEWU10Nmq+jIikvNUz30iZszxng+/6W8cIiLj\nUct9AgpneM/dmm5PRHKbyg9MRGGZ96y5VEUkxw0MhcxG2z34yZ3s/WOJiEyFWu4TEa/0nlUZUkRy\nnEbLTERFPVgIWpr8jkRE5LhybrIOM7vEzLaaWZOZ3TLK558ys81m9ryZrTGzuekPdQyRAm84ZMvL\nWdukiMhkDN3ElAN97mYWBlYC7wEWAdea2aIRqz0LNDrnzgIeBL6R7kCPq/IUOKDkLiK5LZtXCFNp\nuZ8DNDnnXnHO9QD3AVcOX8E59zvn3MCMGeuAmvSGOY6qU6Hlleyc64iITFaOlR+YA+wa9r45uWws\n1wMPjfaBmd1gZhvMbMO+fWmc97TqFOhug46W9H2niEiaWY5N1jFaCeJRIzOz64BG4Jujfe6cu9M5\n1+ica5w5c2bqUY5n4C7VQ2+k7ztFRNIsi8PciaSwTjNQO+x9DXBMFjWzi4B/AN7unOtOT3gpKpnt\nPR/Zk9XNiohMRK71ua8H5plZg5nFgGuAVcNXMLOlwPeBK5xze9Mf5jhKk8n98O6sb1pEJFU5NVmH\ncy4B3AQ8DGwBHnDOvWhmt5nZFcnVvgmUAD8xs41mtmqMr8uM0pO8se4Hd2Z1syIiEzF0E1Pms3sq\n3TI451YDq0csu3XY64vSHNfERGIwowZaX/U1DBGR41H5gcmYcRIcUtlfEcldKj8wGbMXwRt/0qQd\nIpLDNFnHxJ15NfR2wEu/9jsSEZFRqeU+GbXLoKwOXnjA70hEREY1eNOQ+twnIBSChgth9ya/IxER\nGdXgUMgcuUM1OOIV3mTZiR6/IxEROYZGy0xWzVuhrxt2P+93JCIix7AcKxwWHCUneM9dB/2NQ0Rk\nFEOFwzIvv5J7JOY993b5G4eIyChyarKOQKk8GWKlGjEjIjlNLfeJKiyDZTfC5l9o1IyI5Bz1uU/F\neR/3Wu9Pf9/vSEREjmJZLPqbf8m9qMKbMFsTd4hIjlHLfapOXg5Nj8Azd/sciIjIkGyWH0ip5G/g\nvOvL0NIEv/okFM+C0y71OyIRkaGhkGq5T1I4Ch+4B05cAg9+CHY+5XdEIiJZnawjP5M7QKwY/vtP\nvDrvP/4g7Nvmd0QiMs2p/EC6FFfDdf8JoSj86L9pMg8R8ZVK/qZTZQNc96BXUOxH74NOlSYQEb9o\nso70OnExfPBe2L8N7no3PP0D7yan/j6/IxORacRs/HXSJT9Hy4zmlHfC+34Av/l7WP0Zb1nBDKg9\nx5voo24ZzDkbYnF/4xSRvJXNPvfpk9wBTr8KFr0XDr4GO9cNPZq+4n0einit/IFkX7cMSmb5G7OI\n5I1sTtYxvZI7eOdFFfXeY/E13rKOA9C8fijZr/8hrFvpfVZ58tHJvnp+ds+tRCRvqOWebfFKmP9u\n7wGQ6IY3nxtK9tsfhuf+w/usqBJqzx1K9icthUiBf7GLSGBks/yAkvtoIgXJvvhz4Pybvd9ES9NQ\nst+1DrY95K0bLvAS/ECyrz3XO1iIiIyQzck6lNxTYQbV87zHW/7SW3ZkH+x6CnY+6T0/uRKe+I73\nWfWCoWRftwwqGtSVIyKDaaA/C013JffJKpkJCy/zHgC9nfD6n4aS/Ys/hz/d430WKQRLYdRppBAK\nSqFwBhSUJZ9LvVE9R70uG315QSmEwpnbZxGZEstexV8l97SJFkH9+d4DoL8f9r3kJfvWV8f/eecg\n0QVdh6D7MHQfgoO7oLvNe991CFwK4/JjJaMcDGZApGiUswfzpiYMF3j1eCIFI17HvMfA60jys3DB\n0OtQdOJnJTNO8kozi0wzGi2TD0IhmL3Ie6SDc9DbMZTou5OPwddjLO86CAd3egeOkfr7oK8H+nqh\nr9u7kJyNJkUoCjVvHZrzdjyxEig9Ifk4ySsrkcqZUD6y0NFnagWl3r9PaJr+ewSMRsvIscy8Ymix\nYi/JZYJz0J/wEn6iO5n4eyDR4yX/4a8Tyc/6eye+jeb10LzB68pKZf0je+G1J7wSEjIKG0r0IxN/\nQemwbrxSiMYndqYViozoAhzoLpzhnb3pWtKE5Fw9dzO7BPguEAZ+6Jz72ojPC4B/B84GWoAPOud2\npDdUyTizZLdL1DuIZMrp753cz/V2wuHd3n0JWfnvkYP6E9B9ZKi7buAxvDuv+7B3xta2a+jzniPp\nj2Wg2w68g8A5N0ws2UfjUDLbu35VPMt7Ha+CcP62ObNZz33cf0UzCwMrgYuBZmC9ma1yzm0ettr1\nQKtz7lQzuwb4OvDBTAQs01i0yCsEV9ngdyTB09/nJfnejon9XF/PiC6/w9DVNtT115+ANzZ6Z1Zr\nv5GGQM1L8MOTfqrdd+nQsBxOOGNiP1NW453dpCCb9dxTOUSeAzQ5514BMLP7gCuB4cn9SuCLydcP\nAv9qZuayUfpMRMYXCkNRuffIBc55ZxNH9kL7Pu/5yJ5hr/dC+1448JR3AMmGrjZ49keT+9lT3nn0\n+3CBd1Nkee1Ri2OdfRTSnRstd2AOsGvY+2bg3LHWcc4lzKwNqAL2pyNIEckzNuw6QdUpfkfj6e2C\nlx/1riml6vVnvJneug+PWP75kHZ2AAAG00lEQVT7oRsdh6kAXiwM8WzLN4EPTync8aSS3EfrRBt5\n3EllHczsBuAGgLq6uhQ2LSKSJdHCic+3fPpVoy/vaoN9W49dvu03hFuaaFyQ+a7FVJJ7MzD83KIG\neGOMdZrNLAKUAQdGfpFz7k7gToDGxkZ12YhIfios88qXjDTasgxJZXDsemCemTWYWQy4Blg1Yp1V\nwP9Ivn4/8Kj620VE/DNuyz3Zh34T8DDeUMi7nHMvmtltwAbn3Crg34B7zawJr8V+TSaDFhGR40tp\nQKlzbjWwesSyW4e97gKuTm9oIiIyWbpnWUQkDym5i4jkISV3EZE8pOQuIpKHlNxFRPKQ+TUc3cz2\nAa9N8sermX6lDbTP04P2eXqYyj7Pdc7NHG8l35L7VJjZBudco99xZJP2eXrQPk8P2dhndcuIiOQh\nJXcRkTwU1OR+p98B+ED7PD1on6eHjO9zIPvcRUTk+ILachcRkePI6eRuZpeY2VYzazKzW0b5vMDM\n7k9+/pSZ1Wc/yvRKYZ8/ZWabzex5M1tjZnP9iDOdxtvnYeu938ycmQV6ZEUq+2tmH0j+nl80s//I\ndozplsLfdZ2Z/c7Mnk3+bU9w1ozcY2Z3mdleM9s0xudmZrcn/02eN7O3pDUA51xOPvDKC78MnAzE\ngOeARSPW+Z/AHcnX1wD3+x13Fvb5HUA8+fpj02Gfk+uVAmuBdUCj33Fn+Hc8D3gWqEi+n+V33FnY\n5zuBjyVfLwJ2+B13Gvb7QuAtwKYxPr8UeAhvJrtlwFPp3H4ut9wHJ+Z2zvUAAxNzD3clcE/y9YPA\nCjMbbcq/oBh3n51zv3PODUxhvw5vZqwgS+X3DPBl4BtAVzaDy4BU9vcjwErnXCuAc25vlmNMt1T2\n2QEzkq/LOHa2t8Bxzq1llBnphrkS+HfnWQeUm9mJ6dp+Lif30SbmnjPWOs65BDAwMXdQpbLPw12P\nd+QPsnH32cyWArXOuV9lM7AMSeV3PB+Yb2ZPmNk6M7ska9FlRir7/EXgOjNrxps74m+yE5qvJvr/\nfUJSmqzDJ2mbmDtAUt4fM7sOaATentGIMu+4+2xmIeDbwF9nK6AMS+V3HMHrmlmOd2b2uJmd4Zw7\nmOHYMiWVfb4WuNs5909mdh7ezG5nOOf6Mx+ebzKav3K55T6Ribk53sTcAZLKPmNmFwH/AFzhnOvO\nUmyZMt4+lwJnAI+Z2Q68vslVAb6omurf9S+cc73OuVeBrXjJPqhS2efrgQcAnHNPAoV49VfyWUr/\n3ycrl5P7dJyYe9x9TnZRfB8vsQe9LxbG2WfnXJtzrto5V++cq8e7znCFc26DP+FOWSp/1z/Hu3CO\nmVXjddO8ktUo0yuVfd4JrAAws4V4yX1fVqPMvlXAXyVHzSwD2pxzb6bt2/2+ojzO1eZLgW14V9r/\nIbnsNrz/3OD9AfwEaAKeBk72O+Ys7PMjwB5gY/Kxyu+YM73PI9Z9jACPlknxd2zAPwObgReAa/yO\nOQv7vAh4Am8kzUbgXX7HnIZ9/jHwJtCL10q/HrgRuHHY73ll8t/khXT/XesOVRGRPJTL3TIiIjJJ\nSu4iInlIyV1EJA8puYuI5CEldxGRPKTkLoFjZlVmtjH52G1mrydfHzSzzRnY3nIzm1DpAzN7bLQb\nrczsr83sX9MXncjolNwlcJxzLc65Jc65JcAdwLeTr5cA496unrybWSSvKblLvgmb2Q+SddB/a2ZF\nMNiS/kcz+z3wCTObaWY/NbP1ycf5yfXePuys4FkzK01+b4mZPWhmL5nZ/xuoPmpmK5LrvZCs310w\nMiAz+5CZbUtu+/ws/TvINKfkLvlmHl653NOBg8D7hn1W7px7u3Pun4Dv4rX435pc54fJdT4DfDx5\nJnAB0JlcvhT4W7w7KU8GzjezQuBu4IPOuTPxCn59bHgwyRKuX8JL6hcnf14k45TcJd+86pzbmHz9\nDFA/7LP7h72+CPhXM9uIV+NjRrKV/gTwz2Z2M97BIJFc/2nnXLPzqhRuTH7vguT2tiXXuQdvgobh\nzgUec87tc14t8/sRyQL1PUq+GV4lsw8oGva+fdjrEHCec66To33NzH6NVwtlXbIC52jfG2H0kq2j\nUY0PyTq13GW6+i1w08AbM1uSfD7FOfeCc+7rwAbgtON8x0tAvZmdmnz/l8DvR6zzFLA8OcInClyd\nrh0QOR4ld5mubgYakxMTb8ar1gfwt2a2ycyew+tvH3OmK+dcF/Ah4Cdm9gLeSJ07RqzzJt4sQ0/i\nVfT8U7p3RGQ0qgopIpKH1HIXEclDSu4iInlIyV1EJA8puYuI5CEldxGRPKTkLiKSh5TcRUTykJK7\niEge+v/+N/jKjdJ+1gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b782e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thresholds,tpr, thresholds,fpr)\n",
    "plt.legend(['TPR','FPR'])\n",
    "plt.xlabel('Threshold')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goodness of a classification method is often measured by the area under the curve (AUC), which is independent of the threshold. A method with a higher AUC means that under the same FPR, it has higher PPR. Note that in practice, you should evalute UAC in different cross validation folds and then evaluate the mean AUC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC=0.996297\n"
     ]
    }
   ],
   "source": [
    "auc=metrics.roc_auc_score(y,yprob[:,1])\n",
    "print(\"AUC=%f\" % auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## LASSO regularization\n",
    "As with linear regression, we can use LASSO regression to force the weighting coefficients to have small L1 norm, which will have the effect to reducing the number of non-zero coefficients. This is particularly important when there are many features and not enough training samples. the LogisticRegression method of sklearn allows one to add a regularization term (penalty), which could be either L2 or L1 norm. By setting the penalty to l1, you are effectively using LASSO regularization. The parameter C is inversely related to the regularization level. You can observe that as you reduces C, you will get fewer and fewer non-zero coefficients. To determine the appropriate C, we will use cross validation. Note that with the current example, we have sufficient number of samples for the number of features, so we do not expect to see the benefit of LASSO regression. First let us determine the C value leading to the minimum error rate (1-accuracy). We also count the number of non-zero coefficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold 0\n",
      "Fold 1\n",
      "Fold 2\n",
      "Fold 3\n",
      "Fold 4\n",
      "Fold 5\n",
      "Fold 6\n",
      "Fold 7\n",
      "Fold 8\n",
      "Fold 9\n",
      "The minimum test error rate =   2.9263e-02, SE=  5.7586e-03\n",
      "The C value corresponding to minimum error =   1.8330e-01\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xt8FPW9//HXJwmXgEoUJEoAwYIK\n3gVBxVoUVGyreMNLe1rbeg61rb9z6fGC5xyttZ6qta09/vB4qVqtv1q1eKOKRRGjVhHFIiBiMAIq\nYPECQYIBcvn8/phJuyy7ye5OZndD3s/HYx+Znfl+v/OZ2WQ/mfnOfMfcHRERkVyVFDoAERHp3JRI\nREQkEiUSERGJRIlEREQiUSIREZFIlEhERCSSWBOJmU0ysxozqzWzaSmW9zCzB8Pl881sSDh/iJk1\nmNkb4eu2hDqjzGxJWOdmM7M4t0FERNoWWyIxs1LgFuAUYCRwvpmNTCp2IbDB3YcBNwE3JCx7190P\nC18XJcy/FZgKDA9fk+LaBhERaV+cRyRjgFp3X+Hu24AHgMlJZSYD94bTM4AJbR1hmNnewG7uPs+D\nOyl/C5ze8aGLiEimymJsuwr4IOH9amBsujLu3mRmG4G+4bKhZrYQ+Az4L3d/MSy/OqnNqlQrN7Op\nBEculJeXjxo0aFBOG9HS0kJJSfF1JSmu7Ciu7Ciu7OyscS1fvvwTd9+zvXJxJpJURxbJ47GkK/Mh\nMNjdPzWzUcBjZnZghm0GM93vAO4AGD16tC9YsCDjwBNVV1czfvz4nOrGSXFlR3FlR3FlZ2eNy8ze\ny6RcnCl0NZB4GDAQWJuujJmVAX2A9e6+1d0/BXD314F3gf3C8gPbaVNERPIozkTyGjDczIaaWXfg\nPGBmUpmZwAXh9NnAXHd3M9sz7KzHzPYl6FRf4e4fApvM7KiwL+WbwOMxboOIiLQjtlNbYZ/HxcBs\noBS4292Xmtk1wAJ3nwncBdxnZrXAeoJkA3AccI2ZNQHNwEXuvj5c9j3gHqAceCp8iYhIgcTZR4K7\nzwJmJc27KmF6CzAlRb2HgYfTtLkAOKhjIxURkVwV32UGIiLSqSiRiIhIJEokIiISiRKJiIhEokQi\nIiKRKJGIiEgkSiQiIhKJEomIiESiRCIiIpEokYiISCRKJCIiEokSiYiIRKJEIiIikSiRiIhIJEok\nIiISiRKJiIhEokQiIiKRKJGIiEgkSiQiIhKJEomIiESiRCIiIpEokYiISCRKJCIiEokSiYiIRKJE\nIiIikSiRiIhIJEokIiISSayJxMwmmVmNmdWa2bQUy3uY2YPh8vlmNiRp+WAzqzezSxLmrTKzJWb2\nhpktiDN+ERFpX2yJxMxKgVuAU4CRwPlmNjKp2IXABncfBtwE3JC0/CbgqRTNH+/uh7n76A4OW0RE\nshTnEckYoNbdV7j7NuABYHJSmcnAveH0DGCCmRmAmZ0OrACWxhijiIhEFGciqQI+SHi/OpyXsoy7\nNwEbgb5m1hu4HPhxinYdeNrMXjezqR0edYJzb5/HdfMb4lyFiEinVxZj25ZinmdY5sfATe5eHx6g\nJBrn7mvNrD/wjJm97e4v7LDyIMlMBaisrKS6ujrb+Kmra6C5uTmnunGrr69XXFlQXNlRXNnp8nG5\neywv4GhgdsL7K4ArksrMBo4Op8uATwiSy4vAqvBVB6wHLk6xjquBS9qLZdSoUZ6Lc2572U+6flZO\ndeP23HPPFTqElBRXdhRXdhRXdqLGBSzwDL7v4zy19Row3MyGmll34DxgZlKZmcAF4fTZwNww/i+6\n+xB3HwL8Cvipu083s95mtitAePrrJODNGLdBRETaEdupLXdvMrOLCY46SoG73X2pmV1DkOVmAncB\n95lZLcFRx3ntNFsJPBqe7ioD7nf3P8W1DSIi0r44+0hw91nArKR5VyVMbwGmtNPG1QnTK4BDOzZK\nERGJQne2i4hIJEokIiISiRKJiIhEokQiIiKRKJGIiEgkSiQiIhKJEomIiESiRCIiIpEokYiISCRK\nJCIiEokSiYiIRKJEIiIikSiRiIhIJEokIiISiRKJiIhEokQiIiKRKJGIiEgkSiQiIhKJEomIiESi\nRCIiIpEokYiISCRKJCIiEokSiYiIRKJEIiIikSiRiIhIJEokIiISiRKJiIhEokQiIiKRxJpIzGyS\nmdWYWa2ZTUuxvIeZPRgun29mQ5KWDzazejO7JNM2RUQkv2JLJGZWCtwCnAKMBM43s5FJxS4ENrj7\nMOAm4Iak5TcBT2XZpoiI5FGcRyRjgFp3X+Hu24AHgMlJZSYD94bTM4AJZmYAZnY6sAJYmmWbIiKS\nR2Uxtl0FfJDwfjUwNl0Zd28ys41AXzNrAC4HTgQuSVW+jTYBMLOpwFSAyspKqqurs96AuroGmpub\nc6obt/r6esWVBcWVHcWVna4eV5yJxFLM8wzL/Bi4yd3rwwOUbNoMZrrfAdwBMHr0aB8/fnx78e7g\n1pp51NXVkUvduFVXVyuuLCiu7Ciu7HT1uOJMJKuBQQnvBwJr05RZbWZlQB9gPcFRxtlm9jOgAmgx\nsy3A6xm0KSIieRRnInkNGG5mQ4E1wHnA15LKzAQuAOYBZwNz3d2BL7YWMLOrgXp3nx4mm/baFBGR\nPIotkYR9HhcDs4FS4G53X2pm1wAL3H0mcBdwn5nVEhyJnJdLm3Ftg4iItC/OIxLcfRYwK2neVQnT\nW4Ap7bRxdXttiohI4ejOdhERiUSJZCd27u3zOPf2eQWrLyJdgxKJiIhEokQiIiKRKJGIiEgkSiRp\nPLZwDQvfr6NmQwvjrp/LYwvXFDokEZGipESSwmML13DFI0vY1twCwJq6Bq54ZImSiYhICkokKdw4\nu4aGxubt5jU0NnPj7JoCRSQiUrwySiRmdqyZfTuc3jMcomSntbauIav5IiJdWbuJxMx+RDCk+xXh\nrG7A/4szqEIbUFGe1XxJTfehiHQNmRyRnAGcBmwGcPe1wK5xBlVol568P+XdSrebV96tlEtP3r9A\nEYmIFK9MEsm2cEReBzCz3vGGVHinH17FdWceTPfSYPd0KzWuO/NgTj+8qsCRiYgUn0wSyUNmdjtQ\nYWb/BMwB7ow3rMI7/fAqDh9cQb9yo7HZOWrfvoUOSUSkKLWbSNz95wTPU38Y2B+4yt1vjjuwYrFL\nt+Dns2+vy/u61ccgIp1BJp3tN7j7M+5+qbtf4u7PmNkN+QiuGHQvgcF79GLOW/lPJCIinUEmp7ZO\nTDHvlI4OpFiZGRNHVPLSu5/y+bamQocjIlJ00iYSM/uemS0B9jezxQmvlcDi/IVYeBNH9GdbUwsv\nvvNJoUMRESk6bT0h8X7gKeA6YFrC/E3uvj7WqIrMkUP3YNeeZTy7bB0nH7hXocMRESkqaROJu28E\nNgLnA5hZf6AnsIuZ7eLu7+cnxMLrVlrC+P37M/ftj2hpcUpKrNAhiYgUjUw62081s3eAlcDzwCqC\nI5UuZeKI/nxSv403VtcVOhQRkaKSSWf7tcBRwHJ3HwpMAF6KNaoiNH6//pSWGM8u09VbIiKJMkkk\nje7+KVBiZiXu/hxwWMxxFZ0+vbpx5JDdmfPWRxnX0X0gXVvUz1/1VT9q/evm52eg2UwSSZ2Z7QK8\nAPzOzP4H6JLXwU4cUUnNuk18sP7zQociIlI0Mkkkk4HPgX8D/gS8C5waZ1DFasKISgCd3hIRSdBm\nIjGzUuBxd29x9yZ3v9fdbw5PdXU5Q/v15gt79mbOssxPb4mI7OzaTCTu3gx8bmZ98hRP0Zs4opL5\nKz/lsy2NhQ5FRKQoZHJqawuwxMzuMrObW19xB1asJo6spLHZeWH5x4UORUSkKGSSSJ4EriTobH89\n4dUuM5tkZjVmVmtm01Is72FmD4bL55vZkHD+GDN7I3wtMrMzEuqsMrMl4bIFmcTRkY4YvDu79+rG\nszq9JSICtD1ECgDufm8uDYf9K7cQDPq4GnjNzGa6+1sJxS4ENrj7MDM7D7gBOBd4Exjt7k1mtjew\nyMz+6O6tV4sd7+4FGfiqtMQ4/oDgLvem5hbKSjN67L2IyE4rzm/BMUCtu69w923AAwRXgCWaDLQm\nqhnABDMzd/88IWn0JHw6Y7GYOKKSus8bef29DYUORUSk4No9IomgCvgg4f1qYGy6MuHRx0agL/CJ\nmY0F7gb2Ab6RkFgceNrMHLjd3e9ItXIzmwpMBaisrKS6ujrrDaira6C5uXmHutbklBr85unXaTig\ne5v1gZzW3V79+vr6dtuNc/3pJMYVdf2tN1NdMbY8cv1M9ldHrz+T7W8rrkJ8folxFXL96epn8znm\nM/5UcRXD/kv1/RWHNhNJeHrqene/NIe2U41smHxkkbaMu88HDjSzEcC9ZvaUu28Bxrn72nAQyWfM\n7G13f2GHRoIEcwfA6NGjffz48VlvwK0186irqyNV3WPem8/yDQ0plyXWBxg//uis191e/erq6jbX\nHff600mMqxDrT1c/k/0V5/rTaSuuQu6/6upqKip6FGz96epn8znmc/+liqsYfv/TfX91tEwu/x1l\nZrkMd7saGJTwfiCwNl0ZMysD+gDbDVHv7suAzcBB4fu14c+PgEcJTqHl3cQRlaz4ZDMrPq4vxOpF\nRIpGJn0kC4HHzewbZnZm6yuDeq8Bw81sqJl1B84DZiaVmQlcEE6fDcx1dw/rlAGY2T4Ez4pfZWa9\nzWzXcH5v4CSCjvm8mzCiP4Cu3hKRLi+TPpI9gE+BExLmOfBIW5XCPo+LgdlAKXC3uy81s2uABe4+\nE7gLuM/MagmORM4Lqx8LTDOzRqAF+L67f2Jm+wKPhgdIZcD97v6nDLe1Qw3cvRcH7LUrzyxbxz8d\nt28hQhARKQqZXP777Vwbd/dZwKykeVclTG8BpqSodx9wX4r5K4BDc42no00cUcmtz7/Lhs3b2L13\n+k53EZGdWSYPthpoZo+a2Udmts7MHjazgfkIrthNHFlJc4tTvVynt0Sk68qkj+Q3BH0ZAwgu1/1j\nOK/LO6SqD3vu2qMoB3F8bOEaFr5fx/yV6xl3/VweW7gmr/VFpOvIJJHs6e6/CUf/bXL3e4A9Y46r\nUygpMSYc0J8Xaj5mW1NLocP5m8cWruGKR5awrTmIaU1dA1c8siTjZBC1voh0LZl0tn9iZv8A/D58\nfz5B57sQPKPkgdc+4NWV6zl2eL9ChwPAjbNraGhs3m5eQ2MzP3zoDa59clm79ddv3kpL0h0/DY3N\n3Di7htMPr+rIUEVkJ5BJIvkOMB24ieBqrZfDeQIcO6wfPcpKmLNsXVEkkpYWZ01d6sdrtjicdGBl\nu23cP//9lPPXpmlXRLq2TO5sP8vdT8tTPJ1OefdSjh3WjznL1vGjU0eS272bHeOD9Z9z6YxFaZdX\nVZTz0zMObred52s+TpmMupWWsOqTzQzp1ztSnCKyc8nkzvbkgRYlyYQRlaze0MDydYW5y72lxfnt\nvFWc/KsXeHPNZ5w7eiDl3bb/aMu7lXLpyftn1N6lJ+9PebfS7eZ1KzXMnEn/8wJ3vriC5uRzXyLS\nZWVyauslM5sOPEgwVAkA7v6X2KLqZCaM6A+Pwpxl69h/r13zuu73Pw2OQuavXM9x++3JdWceTFVF\nOUd/oR+XzVjMtuYWqirKufTk/TPu32gtl1z/qH378p+PLuHaJ5fxxOIPufHsQxhemd/tFZHik0ki\nOSb8eU3CPGf7O927tMrdenLIwD7MWbaOHxw/LC/rbHHn3pdXcf1Tb1NWYtxw1sGcM3rQ306tnX54\nFb9/NejrePC72Q/6lq7+nReMZuaitVw9cylfufnP/POEYXz3S1+gm57LItJltddHUgLc6u4P5Sme\nTmvCAZX86tnlfLxpK3vu2iPWdb336WZueHULNRuW8qXwKGRARW5DnWfLzJh8WBXjhvXjR48v5edP\nL2fWkr9y45RDOHBAn7zEICLFpb0+khbg4jzF0qlNHNkfd3ju7fhuTmxpce55aSWTfvUi729q4Wdn\nH8I93z4yb0kkUb9denDL14/gtn84go82bWXy9Jf4xdM1NKrvRKTLyeTU1jNmdgk79pGsT1+l6xm5\n924M6NOTOcvWcc6Rg9qvkKX3Pt3MpTMW8+rK9Xxpvz2ZPKCeM0d3/HqyNemgvTlq375c88Rb/N+5\ntQzYxeg3bAPvffo5C9+vY1tzC+Oun5tVH42IdC6Z3kcC8IOEeQ5oyNsEZsYJI/rz8Otr2NLYTM+k\nq56y1TpEybbmFg798Ww+39pMz+6l/OzsQ5gyaiDPP/98B0UeXUWv7vzynMM49ZAB/PsDCzjjf1+m\nrMRoCo9OWu+MB5RMRHZC7faQuvvQFC8lkRQmjqikobGZee9Gu/E/eYiSjQ1NNLvz7yftt12HerE5\n/oD+/Pex5fTqXvq3JNKq9c54Edn5pE0kZnZZwvSUpGU/jTOozuqoffvSq3spc5ati9TOz2a/vcMQ\nJy0Ov35hZaR286G8zGjY1pxyme6MF9k5tXVEcl7C9BVJyybFEEun17NbKccN35Nnl32Ee26dzm+t\n/Yy1dVtSLussX8TpOv8LcVGAiMSvrURiaaZTvZfQhBH9+etnW1i69rOs6m1pbObns2s4bfqfKUmz\ndzvLF3GqO+NLSyzjO+tFpHNpq7Pd00ynei+h4w/ojxlZnd5asGo9lz+8mHc/3sxZRwzkiMEVXPvk\nsu1Ob2UzxEmhJd8Z37tHKZu3NtOjTDctiuyM2kokh5rZZwRHH+XhNOH7nrFH1kn126UHRwzenTnL\n1tG7e9sXxdVvbeJnf3qb+155jwF9yrn3O2P40n7Bo1569yjLeYiTYpB4Z/x9F45lyu3zuGzGYg4c\n0IfBfXsVODoR6Uhp/0V091J3383dd3X3snC69X23fAbZ2UwY0Z8313zG1jYedvXc2x9x0i+f575X\n3uNbxwzh6X877m9JBIIv4sMHVzB26B68NO2ETpVEknUvK2H6+YdjBj+4/y9sbUrdGS8inZPONcTg\nxBHBMz/qPt+2w7L1m7fxrw8s5Nv3vEbvHmXMuOgYfnTqgfTukcktPZ3XoD168fMph7JkzUZ+msHD\ntUSk89i5v70KZFj/XRi8Ry/qPm+kcrfgLKC7M3PRWn78x7fYtKWRf5kwnO8f/wV6lEW7cbEzOenA\nvbjw2KHc9eeVjN23L18+eO9ChyQiHUBHJDEwM4b07UVdQyPzV65n7E/n8JWbX+RfHniDQXv04on/\n80X+7cT9ulQSaXX5pAM4bFAFl89YzHufbm6/QifVOjLB/JXrGXf93Kyfd1/o+i+vbezU8at+UL9m\nQ0tO9bOlRNKGB797NFeMzf6S28cWruGVFX8fimzdZ1t568NNnH7YAB753jF5f2ZJMeleVsL0rx1O\nSYnxg/v/wpbGna+/JHlkgtYhYjL9Yy6G+ve8ua1Tx6/6udfPhU5txeDG2TV/+xATvbZqA6XpbhLp\nQgbu3otfTDmUf/ztAn46axnXTD6o0CF1qBtn1+wwMkFDYzOXPbyYB1/7YLv5dXUN3L78le3mvf7+\nBrYlXaiRrn4qHVI/6dc37+tPU/8Lu7HD/srn+tPVT/wci+LzS1H/xtk1sV20o0QSg3R3oHeWO9Pz\nYeLISv7pi0P59YsrGTN0D756yIBCh9Rh0n3O25padnhEcbOzw7zkL4G26qcrt9PW95J22yhE/Imf\nY7Huvzi/f5RIYjCgopw1KT60znJner5cNukAFry3gWkPL+GgAX0Y0q93oUOKZFtTC9PnvpP2bt2q\ninIeumj7p1VWV1czfvz288ZdPzfl70+q+qnszPX/Y2zJDvsrn+tPVz/xcyzW/Rfn90+sfSRmNsnM\nasys1sympVjew8weDJfPN7Mh4fwxZvZG+FpkZmdk2mYxSDVESGe6Mz1fupWWMP1rR1C6E/SXLFm9\nkdOm/5mb59Yyep8Kenbb/k8rm88/6u9PR9TvnvTN0NniV/38fv/ElkjMrBS4BTgFGAmcb2Yjk4pd\nCGxw92HATcAN4fw3gdHufhjBAJG3m1lZhm0W3OmHV3HdmQfTPXyOeVVFOdedeXCnvqkwLlUV5fxi\nyqEsXfsZ1z75VqHDydrWpmZ+8XQNp//vS6zfvI27LhjNjO+N4/ozD8n584/6+9MR9b91UPdOHb/q\n5/f7J85TW2OAWndfAWBmDwCTgcRvi8nA1eH0DGC6mZm7f55Qpid/H9srkzaLQuIQIQ9+t/3D0a5s\n4shKph63L3e8sIKxQ/ty6qGdo79kyeqNXPKHRdSs28RZRwzkqq+OpE+vYNCHqJ9/oesfM6Abizbt\nUrD1q37H1K+rq2P25SdkXT9bcSaSKiDxEoPVwNh0Zdy9ycw2An2BT8xsLHA3sA/wjXB5Jm0CYGZT\ngakAlZWVVFdX57QR9fX1OdetC89TxlE/k7jiXH86iXFlU39MT2duRQmXPrSQzavfZq/eJR0afy6f\nY7r1N7Y4M2sbeXJlI7t1N/71iB4c1n8DC199KaP6idqKqxCfX2JchVx/uvrZfI75jD9VXMWw/5qb\nm3Oun404E0mq61yT+yHTlnH3+cCBZjYCuNfMnsqwTcL6dwB3AIwePdrHjx+fYdjbCzrRcqt7a808\ngHY7B3Opn0lcca4/ncS4sq0/4ogGvnLzi9xb241Hv38MFTWvZr3+RInrz+VzTBX/4tV1XPKHRSxf\n18jZowZy5Vf+fhSSSf1kbcVViM8vMa6Kih4FW3+6+tl8jvncf6niKuTn11q/rq4u5++vbMSZSFYD\ngxLeDwTWpimz2szKgD7A+sQC7r7MzDYDB2XYpnRSVRXl/PKcQ/nOPQv4yRPFdbZya1Mz/zPnHW5/\nYQX9dunOb751JMcf0L/QYYkUhTgTyWvAcDMbCqwheOLi15LKzAQuAOYBZwNz3d3DOh+Ep7P2AfYH\nVgF1GbQpndgJB1Ty3S/ty+3Pr6CsxGhqccZdPzfrYfRbh4jY1hwMEfGVwc2MzyKOxPpHXjuHEoN1\nm7YyZdRA/uurI+lTrgGwRVrFlkjCJHAxMBsoBe5296Vmdg2wwN1nAncB95lZLcGRSOvjfY8FpplZ\nI9ACfN/dPwFI1WZc2yCFsV//XTCDpvDmqzV1DUx7ZDGbtjZyykHtD/T41Jsf8t9PLttuiIjffAaD\nXlmVU/2P67cCMPW4ofzHl4vuIkGRgov1hkR3nwXMSpp3VcL0FmBKinr3Afdl2qbsXH75zDskP/J+\nS2MLVz62lCsfy+3/hsYWItUHeHLxX5VIRFLQne1SdNoayuEnkw9st/6Vj6dPFlHqa4gbkdSUSKTo\npBtipqqinG8cPaTd+rc9vyKW+hriRiQ1DSMvRSeOISK6l1DUQ0yIdGY6IpGi03p11mUzFrOtuYWq\nivKsrtpKVf8rg5sj1c/2qjGRrkSJRIpSRw8xke3dvRriRiRzOrUlIiKRKJGIiEgkOrW1E4t6Skan\ndEQkEzoiERGRSJRIREQkEiUSERGJRIlEREQiUSIREZFIlEhERCQSJRIREYlEiURERCLRDYlFTDcE\nikhnoCMSERGJRIlEREQiUSIREZFI1EcisVEfj0jXoCMSERGJRIlEREQiUSIREZFIlEhERCQSJRIR\nEYlEiURERCKJNZGY2SQzqzGzWjOblmJ5DzN7MFw+38yGhPNPNLPXzWxJ+POEhDrVYZtvhK/+cW6D\niIi0Lbb7SMysFLgFOBFYDbxmZjPd/a2EYhcCG9x9mJmdB9wAnAt8Apzq7mvN7CBgNlCVUO/r7r4g\nrthFRCRzcd6QOAaodfcVAGb2ADAZSEwkk4Grw+kZwHQzM3dfmFBmKdDTzHq4+9YY4+1wuiGva4v6\n+au+6ketX11dHamNTMV5aqsK+CDh/Wq2P6rYroy7NwEbgb5JZc4CFiYlkd+Ep7WuNDPr2LBFRCQb\ncR6RpPqC92zKmNmBBKe7TkpY/nV3X2NmuwIPA98AfrvDys2mAlMBKisrc87M9fX1ecvq2egKcdXV\nNQDk3F5i/Vziirr+THSFz7EjKa7s5C0ud4/lBRwNzE54fwVwRVKZ2cDR4XQZQd+Ihe8HAsuBcW2s\n41vA9PZiGTVqlOfqueeey7lunLpCXOfc9rKfc9vLHVI/l7iirj8TXeFz7EiKKztR4wIWeAbf93Ge\n2noNGG5mQ82sO3AeMDOpzEzggnD6bGCuu7uZVQBPEiSel1oLm1mZmfULp7sBXwXejHEbRESkHbEl\nEg/6PC4mOOpYBjzk7kvN7BozOy0sdhfQ18xqgR8CrZcIXwwMA65Musy3BzDbzBYDbwBrgF/HtQ0i\nItK+WIeRd/dZwKykeVclTG8BpqSody1wbZpmR3VkjCIiEo3ubBcRkUiUSEREJBIlEhERiUSJRERE\nIlEiERGRSGK9akskimIYq0hE2qcjEhERiUSJREREIlEiERGRSJRIREQkEiUSERGJRIlEREQiUSIR\nEZFIlEhERCQSJRIREYlEiURERCJRIhERkUiUSEREJBIlEhERiUSJREREIlEiERGRSJRIREQkEiUS\nERGJRIlEREQiUSIREZFIlEhERCQSJRIREYlEiURERCKJNZGY2SQzqzGzWjOblmJ5DzN7MFw+38yG\nhPNPNLPXzWxJ+POEhDqjwvm1ZnazmVmc2yAiIm2LLZGYWSlwC3AKMBI438xGJhW7ENjg7sOAm4Ab\nwvmfAKe6+8HABcB9CXVuBaYCw8PXpLi2QURE2hfnEckYoNbdV7j7NuABYHJSmcnAveH0DGCCmZm7\nL3T3teH8pUDP8Ohlb2A3d5/n7g78Fjg9xm0QEZF2lMXYdhXwQcL71cDYdGXcvcnMNgJ9CY5IWp0F\nLHT3rWZWFbaT2GZVqpWb2VSCIxeAejOryXE7+iXFUywUV3YUV3YUV3Z21rj2yaRQnIkkVd+FZ1PG\nzA4kON11UhZtBjPd7wDuaD/MtpnZAncfHbWdjqa4sqO4sqO4stPV44rz1NZqYFDC+4HA2nRlzKwM\n6AOsD98PBB4Fvunu7yaUH9hOmyIikkdxJpLXgOFmNtTMugPnATOTyswk6EwHOBuY6+5uZhXAk8AV\n7v5Sa2F3/xDYZGZHhVdrfRN4PMZtEBGRdsSWSNy9CbgYmA0sAx5y96Vmdo2ZnRYWuwvoa2a1wA+B\n1kuELwaGAVea2Rvhq3+47HuHwtsvAAAGnElEQVTAnUAt8C7wVFzbEIp8eiwmiis7iis7iis7XTou\nCy5+EhERyY3ubBcRkUiUSEREJBIlkiRmdqOZvW1mi83s0bDjP1W5Nod/iSGuKWa21MxazCzt5Xxm\ntiocQuYNM1tQRHHle3/tYWbPmNk74c/d05RrTuiHS74YpCPjyWm4oLhlENe3zOzjhH30j3mI6W4z\n+8jM3kyz3MLhkWrDv9Mj4o4pw7jGm9nGhH11VZ7iGmRmz5nZsvBv8V9SlIl3n7m7XgkvgntWysLp\nG4AbUpQpJejo3xfoDiwCRsYc1whgf6AaGN1GuVVAvzzur3bjKtD++hkwLZyelupzDJfV52Eftbv9\nwPeB28Lp84AHiySubwHT8/X7FK7zOOAI4M00y79McJGNAUcB84skrvHAE/ncV+F69waOCKd3BZan\n+Bxj3Wc6Ikni7k97cMUZwCtsf99Kq0yGf+nouJa5e65358cmw7jyvr/YfvideynsUDo5DxdUBHHl\nnbu/QHg/WRqTgd964BWgIhw+qdBxFYS7f+jufwmnNxFcJZs84kes+0yJpG3fIfXlxamGf0k5VEsB\nOPC0BaMmT223dH4UYn9VenDfEeHP/mnK9TSzBWb2ipnFlWwy2f7thgsCWocLilOmn8tZ4emQGWY2\nKMXyfCvmv7+jzWyRmT0VjsyRV+Ep0cOB+UmLYt1ncQ6RUrTMbA6wV4pF/+nuj4dl/hNoAn6XqokU\n8yJfR51JXBkY5+5rw/tunjGzt8P/pAoZV973VxbNDA73177AXDNb4n8fSaGjRB4uKCaZrPOPwO89\nGOvuIoKjphN2rJZXhdhXmfgLsI+715vZl4HHCEYozwsz2wV4GPhXd/8seXGKKh22z7pkInH3iW0t\nN7MLgK8CEzw8wZgkk+FfOjyuDNtYG/78yMweJTh9ESmRdEBced9fZrbOzPZ29w/DQ/iP0rTRur9W\nmFk1wX9zHZ1IshkuaLUlDRcUo3bjcvdPE97+mr8/6qGQYvl9iirxy9vdZ5nZ/5pZP3ePfTBHM+tG\nkER+5+6PpCgS6z7Tqa0kZjYJuBw4zd0/T1Msk+Ff8s7MepvZrq3TBBcOpLzCJM8Ksb8Sh9+5gBRD\n6ZjZ7mbWI5zuB4wD3oohlpyHC4ohlqziSjqPfhrB+fdCmwl8M7wS6ShgY+tpzEIys71a+7XMbAzB\n9+unbdfqkPUawSghy9z9l2mKxbvP8n2FQbG/CIZe+QB4I3y1XkkzAJiVUO7LBFdHvEtwiifuuM4g\n+K9iK7AOmJ0cF8HVN4vC19JiiatA+6sv8CzwTvhzj3D+aODOcPoYYEm4v5YAF8YYzw7bD1xD8A8L\nQE/gD+Hv36vAvnHvowzjui78XVoEPAcckIeYfg98CDSGv1sXAhcBF4XLjeChee+Gn1vaqxjzHNfF\nCfvqFeCYPMV1LMFpqsUJ31tfzuc+0xApIiISiU5tiYhIJEokIiISiRKJiIhEokQiIiKRKJGIiEgk\nSiQiWTCz+gh1Lw5HX/XwvpXW+WlHZjWzvc3siYT3Y8zsBQtG7H3bzO40s15m9lUz+3HuWyaSOyUS\nkfx5CZgIvJc0/xSCoTSGA1OBWxOW/ZDgjnLMrJLgXpPL3X1/gpGX/0Qw4uuTwGlm1ivODRBJRYlE\nJAfhUcSNZvamBc9/OTecXxIOjbHUzJ4ws1lmdjaAuy9091UpmmtrZNazCJIFwA+Ae919Xtieu/sM\nd1/nwQ1h1QRD+4jklRKJSG7OBA4DDiU4yrgx/PI/ExgCHAz8I3B0Bm2lHJnVzIYCG9x9azj/IOD1\nNtpZAHwxi20Q6RBKJCK5OZZgVNxmd18HPA8cGc7/g7u3uPtfCYYVaU+6kVn3Bj7OIqaPCIamEckr\nJRKR3KR76FQuD6NKNzJrA8EYXK2WAqPaaKdnWEckr5RIRHLzAnCumZWa2Z4Ej2F9FfgzwYOgSsLO\n8fEZtJVuZNblBKfJWk0HLjCzsa0zzOwfzKz1mSz7URyjPUsXo0QikptHCUZbXQTMBS4LT2U9THCE\n8SZwO8GT6jYCmNk/m9lqgiOOxWZ2Z9jWLGAFwci/vyZ4fjvuvhl418yGhe/XEQz1/vPw8t9lBH0i\nrc/BOJ7g6i2RvNLovyIdzMx28eApeX0JjlLGhUkml7bOAEa5+3+1U64SuN/dJ+SyHpEouuQTEkVi\n9oSZVQDdgZ/kmkQA3P3RMCG1ZzDw77muRyQKHZGIiEgk6iMREZFIlEhERCQSJRIREYlEiURERCJR\nIhERkUj+P40vt0Ko7WJhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a12f606a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "npen = 20\n",
    "C_test = np.logspace(-2,2,npen)\n",
    "\n",
    "# Create the cross-validation object and error rate matrix\n",
    "nfold = 10\n",
    "kf = KFold(n_splits=nfold,shuffle=True)\n",
    "err_rate = np.zeros((npen,nfold))\n",
    "num_nonzerocoef = np.zeros((npen,nfold))\n",
    "# Create the logistic regression object\n",
    "logreg = linear_model.LogisticRegression(penalty='l1',warm_start=True)\n",
    "\n",
    "# Loop over the folds in the cross-validation\n",
    "for ifold, Ind in enumerate(kf.split(Xs)):        \n",
    "            \n",
    "    # Get training and test data\n",
    "    Itr, Its = Ind\n",
    "    Xtr = Xs[Itr,:]\n",
    "    ytr = y[Itr]\n",
    "    Xts = Xs[Its,:]\n",
    "    yts = y[Its]\n",
    "    \n",
    "    # Loop over penalty levels\n",
    "    for ipen, c in enumerate(C_test):\n",
    "        \n",
    "        # Set the penalty level        \n",
    "        logreg.C= c\n",
    "    \n",
    "        # Fit a model on the training data\n",
    "        logreg.fit(Xtr, ytr)\n",
    "    \n",
    "        # Predict the labels on the test set.\n",
    "        yhat = logreg.predict(Xts)\n",
    "        \n",
    "        # Measure the accuracy\n",
    "        err_rate[ipen,ifold] = np.mean(yhat != yts)\n",
    "        num_nonzerocoef[ipen,ifold]=np.sum(abs(logreg.coef_)>0.001)\n",
    "    print(\"Fold %d\" % ifold)\n",
    "    \n",
    "err_mean = np.mean(err_rate, axis=1)\n",
    "num_nonzerocoef_mean = np.mean(num_nonzerocoef, axis=1)\n",
    "err_se = np.std(err_rate,axis=1)/np.sqrt(nfold-1)\n",
    "plt.errorbar(np.log10(C_test), err_mean, marker='o',yerr=err_se)\n",
    "plt.ylim([0.02,0.05])\n",
    "plt.grid()\n",
    "plt.xlabel('log10(C)')\n",
    "plt.ylabel('Error rate')\n",
    "\n",
    "imin = np.argmin(err_mean)\n",
    "\n",
    "print(\"The minimum test error rate = %12.4e, SE=%12.4e\" % (err_mean[imin], err_se[imin]))\n",
    "print(\"The C value corresponding to minimum error = %12.4e\" % (C_test[imin]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us find the optimal C using one standard error rule. Note that because C is inversely related to the regularization strength, we want to find the minimal C that satisfies the error criterion. We will also find the corresponding test accuracy with this optimal C, and compare with the test accuracy without regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal C=  1.6238e-02\n",
      "The test error rate =   3.0754e-02, SE=  5.5593e-03\n",
      "Accuracy =  0.9692, SE=0.0056\n"
     ]
    }
   ],
   "source": [
    "err_tgt = err_mean[imin] + err_se[imin]\n",
    "iopt = np.where(err_mean < err_tgt)[0][0]\n",
    "C_opt = C_test[iopt]\n",
    "\n",
    "print(\"Optimal C=%12.4e\" % C_opt)\n",
    "print(\"The test error rate = %12.4e, SE=%12.4e\" % (err_mean[iopt], err_se[iopt]))\n",
    "\n",
    "print('Accuracy =  {0:.4f}, SE={1:.4f}'.format(1-err_mean[iopt], err_se[iopt]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now plot the nubmer of non-zero coefficients for the first class for different C values. Also determine and print the number of non-zero coefficients corresponding to C_opt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of non-zero coefficients for the optimal C = 5.900000\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XecVPW9//HXh12KsPTFFZDelIgo\nuyJFETQ2EvXa0jQxV3OtUaPGlN/NjSk3NybEFG+uxkTTjBEFe68gigrs0ruURTosfZey7fP7Ywaz\n4O5yZpczZ3bm/Xw85rFTTnnPWfjM2e98z/dr7o6IiKS/ZlEHEBGR5FDBFxHJECr4IiIZQgVfRCRD\nqOCLiGQIFXwRkQyhgi8ikiFU8EVEMoQKvohIhsiOOkBNubm53rt37watW1ZWRps2bY5uoKNAuRKj\nXIlRrsSkY66ioqISd+8SaGF3T5lbfn6+N9SUKVMavG6YlCsxypUY5UpMOuYCCj1gjVWTjohIhlDB\nFxHJECr4IiIZQgVfRCRDqOCLiGSIUAu+md1uZgvNbJGZfSvMfYmISP1CK/hmdhLwH8BwYCjweTMb\nENb+RESkfmFeeHUi8KG77wUws3eAS4FfhrhPEUmitdv38v7KEtbv2HfI88VrypldvixpOTrntGRU\nv870PzYHM0vafpsa85DmtDWzE4HngJHAPuAtYhcI3HrYctcD1wPk5eXlT5w4sUH7Ky0tJScnp1GZ\nw6BciVGuxCQ71+5yZ+m2KhZtq2Lxtiq27vtX/Ti0zPqnngnTwRQdWhondmrG4M5ZDO6cRedjDm3E\nSMff47hx44rcvSDIsqEVfAAzuw64BSgFFgP73P2OupYvKCjwwsLCBu1r6tSpjB07tkHrhkm5EqNc\niQk7V9mBSmau3s70FSVMX7mNJRt3A9C2ZTYj+nVmdL/OjO6f+6kz62Qfr4N/aUxfsY33V5ZQUloO\nQO/OrRndP5fR/XMZ2bcz82a9n3a/RzMLXPBDHUvH3R8BHomH+h9gXZj7E5HGKa+sZu7anUxfUcL7\nK0uY8/FOKqudFtnNKOjVkbvPH8Sofp0Z0r092Vmp08mvR6fWfLFTT754Wk/cneWbS3lvRQnvryjh\nubkbeGzGx5hBz7bNuGDvEkb1z+W03h1p3SKlhhMLXajv1syOdfctZtYTuIxY846kmdUlZazcUtqo\nbQzu1o5uHY45SokkEZt37+f5uRt4b0UJs4q3s7e8CjM4uXt7/mNMX0b3y6Wgd0daNc+KOmogZsag\n49oy6Li2XHdGHyqqqpm/bhfTV5TwUuEK/jK9mIemraJ5lnFqz46M7pfLiV3b0izCtv+lJZWMTcJ+\nwv54e8rMOgMVwC3uviPk/UkSuTuPz1zLj55fRHlVdaO21TzL+MrwnpzaqnHbkcTs3FvOZQ+8z/qd\n++jXpQ1X5B/PqH6x5o/2rZtHHe+oaJ7VjPxeHcnv1ZGTs9Zz+qgzmVV8sJmqhN++tZwQW7YDadfC\n+OYV4e8n7CadM8PcvkRnf0UVP3xuIU8WruOsgV2449yBZDXwDKmiuppJhev4x4yPedyc5b6UG8b0\nS5uCk6qqq527npzHlj37eeqmkeT36hR1pKQ4pkUWYwZ2YczA2IjCO8rKWXdYL6NkmzO7Yd9dJiqz\nGrDkqFi/cx83/aOI+et2cdvZ/bn9swPJata4P4eH9ezIDWP68r3H3uXBd1by6IdruPGsfvz76N4Z\n186aLA9NW8VbS7fw44s/kzHFvjYd27SgY5sWkWbYtiI5zWWp862LNAnTV5Rw0f++x+qtZfzpawXc\ned6gRhf7g3rntuHGoa14+bYzGd67ExNeW8aYX07lr9NXc6Cy6qjsQ2JmrNrGr15fxudO7srXRvaK\nOo4kiQq+BOLu/OGdlXz1kRnk5rTguW+O5tzBeaHs68Su7Xjk66fx1E0j6delDT96YTFn/+odJhWu\npao64sbWNLB1zwFufXwOvTq15heXn6wLlTKICr4cUemBSm5+bDb3vrKUC4d05ZmbR9O3S/gXr+T3\n6sTE60fw92uH06lNC+6ePJ/zfzuNVxZsJMzrR9JZVbVz+8Q57N5fwQNXDyOnpZrLMol+21KvFVtK\nufEfRawuKeM/x5/IN87sk9QzQjNjzMAunDkgl1cXbuJXry/jpsdmM6R7e+4+fxBnDsjVGWoCfvvm\nct5fuY0JV5zMCce1izqOJJkKvtTp1YWb+PakebTMbsaj1w1nVL/cyLKYGRcO6cp5nzmOZ+as5zdv\nLOdrf57J6X068Z0LBmX0l45BTV22hf99ewVfKDieKwt6RB1HIqCCL59SVe3c9/oyHpi6kqE9OvDg\nVcNS5qKorGbGFfnHc9HQrkycuZb/fXsFlz/4AeeccCx3nTeIwd101lqb9Tv3cccTcznhuLb85JKT\noo4jEVHBl0NsLyvn9olzePejEr48vCc/ungwLbNT7wrLltlZXDOqN1cWHB+7cvKdlYy//10uHtqN\nO84dSJ/cNlFHTBnlldXc8thsKqqcB6/ObzJXzMrRp4Ivn1i4fhc3PFrE1j0HuPeyIXxpeM+oIx1R\n6xbZ3DKuP1ef3ouHpq3kL9OLeWnBRr5Q0IPbzulP1/ap8ZdJlH7+yhLmrt3JA1cN0wdhhlMvHQFg\nUuFaLnvwfdydSTeObBLFvqb2rZvznQtO4J3vjOXq03syuWgtZ02Yys9eWsz2svKo40Xm5QUb+cv0\nYr4+qjfjh3SNOo5ETAVf+PkrS7h78nwKenXkhVvPYGiPDlFHarBj27bix5ecxNt3jeWik7vxyHur\nGfPLKfz2zeXs2V8RdbykWl1Sxncmz+eUHh34f+NPjDqOpAAV/Aw3d+1OHnpnFV8s6MHfrx1O55yW\nUUc6Knp0as19XxjKa98awxn9c/ntmx8x5pdTePjdVeyvSP+rdvdXVHHTP4rIzjL+76phtMjWf3VR\nwc9o7s7PX15C5zYt+K+LBqfU+OZHy4C8tvzhq/k8d8toTurenv9+aQljJ0zl8ZkfU9HIET5T2T3P\nLWLppj385oun0D1FelhJ9NLvf7gENmXZFmas3s7tnx2Q9ldcDu3RgUevO53H/2MEXTu04vtPL+C8\n30zj+XkbqE6z4RomF63jicK13DKuH+MGHRt1HEkhKvgZqtqdX7yyjN6dW/PlJvYFbWOM7NeZp28a\nxcNfK6BldjNue3wO4+9/l8lF69i8e3/U8Rpt6abd/ODZBYzo24k7Pjsw6jiSYtL7tE7qNH19Jcs2\n7+WBq4bRPA2bcupjZnx2cB5nn3AsL8zfwK/fWM63J80DoF+XNozun9skJwE5OOZR21bNuf/Lp6Zl\nE500jgp+BtpXXsXTH1VwSo8OXHjScVHHiUyzZsYlp3TnopO7sXjj7k8mwZ5UuI6/f7CGZgZDurdn\nVP9cckqrGFFRlbIXLbk733tqPsUlZTz2jREc27ZV1JEkBangZ6C/vL+aHQecP1x4ggYeI1b4T+re\nnpO6t+f6Mf0or6xmzsc7mL5yG++vKOFP01ZRWe38bu7r5PfsyBkDclNuIu9HP1zDi/M3cvf5gxjZ\nr3PUcSRFqeBnmO1l5Tw4ZSWndMni9L4qDLVpkd2M0/t25vS+nbnz3IGUHqjkkeensueYbkxfuY0J\nry0DoG3LbE7v25nR/TvTt0sOUXx0LiypZOec9fz0xcWMG9SFm87qF0EKaSpU8DPM799eQVl5JVcO\nUle9oHJaZjO0SzZjxw4GoKT0AB+s3PZJE9CbSzZHG7BwLt07HMOvv3AKzY7S7GOSnlTwM8ja7Xt5\n9MNirszvQfec7VHHabJyc1py0dBuXDS0GxA7rlv2RNPDZ/bsOQwbdioD8trSrlXT+YJZoqGCn0F+\n9foyspoZd5w7kKVzPow6Ttro0ak1PTq1jmTfe1ZnaS4ACSw1vnGS0C1Yt4vn5m7gujP6cFx79eAQ\nyUQq+BnA3bn31SV0bN2cG/SlnkjGUsHPANM+in25eOvZA9TOK5LBVPDTXFW1c+8rS+nZqTVXj+gV\ndRwRiZAKfpp7ds56lmzczbfPH6QhckUynCpAGttfUcWv31jOkO7t+bxmOxLJeCr4aezvHxSzfuc+\nvn/hCbogR0RU8NPVzr3l/P7tFYwd1IVR/XOjjiMiKUAFP009MHUlew5U8t0LTog6ioikCBX8NLR+\n5z7++n4xl516PCd2bRd1HBFJESr4aei+12OjOd51nmY8EpF/CbXgm9kdZrbIzBaa2eNmpmv6Q7Z4\nw26embOefx/dm26avFpEagit4JtZd+A2oMDdTwKygC+FtT+JuffVpbRr1Zybz+ofdRQRSTFhN+lk\nA8eYWTbQGtgQ8v4y2nsflTBt+VZuPbt/k5qLVUSSw9w9vI2b3Q78DNgHvO7uV9WyzPXA9QB5eXn5\nEydObNC+SktLycnJaUTacCQrV7U7P/5gP6Xlzr1jjqH5EfrdZ/rxSpRyJUa5EtOYXOPGjSty94JA\nC7t7KDegI/A20AVoDjwLXF3fOvn5+d5QU6ZMafC6YUpWrmfnrPNe333Rn569NtDymX68EqVciVGu\nxDQmF1DoAetymE06nwVWu/tWd68AngZGhbi/jHWgsooJry1jcNd2XDK0e9RxRCRFhVnwPwZGmFlr\nMzPgHGBJiPvLWI9+sIZ1O/bx/fEaQkFE6taggm9mPzrSMu4+A5gMzAYWxPf1x4bsT+q2a18Fv5+y\ngjMH5HLmgC5RxxGRFNbQOW2Lgizk7vcA9zRwHxLAH95Zyc69FRpCQUSOqM4zfDP7RfznlYe/5u4v\nhBlKgtm4ax9/fm81l57anZO6t486joikuPqadMabWXPg+8kKI4n59evLcYc7z9UQCiJyZPU16bwK\nlABtzGx3jecNcHfXqFwRWrZpD0/NXse1o/vQo1PrqOOISBNQ3xn+D9y9PfCSu7ercWurYh+9X7y6\nlJyW2XzzbA2hICLB1FfwP4j/3F3PMhKBD1Zu4+2lW7h5XH86tG4RdRwRaSLqa9JpYWbXAKPM7LLD\nX3T3p8OLJXVxd+59ZQld27fi66N6Rx1HRJqQ+gr+jcBVQAfgosNec2JXzkqSvbRgI/PW7WLCFSfT\nqnlW1HFEpAmps+C7+3vAe2ZW6O6PJDGT1KG8spoJry3jhOPactmw46OOIyJNTJArbSea2Q/M7I8A\nZjbAzD4fci6pxT9nrGHNtr1898ITyNIQCiKSoCAF/89AOf8a+Gwd8N+hJZJa7dlfwf1vr2Bk386M\nHaghFEQkcUEKfj93/yVQAeDu+4j1xZck+uO0VWwvK+f7408gNhadiEhighT8cjM7htgXtZhZP+BA\nqKnkEJt37+dP767ioqHdOPn4DlHHEZEmKsjgafcQu+q2h5k9BowGvh5mKDnUb99cTlW1c/d5g6KO\nIiJN2BELvru/YWazgRHEmnJud/eS0JMJACu27OGJWWu5ZlRvenbWEAoi0nBBh0ceDYyp8fjFELJI\nLe59ZRltWmRz69kDoo4iIk3cEdvwzexe4HZgcfx2u5n9POxgArOKt/Pmks3cOLYfndpoCAURaZwg\nZ/jjgVPcvRrAzP4GzEHDJofK3fmfl5eQ164l147uE3UcEUkDQac4rNk1RDNtJMFrizYx5+Od3Hnu\nQI5poSEURKTxgpzh/xyYY2ZTiH1pOwad3YeqoqqaX7y6jAHH5nC5hlAQkaMkSC+dx81sKnAasYL/\nXXffFHawTDZx1lpWl5TxyDUFZGc1aJ55EZFPCfKl7aXAXnd/3t2fA/ab2b+FHy0zlR6o5HdvLmd4\nn06cfcKxUccRkTQS5PTxHnffdfCBu+8kdjGWhOBP01ZRUlrO9y/UEAoicnQFKfi1LRO0/74kYMue\n2BAK44ccx6k9O0YdR0TSTJCCX2hmvzazfmbW18x+AxSFHSwT3f/WR5RXVnP3+SdEHUVE0lCQgn8r\nseGRnwCeBPYBt4QZKhOt3FrK4zPX8pXTe9Int03UcUQkDQXppVMGfC8JWTLahFeX0Sq7GbedoyEU\nRCQc6vOXAorW7ODVRZu44ax+5Oa0jDqOiKQpFfyI7Sgr50fPL6JL25Z840wNoSAi4VFvmwgtXL+L\nGx4tYuueA9z/5VNp3UK/DhEJT5ALr443s2fMbKuZbTazp8xM1/s30uSidVz+4PtUu/PkjSO54KTj\noo4kImkuSJPOX4Dnga5Ad+CF+HPSAOWV1fzg2QV8e9I8hvXsyAu3nsEpPTRtoYiEL0gbQhd3r1ng\n/2pm3worUDrbtGs/Nz9WxOyPd3LDmL7cff4gjZUjIkkTpOCXmNnVwOPxx18GtoUXKT3NWLWNW/45\nh73llfzfV4bxuZO7Rh1JRDJMkNPLa4EvAJuAjcAV8efqZWaDzGxujdvuTPzLwN3583ur+crDM2jX\nKptnbxmtYi8ikaj3DN/MsoDL3f3iRDfs7suAU2psZz3wTENCNlV7yyt5aP4BPty4mHMH53HfF4bS\nrlXzqGOJSIaq9wzf3auAS47Cfs4BVrr7mqOwrSZhzbYyLnvgfWZsrOLb5w3koavzVexFJFLm7vUv\nYPYzYtMaPgGUHXze3WcH3onZn4HZ7v77Wl67HrgeIC8vL3/ixIlBN3uI0tJScnJyGrTu0TZvayUP\nzTuAGVwz0BneIzVy1ZRKx6sm5UqMciUmHXONGzeuyN0LAi3s7vXegCm13N4+0no11m8BlAB5R1o2\nPz/fG2rKlCkNXvdoqaqq9t+8scx7f+9Fv/C30/zjbWUpkas2ypUY5UqMciWmMbmAQg9Yj4MMnjau\nQR87/3IhsbP7zY3cTkrbta+CO5+Yy1tLt3DZqd352aVDOKZFFiujDiYiEnfEgm9mecD/AN3c/UIz\nGwyMdPdHAu7jy/yrS2daWrppNzc8WsT6Hfv4ySWf4asjemm2KhFJOUG6Zf4VeA3oFn+8HAjUvdLM\nWgPnAk83JFxTcKCyiqsfnsG+8iqeuGEEXxvZW8VeRFJSkIKf6+5PAtUA7l4JVAXZuLvvdffOXmNO\n3HTz5uItlJSWM+HKoeT36hR1HBGROgUp+GVm1hlwADMbAaRtAU/Uk4Vr6dq+FWf0z406iohIvYIM\nrXAXscHT+pnZdKALcGWoqZqITbv28+5HW7l5bH+ymqkZR0RSW5BeOkVmdhYwCDBgmbtXhJ6sCXhq\n9jqqHa7I12jRIpL6goyHvxL4hrsvcveF7l5hZi8mIVtKc3cmFa5leJ9O9Nak4yLSBARpw68AxpnZ\nX8ysRfy57iFmahIK1+ygeNtevlDQI+ooIiKBBCn4e939i8AS4F0z60X8C9xM9uSstbRpkcX4IZqp\nSkSahiBf2hqAu//SzIqI9cnP6P6HZQcqeWnBRi46uZvmoRWRJiNItfrhwTvu/paZnQd8PbRETcBL\nCzayt7yKKwv0Za2INB1Beum8YGbdgV41lp8aZqhUN7lwHX1z25Dfq2PUUUREAgsyls69wJeAxfzr\nClsHpoWYK2WtLiljZvF2vnPBIA2hICJNSpAmnUuBQe5+IOwwTcHkorU0M7h8mJpzRKRpCdJLZxWg\nqZqAqmrnqaL1nDWwC3ntWkUdR0QkIUHO8PcCc83sLeCTs3x3vy20VCnq3Y+2smn3fu65aHDUUURE\nEhak4D8fv2W8SYXr6Ni6OeecmBd1FBGRhAXppfO3+BW2A+NPZeRYOjvKynlj8WauGtGTFtlBWsJE\nRFJLkF46Y4G/AcXELsLqYWbXuHtG9dJ5bu56yququTJfQymISNMUpEnnPuA8d18GYGYDiU1ZmB9m\nsFQzqWgdJ3Vvx+Bu7aKOIiLSIEHaJpofLPYA7r6cDOu1s2jDLhZt2K2zexFp0oKc4Rea2SPAo/HH\nVwFF4UVKPZMK19EiqxmXnNLtyAuLiKSoIAX/JuAW4DZibfjTgAfCDJVKDlRW8ezc9Zz7mTw6tG5x\n5BVERFJUkF46B4Bfx28Z560lW9i5t0Lj3otIkxekl85o4EccOnga7t43vFipQ5OUi0i6CNKk8whw\nB7F2+6ojLJtWNu3az7TlmqRcRNJDkIK/y91fCT1JCtIk5SKSToIU/ClmNgF4mkPH0pkdWqoU4O5M\nLlqnScpFJG0EKfinx38W1HjOgbOPfpzUUbhmB6tLyrh5bL+oo4iIHBVBeumMS0aQVDOp8OAk5V2j\njiIiclRoFLBalB2o5MX5G/ncyV1p01KTlItIelDBr8XL8UnK1fdeRNJJnQXfzK6M/+yTvDipYZIm\nKReRNFTfGf734z+fSkaQVHFwkvIrCo7XJOUiklbqa6DeZmZTgD5m9qkZr9z94vBiRUeTlItIuqqv\n4H8OGEZslMz7khMnWpqkXETSWZ0F393LgQ/NbJS7bzWztrGnvTToxs2sA/AwcBKxvvvXuvsHjQ0d\nlvdWlLBp935+qEnKRSQNBelzmGdmrwOdADOzrcA17r4wwLq/A1519yvi8+K2bkTW0D1ZuDY+Sfmx\nUUcRETnqgnTL/CNwp7v3cveewF3x5+plZu2AMcQGX8Pdy919Z2PChmnn3nLeWLSZS07pTsvsrKjj\niIgcdebu9S9gNs/dhx7puVrWO4XYB8NiYCix0TZvd/eyw5a7HrgeIC8vL3/ixIkJvwmA0tJScnJy\nGrQuwJtrKvjHknJ+MqoVPdsdvYLf2FxhUa7EKFdilCsxjck1bty4IncvOPKSxAYJq+8GPAP8F9A7\nfvsB8GyA9QqASuD0+OPfAT+tb538/HxvqClTpjR4XXf3z90/zcf/blqjtlGbxuYKi3IlRrkSo1yJ\naUwuoNCPUI8P3oI06VwLdCE2WubTQC7w7wHWWwesc/cZ8ceTifX6STmLN+xm4frdurJWRNJakMHT\ndhCbzzYh7r7JzNaa2SB3XwacQ6x5J+VMKlqrScpFJO2FPTLYrcBj8R46qwj2l0FSVVc7z8/dwLmD\nNUm5iKS3UAu+u8/l0HH0U85HW0rZVlbO2EFdoo4iIhKqjB8tc2bxdgCG9+kUcRIRkXAd8Qw/Plrm\nrcR66HyyvKfJWDqzVm/n2LYt6dkppa8JExFptCBNOs8Su3jqBaA63DjJ5e7MKt7OaX06aWRMEUl7\nQQr+fne/P/QkEVi3Yx8bd+1neG8154hI+gtS8H9nZvcArwMHDj7p7rNDS5Uks+Lt96ep4ItIBghS\n8IcAXwXO5l9NOh5/3KTNKt5O21bZDDqubdRRRERCF6TgXwr09dhwyWll5urtFPTqSFYztd+LSPoL\n0i1zHtAh7CDJtq30ACu3lnGaumOKSIYINB4+sNTMZnFoG36T7pZZuGYHgL6wFZGMEaTg3xN6igjM\nWr2dFtnNGHJ8+6ijiIgkRZDB095JRpBkm1W8nVOO76DJTkQkYxyxDd/M9pjZ7vhtv5lVmdnuZIQL\nS9mBShZu2M1pfTpGHUVEJGmCnOEf0mfRzP4NGB5aoiSY8/FOqqpd/e9FJKMkPHiauz9LE++DP7N4\nO80M8nvpDF9EMkeQwdMuq/GwGbHhjuufCDfFzVq9nRO7tqNtq+ZRRxERSZogvXQuqnG/EigGLgkl\nTRKUV1YzZ+0OvnRaz6ijiIgkVZA2/JSbpaoxFm7Yxf6Kao1/LyIZp86Cb2Y/rGc9d/efhpAndLNW\na8A0EclM9Z3hl9XyXBvgOqAz0DQLfvF2+uS2oUvbllFHERFJqjoLvrvfd/C+mbUFbic2CflE4L66\n1ktl1dXOrOIdnP+ZvKijiIgkXb1t+GbWCbgTuAr4GzDM3XckI1gYPtpSyq59FWrOEZGMVF8b/gTg\nMuCPwBB3L01aqpBownIRyWT1XXh1F9AN+AGwocbwCnua6tAKhcWasFxEMld9bfgJX4Wb6mat1oTl\nIpK50q6o12Xdjr1s0ITlIpLBMqbga8JyEcl0GVPwZ67eoQnLRSSjZUzBn1WsCctFJLNlRMHfXlbO\nii2lFKg5R0QyWEYU/Fnqfy8ikiEFPz5h+cmasFxEMlhmFHxNWC4ikv4FXxOWi4jEBJnxqsHMrBjY\nA1QBle5eEOb+aqMJy0VEYkIt+HHj3L0kCfuplSYsFxGJSfsmHU1YLiISY+4e3sbNVgM7AAcecvc/\n1rLM9cD1AHl5efkTJ05s0L5KS0vJyck55LnKaufmN/dyVo9srjoxmhmuasuVCpQrMcqVGOVKTGNy\njRs3rihwc7m7h3YDusV/HgvMA8bUt3x+fr431JQpUz71XNGa7d7ruy/6S/M3NHi7jVVbrlSgXIlR\nrsQoV2Iakwso9IA1OdQmHXffEP+5BXgGGB7m/g5XqAHTREQ+EVrBN7M28blwMbM2wHnAwrD2V5uZ\nq3downIRkbgwe+nkAc/EJxvJBv7p7q+GuL9DVFc7hWu2c95gTVguIgIhFnx3XwUMDWv7R7Jiayk7\n92rCchGRg9K2W+bM1RowTUSkprQt+LM0YbmIyCHSt+BrwnIRkUOkZcHXhOUiIp+WlgVfE5aLiHxa\nWhZ8TVguIvJpaVnwZxVvJ18TlouIHCLtCv7BCcvVnCMicqi0K/iasFxEpHbpV/A1YbmISK3Sr+Br\nwnIRkVqlVcHfW64Jy0VE6pJWBV8TlouI1C2tCv7M1ZqwXESkLmlV8GcVa8JyEZG6pE3Br6x25ny8\nU805IiJ1SJuCv2Z3NfsqqtT/XkSkDmlT8JfvqAY0YJqISF3SqOBXacJyEZF6pEXBr652lu+o4rTe\n6p0jIlKXtCj4K7aWUlah5hwRkfqkRcHXhOUiIkeWFgV/VvF2OrQ0TVguIlKP9Cj4q7czsGMzTVgu\nIlKP7KgDNNaByipG98+lU8XWqKOIiKS0Jn+G3zI7iwlXDmVktyb/2SUiEqomX/BFRCQYFXwRkQyh\ngi8ikiFU8EVEMoQKvohIhlDBFxHJECr4IiIZQgVfRCRDmLtHneETZrYVWNPA1XOBkqMY52hRrsQo\nV2KUKzHpmKuXu3cJsmBKFfzGMLNCdy+IOsfhlCsxypUY5UpMpudSk46ISIZQwRcRyRDpVPD/GHWA\nOihXYpQrMcqVmIzOlTZt+CIiUr90OsMXEZF6NNmCb2YTzGypmc03s2fMrEMdy11gZsvMbIWZfS8J\nua40s0VmVm1mdX7rbmbFZrbAzOaaWWEK5Ur28epkZm+Y2Ufxnx3rWK4qfqzmmtnzIeap9/2bWUsz\neyL++gwz6x1WlgRzfd3Mttaksq63AAAGX0lEQVQ4Rt9IQqY/m9kWM1tYx+tmZvfHM883s2FhZwqY\na6yZ7apxrH6YpFw9zGyKmS2J/1+8vZZlwj1m7t4kb8B5QHb8/i+AX9SyTBawEugLtADmAYNDznUi\nMAiYChTUs1wxkJvE43XEXBEdr18C34vf/15tv8f4a6VJOEZHfP/AzcAf4ve/BDyRIrm+Dvw+Wf+e\n4vscAwwDFtbx+njgFcCAEcCMFMk1Fngxmccqvt+uwLD4/bbA8lp+j6EesyZ7hu/ur7t7Zfzhh8Dx\ntSw2HFjh7qvcvRyYCFwScq4l7r4szH00RMBcST9e8e3/LX7/b8C/hby/+gR5/zXzTgbOsfAnU47i\n93JE7j4N2F7PIpcAf/eYD4EOZtY1BXJFwt03uvvs+P09wBKg+2GLhXrMmmzBP8y1xD4VD9cdWFvj\n8To+fYCj4sDrZlZkZtdHHSYuiuOV5+4bIfYfAji2juVamVmhmX1oZmF9KAR5/58sEz/h2AV0DilP\nIrkALo83A0w2sx4hZwoilf//jTSzeWb2ipl9Jtk7jzcFngrMOOylUI9ZSk8Ea2ZvAsfV8tJ/uvtz\n8WX+E6gEHqttE7U81+huSUFyBTDa3TeY2bHAG2a2NH5mEmWupB+vBDbTM368+gJvm9kCd1/Z2GyH\nCfL+QzlGRxBkny8Aj7v7ATO7kdhfIWeHnOtIojhWQcwmNhxBqZmNB54FBiRr52aWAzwFfMvddx/+\nci2rHLVjltIF390/W9/rZnYN8HngHI83gB1mHVDzTOd4YEPYuQJuY0P85xYze4bYn+2NKvhHIVfS\nj5eZbTazru6+Mf6n65Y6tnHweK0ys6nEzo6OdsEP8v4PLrPOzLKB9oTffHDEXO6+rcbDPxH7Xitq\nofx7aqyaRdbdXzazB8ws191DH2PHzJoTK/aPufvTtSwS6jFrsk06ZnYB8F3gYnffW8dis4ABZtbH\nzFoQ+5IttB4eQZlZGzNre/A+sS+ga+1RkGRRHK/ngWvi968BPvWXiJl1NLOW8fu5wGhgcQhZgrz/\nmnmvAN6u42QjqbkOa+e9mFj7cNSeB74W73kyAth1sPkuSmZ23MHvXcxsOLE6uK3+tY7Kfg14BFji\n7r+uY7Fwj1myv6k+WjdgBbG2rrnx28GeE92Al2ssN57Yt+EriTVthJ3rUmKf0geAzcBrh+ci1tti\nXvy2KFVyRXS8OgNvAR/Ff3aKP18APBy/PwpYED9eC4DrQszzqfcP/ITYiQVAK2BS/N/fTKBv2Mco\nYK6fx/8tzQOmACckIdPjwEagIv5v6zrgRuDG+OsG/F888wLq6bWW5FzfrHGsPgRGJSnXGcSaZ+bX\nqFvjk3nMdKWtiEiGaLJNOiIikhgVfBGRDKGCLyKSIVTwRUQyhAq+iEiGUMGXtGRmpY1Y95vx0Qo9\n3u//4PN1jmRoZl3N7MUaj4eb2TSLjXC51MweNrPWZvZ5M/txw9+ZSMOp4It82nTgs8Caw56/kNgl\n+AOA64EHa7x2J7ErXDGzPGJ99b/r7oOIjVT6KrEREl8CLjaz1mG+AZHaqOBLWouflU8ws4UWm3/g\ni/Hnm8UvqV9kZi+a2ctmdgWAu89x9+JaNlffSIaXEyvqALcAf3P3D+Lbc3ef7O6bPXbhy1RiQ4KI\nJJUKvqS7y4BTgKHEztonxIv0ZUBvYAjwDWBkgG3VOpKhmfUBdrj7gfjzJwFF9WynEDgzgfcgclSo\n4Eu6O4PYKJJV7r4ZeAc4Lf78JHevdvdNxIYjOJK6RjLsCmxNINMWYkNaiCSVCr6ku7omJ2nIpCV1\njWS4j9gYOwctAvLr2U6r+DoiSaWCL+luGvBFM8sysy7Epr+bCbxHbMKQZvEvWccG2FZdIxkuJ9Y8\ndNDvgWvM7PSDT5jZ1WZ2cE6AgaTG6KiSYVTwJd09Q2x0wnnA28B34k04TxE7Y18IPERs5qFdAGZ2\nm5mtI3YGP9/MHo5v62VgFbGRMv9EbH5b3L0MWGlm/eOPNxMbwvhX8W6ZS4i12R8ch30csd46Ikml\n0TIlY5lZjsdmPepM7Kx/dPzDoCHbuhTId/cfHGG5POCf7n5OQ/Yj0hgpPeOVSMheNLMOQAvgpw0t\n9gDu/kz8g+NIegJ3NXQ/Io2hM3wRkQyhNnwRkQyhgi8ikiFU8EVEMoQKvohIhlDBFxHJECr4IiIZ\n4v8DlAu8X20d+yYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a130eae48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_nonzerocoef_mean = np.mean(num_nonzerocoef, axis=1)\n",
    "plt.plot(np.log10(C_test),num_nonzerocoef_mean)\n",
    "\n",
    "plt.grid()\n",
    "plt.xlabel('log10(C)')\n",
    "plt.ylabel('Num of nonzero coeff.')\n",
    "\n",
    "print(\"The number of non-zero coefficients for the optimal C = %f\" % num_nonzerocoef_mean[iopt])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the optimal `C`, fit the model on the entire training data with l1 regularization. Find the resulting weight matrix, `W_l1`.  Plot the first row of this weight matrix and compare it to the first row of the weight matrix without the regularization.  You should see that, with l1-regularization, the weight matrix is much more sparse. Please also compare the accuracy for the training data using optimal `C` with the previous results not using LASSO regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on the training data is 0.970717\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5,1,'l1-regularization')"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbUAAAGrCAYAAABHdgk2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xt8XWWd7/HP17SFUMAArUrTQutY\nOzKi1MlUkTkjI5cWL7TH45HWy6gHp7w84t0yrTMHEGcGx3p0xpFRK3bwgi1YS6dKncAMcryBNKVI\noRitBdokaAMlXCRD2/R3/lgruBqS7J3sna69V77v12u/2OtZa6/1S0jzzXr2s59HEYGZmVkRPCfv\nAszMzKrFoWZmZoXhUDMzs8JwqJmZWWE41MzMrDAcamZmVhgONbMaIulMSR0VvP5tkm6qZk3peT8u\n6epqn9es2uTPqVnRSHoAaAReGBG/S9veA7w9Is7MsbSSJJ0JfDMipo/nGsxGy3dqVlQTgA9W+6SS\nJlT7nNVSy7WZHS4ONSuqlcDHJDUNtlPSqyVtlvRY+t9XD3UiSQ9I+itJdwO/kzRB0jRJ35HULel+\nSR/IHN8o6WuSHpV0n6RLsl2KkkLSizLb10j62yGuvVzSryU9IWm7pP+e2fcuST+R9DlJe4HL07Yf\np/svkfRk5rFf0jXpvnentT0haaeki9L2ycD3gWmZ102TdLmkb2aufb6keyX1SLpV0ksGfL8+Junu\n9Pt7naQjh/r+mlWTQ82Kqg24FfjYwB2SjgduBD4PnAB8FrhR0gnDnG8J8HqgCTgIfBf4OdAMnAV8\nSNL89NjLgJnAC4FzgLdX8HX8GvhvwHOBTwDflHRiZv8rgZ3A84C/y74wIj4dEUdHxNHAS4Bu4Pp0\n9x7gDcCxwLuBz0l6Rdpdex7Q1f/aiOjKnlfSi4E1wIeAqcAm4LuSJmUOewuwAJgFvAx4VwXfA7Oy\nOdSsyC4F3i9p6oD21wO/iohvRMSBiFgD/AJ44zDn+nxE7I6IXuBPgKkRcUVE7IuIncBXgMXpsW8B\n/j4iHo2IDpLwHJWI+HZEdEXEwYi4DvgVMC9zSFdE/HP6dfQOdg5JjcAG4J8iYlN63hsj4teR+H/A\nTSThWY4LgBsj4uaI2A98huQ9zOzd7ufTuveS/AFw2gi+bLNRcx+8FVZE3CPpe8By4L7MrmnAgwMO\nf5DkrmsouzPPTybpnuvJtDUAP8qcP3t89vmISPoL4CMkd34ARwNTRnjurwLtEfEPmfOeR3JH+WKS\nP26PAraVWdYh37+IOChpN4d+/36Tef5U+hqzMec7NSu6y4C/5NBfuF0kwZR1EtA5zHmyw4R3A/dH\nRFPmcUxEvC7d/xCQHTk4Y8C5niIJkX4vGOyCkk4muQO8GDghIpqAewANUddg51gOzAEuzLQdAXyH\n5A7r+el5N2XOW2pI9CHfP0ki+RqH+/6ZHRYONSu0iNgBXAd8INO8CXixpLemgz4uAE4Bvlfmae8A\nHk8HjzRKapD0Ukl/ku6/Hlgh6ThJzSShlHUX8Nb0dQuA1wxxnckkAdMNyeAO4KVl1th/N/YBYNGA\nrslJwBHpeQ+kx52b2f9b4ARJzx3i1NcDr5d0lqSJwEeBp4Gfllub2VhxqNl4cAVJQAAQEY+QDJL4\nKPAIcAnwhoh4uJyTRUQfyftvpwH3Aw8DV5MM5ui/Xke67z+AdSS/9Pt9MH19D/A2kve7BrvOduD/\nAreRBM2pwE/KqTF1AclAjvsyIxm/FBFPkITd9cCjwFuBjZnr/oJkIMjOdHTjIV2HEdFOMvjln9Ov\n/Y3AGyNi3whqMxsT/vC12RiT9F5gcUQMdUdmZlXiOzWzKpN0oqQzJD1H0hySO8Ib8q7LbDzw6Eez\n6psEfJnkM1o9wFrgX3KtyGyccPejmZkVhrsfzcysMGqy+3HKlCkxc+bMvMswM7MasWXLlocjYuDs\nQM9Sk6E2c+ZM2tra8i7DzMxqhKSBswANyt2PZmZWGA41MzMrDIeamZkVRk2+p1aJDVs7WdnaTldP\nL9OaGlk2fw6L5g43+bqZmRVFyTs1Sasl7ZF0zxD7z0xXt70rfVya2bdAUrukHels4WNqw9ZOVqzf\nRmdPLwF09vSyYv02Nmz15OFmZuNBOd2P15CsYDucH0XEaenjCgBJDcBVJKvongIskXRKJcWWsrK1\nnd79fYe09e7vY2Vr+1he1szMakTJUIuIHwJ7R3HuecCOiNiZzt69Flg4ivOUratn0IV/h2w3M7Ni\nqdZAkdMl/VzS9yX9UdrWzKGr8nYwzMrCkpZKapPU1t3dPaoipjU1jqjdzMyKpRqhdidwckS8nGR9\npf61oTTIsUNONBkRqyKiJSJapk4t+aHxQS2bP4fGiQ2HtDVObGDZ/DmjOp+ZmdWXikMtIh6PiCfT\n55uAiZKmkNyZZZexn06yDPyYWTS3mSvfdCqTGpIvq7mpkSvfdKpHP5qZjRMVD+mX9ALgtxERkuaR\nBOUjJEtuzJY0C+gEFpOssDumFs1tZs0duwC47qLTx/pyZmZWQ0qGmqQ1wJnAFEkdwGXARICI+BLw\nZuC9kg4AvSQr/AZwQNLFQCvQAKyOiHvH5KswMzOjjFCLiCUl9n8B+MIQ+zYBm0ZXmpmZ2ch4miwz\nMysMh5qZmRVG4eZ+NLPDz3OuWq1wqJlZRfrnXO2foq5/zlXAwWaHnbsfzawinnPVaolDzcwq4jlX\nrZY41MysIp5z1WqJQ83MKuI5V62WeKCImVWkfzDIJevuZl/fQZo9+tFy5FAzs4p5zlWrFe5+NDOz\nwnComZlZYTjUzMysMBxqZmZWGA41MzMrDIeamZkVhkPNzMwKo2SoSVotaY+ke4bY/zZJd6ePn0p6\neWbfA5K2SbpLUls1CzczMxuonDu1a4AFw+y/H3hNRLwM+CSwasD+P4+I0yKiZXQlmpmZlafkjCIR\n8UNJM4fZ/9PM5u3A9MrLMjMzG7lqv6d2IfD9zHYAN0naImnpcC+UtFRSm6S27u7uKpdlZmbjQdXm\nfpT05ySh9qeZ5jMiokvS84CbJf0iIn442OsjYhVp12VLS0tUqy4zMxs/qnKnJullwNXAwoh4pL89\nIrrS/+4BbgDmVeN6ZmZmg6k41CSdBKwH3hERv8y0T5Z0TP9z4Fxg0BGUZmZm1VCy+1HSGuBMYIqk\nDuAyYCJARHwJuBQ4AfgXSQAH0pGOzwduSNsmAN+KiH8fg6/BzMwMKG/045IS+98DvGeQ9p3Ay5/9\nCjMzs7HhGUXMzKwwHGpmZlYYDjUzMysMh5qZmRWGQ83MzArDoWZmZoXhUDMzs8Ko2tyPZmb1ZMPW\nTla2ttPV08u0pkaWzZ/DornNeZdlFXKomdm4s2FrJyvWb6N3fx8AnT29rFi/DcDBVufc/Whm487K\n1vZnAq1f7/4+Vra251SRVYtDzczGna6e3hG1W/1wqJnZuDOtqXFE7VY/HGpmNu4smz+HxokNh7Q1\nTmxg2fw5OVVk1eKBImY27vQPBrlk3d3s6ztIs0c/FoZDzczGpUVzm1lzxy4Arrvo9JyrsWpx96OZ\nmRWGQ83MzAqjrFCTtFrSHkn3DLFfkj4vaYekuyW9IrPvnZJ+lT7eWa3CzczMBir3Tu0aYMEw+88D\nZqePpcAXASQdD1wGvBKYB1wm6bjRFmtmZjacskItIn4I7B3mkIXA1yNxO9Ak6URgPnBzROyNiEeB\nmxk+HM3MzEatWu+pNQO7M9sdadtQ7c8iaamkNklt3d3dVSrLzMzGk2oN6dcgbTFM+7MbI1YBqwBa\nWloGPcZqg2c3N7NaVa1Q6wBmZLanA11p+5kD2m+t0jUtB/U4u7lD2Gz8qFb340bgL9JRkK8CHouI\nh4BW4FxJx6UDRM5N26xO1dvs5v0h3NnTS/D7EN6wtTPv0sxsDJR1pyZpDckd1xRJHSQjGicCRMSX\ngE3A64AdwFPAu9N9eyV9EticnuqKiBhuwInVuHqb3Xy4EPbdmtnYyauHpKxQi4glJfYH8L4h9q0G\nVo+8NKtF05oa6RwkwGp1dvN6C2GzIsjzbQrPKGIjUm+zm3uJEbPDL8+3KRxqNiKL5jZz5ZtOZVJD\n8qPT3NTIlW86tWa78uothM2KIM8eEs/SbyNWT7Obe4kRs8Mvz7cpfKdmhbdobjNzT2rilbOO5yfL\nX+tAMxtjefaQ+E7NzMyqKs8eEoeamZlVXV5vU7j70czMCsOhZmZmheFQMzOzwnComZlZYTjUzMys\nMBxqZmZWGA41MzMrDIeamZkVhkPNzMwKw6FmZmaFUVaoSVogqV3SDknLB9n/OUl3pY9fSurJ7OvL\n7NtYzeLNzMyySs79KKkBuAo4B+gANkvaGBHb+4+JiA9njn8/MDdzit6IOK16JZuZmQ2unDu1ecCO\niNgZEfuAtcDCYY5fAqypRnFmZmYjUU6oNQO7M9sdaduzSDoZmAXckmk+UlKbpNslLRrqIpKWpse1\ndXd3l1GWmZnZocoJNQ3SFkMcuxhYFxF9mbaTIqIFeCvwj5L+YLAXRsSqiGiJiJapU6eWUZaZmdmh\nygm1DmBGZns60DXEsYsZ0PUYEV3pf3cCt3Lo+21mZmZVU06obQZmS5olaRJJcD1rFKOkOcBxwG2Z\ntuMkHZE+nwKcAWwf+FozM7NqKDn6MSIOSLoYaAUagNURca+kK4C2iOgPuCXA2ojIdk2+BPiypIMk\nAfqp7KhJMzOzaioZagARsQnYNKDt0gHblw/yup8Cp1ZQn5mZWdk8o4iZmRWGQ83MzAqjrO5HMzPL\n14atnaxsbaerp5dpTY0smz+HRXMH/cjwuOZQMzOrcRu2drJi/TZ69ycfAe7s6WXF+m0ADrYB3P1o\nZlbjVra2PxNo/Xr397GytT2nimqXQ83MrMZ19fSOqH08c6iZmdW4aU2NI2ofzxxqZmY1btn8OTRO\nbDikrXFiA8vmz8mpotrlgSJmZjWufzDIJevuZl/fQZo9+nFIDjUzszqwaG4za+7YBcB1F52eczW1\ny92PZmZWGA41MzMrDIeamZkVhkPNzMwKw6FmZmaF4VAzM7PCcKiZmVlhlBVqkhZIape0Q9LyQfa/\nS1K3pLvSx3sy+94p6Vfp453VLN7MzCyr5IevJTUAVwHnAB3AZkkbI2L7gEOvi4iLB7z2eOAyoAUI\nYEv62kerUr2ZmVlGOXdq84AdEbEzIvYBa4GFZZ5/PnBzROxNg+xmYMHoSjUzMxteOaHWDOzObHek\nbQP9D0l3S1onacYIX4ukpZLaJLV1d3eXUZaZmdmhygk1DdIWA7a/C8yMiJcB/wF8bQSvTRojVkVE\nS0S0TJ06tYyyzMzMDlVOqHUAMzLb04Gu7AER8UhEPJ1ufgX443Jfa2ZmVi3lhNpmYLakWZImAYuB\njdkDJJ2Y2TwfuC993gqcK+k4SccB56ZtZmZmVVdy9GNEHJB0MUkYNQCrI+JeSVcAbRGxEfiApPOB\nA8Be4F3pa/dK+iRJMAJcERF7x+DrMDMzK289tYjYBGwa0HZp5vkKYMUQr10NrK6gRjMzs7J4RhEz\nMysMh5qZmRWGQ83MzArDoWZmZoXhUDMzs8JwqJmZWWE41MzMrDAcamZmVhgONTMzKwyHmpmZFYZD\nzczMCsOhZmZmheFQMzOzwnComZlZYTjUzMysMBxqZmZWGA41MzMrjLJCTdICSe2SdkhaPsj+j0ja\nLuluSf8p6eTMvj5Jd6WPjdUs3szMLGtCqQMkNQBXAecAHcBmSRsjYnvmsK1AS0Q8Jem9wKeBC9J9\nvRFxWpXrNjMze5Zy7tTmATsiYmdE7APWAguzB0TEDyLiqXTzdmB6dcs0MzMrrZxQawZ2Z7Y70rah\nXAh8P7N9pKQ2SbdLWjTUiyQtTY9r6+7uLqMsMzOzQ5XsfgQ0SFsMeqD0dqAFeE2m+aSI6JL0QuAW\nSdsi4tfPOmHEKmAVQEtLy6DnNxsvNmztZGVrO109vUxramTZ/Dksmjvc35JmBuWFWgcwI7M9Hega\neJCks4G/Bl4TEU/3t0dEV/rfnZJuBeYCzwo1M0ts2NrJivXb6N3fB0BnTy8r1m8DcLCZlVBO9+Nm\nYLakWZImAYuBQ0YxSpoLfBk4PyL2ZNqPk3RE+nwKcAaQHWBiZgOsbG1/JtD69e7vY2Vre04VmdWP\nkndqEXFA0sVAK9AArI6IeyVdAbRFxEZgJXA08G1JALsi4nzgJcCXJR0kCdBPDRg1aWYDdPX0jqjd\nzH6vnO5HImITsGlA26WZ52cP8bqfAqdWUqDZeDOtqZHOQQJsWlNjDtWY1RfPKGJWY5bNn0PjxIZD\n2honNrBs/pycKjKrH2XdqZnZ4dM/GOSSdXezr+8gzR79aFY2h5pZDVo0t5k1d+wC4LqLTs+5GrP6\n4e5HMzMrDIeamZkVhkPNzMwKw6FmZmaF4VAzM7PCcKiZmVlhONTMzKwwHGpmZlYYDjUzMysMh5qZ\nmRWGQ83MzArDoWZmZoXhUDMzs8JwqJmZWWGUFWqSFkhql7RD0vJB9h8h6bp0/88kzczsW5G2t0ua\nX73SzczMDlVyPTVJDcBVwDlAB7BZ0saI2J457ELg0Yh4kaTFwD8AF0g6BVgM/BEwDfgPSS+OiL5q\nfyH1bMPWTla2ttPV08s0LwhpZjZq5SwSOg/YERE7ASStBRYC2VBbCFyePl8HfEGS0va1EfE0cL+k\nHen5bqtO+YNbcOu3eEH3bh788bFjeZmqePjJp9n/8O/4wMF4pm3/jWLLlMlMOfqIHCsb3rseehyg\nLr7HUH/1Qv3VXG/1Qv3VXI/1/mbqDDiMC92W0/3YDOzObHekbYMeExEHgMeAE8p8LQCSlkpqk9TW\n3d1dXvVDOH7yERw1qaGicxwuu/f2cjATaAAHDwa79/bmVFF5jprUUDffY6i/eqH+aq63eqH+aq7H\neo+ffHj/OC/nTk2DtEWZx5Tz2qQxYhWwCqClpWXQY8q18OrPVPLyw+rM5TcO+g0RcP+nXn+4yynb\nyXkXMEL1Vi/UX831Vi/UX82ut7Ry7tQ6gBmZ7elA11DHSJoAPBfYW+Zrx7VpTY0jajczs6GVE2qb\ngdmSZkmaRDLwY+OAYzYC70yfvxm4JSIibV+cjo6cBcwG7qhO6cWwbP4cGice2p3QOLGBZfPn5FSR\nmVn9Ktn9GBEHJF0MtAINwOqIuFfSFUBbRGwEvgp8Ix0Ispck+EiPu55kUMkB4H0e+Xio/lGOHv1o\nZlY5JTdUtaWlpSXa2tryLsPMzGqEpC0R0VLqOM8oYmZmheFQMzOzwnComZlZYdTke2qSuoEHKzzN\nFODhKpRzONVbza537NVbzfVWL9RfzeO13pMjYmqpg2oy1KpBUls5byrWknqr2fWOvXqrud7qhfqr\n2fUOz92PZmZWGA41MzMrjCKH2qq8CxiFeqvZ9Y69equ53uqF+qvZ9Q6jsO+pmZnZ+FPkOzUzMxtn\nHGpmZlYYhQw1SQsktUvaIWl53vWUImm1pD2S7sm7lnJImiHpB5Luk3SvpA/mXdNwJB0p6Q5JP0/r\n/UTeNZVDUoOkrZK+l3ct5ZD0gKRtku6SVPOTt0pqkrRO0i/Sn+XDtzzzKEiak35v+x+PS/pQ3nUN\nR9KH039z90haI+nIMb9m0d5Tk9QA/BI4h2Q9t83AkojYnmthw5D0Z8CTwNcj4qV511OKpBOBEyPi\nTknHAFuARbX6PZYkYHJEPClpIvBj4IMRcXvOpQ1L0keAFuDYiHhD3vWUIukBoCUi6uKDwZK+Bvwo\nIq5Ol9U6KiJ68q6rHOnvuU7glRFR6UQVY0JSM8m/tVMiojddsWVTRFwzltct4p3aPGBHROyMiH3A\nWmBhzjUNKyJ+SLJkT12IiIci4s70+RPAfUDNrpUTiSfTzYnpo6b/mpM0HXg9cHXetRSRpGOBPyNZ\nNouI2FcvgZY6C/h1rQZaxgSgMV08+igOwyLRRQy1ZmB3ZruDGv6FW+8kzQTmAj/Lt5LhpV15dwF7\ngJsjoqbrBf4RuAQ4mHchIxDATZK2SFqadzElvBDoBv417eK9WtLkvIsagcXAmryLGE5EdAKfAXYB\nDwGPRcRNY33dIoaaBmmr6b/K65Wko4HvAB+KiMfzrmc4EdEXEacB04F5kmq2m1fSG4A9EbEl71pG\n6IyIeAVwHvC+tFu9Vk0AXgF8MSLmAr8Dav79d4C0q/R84Nt51zIcSceR9JLNAqYBkyW9fayvW8RQ\n6wBmZLancxhueceb9L2p7wDXRsT6vOspV9rFdCuwIOdShnMGcH76HtVa4LWSvplvSaVFRFf63z3A\nDSRvBdSqDqAjc8e+jiTk6sF5wJ0R8du8CynhbOD+iOiOiP3AeuDVY33RIobaZmC2pFnpXzSLgY05\n11Qo6cCLrwL3RcRn866nFElTJTWlzxtJ/rH9It+qhhYRKyJiekTMJPn5vSUixvwv3EpImpwOGiLt\nxjsXqNnRvBHxG2C3pDlp01lATQ50GsQSarzrMbULeJWko9LfGWeRvP8+piaM9QUOt4g4IOlioBVo\nAFZHxL05lzUsSWuAM4EpkjqAyyLiq/lWNawzgHcA29L3qQA+HhGbcqxpOCcCX0tHjD0HuD4i6mKY\nfB15PnBD8ruLCcC3IuLf8y2ppPcD16Z//O4E3p1zPSVJOopkZPdFeddSSkT8TNI64E7gALCVwzBl\nVuGG9JuZ2fhVxO5HMzMbpxxqZmZWGA41MzMrDIeaFVo6H+HZeddRSqV1SnpS0gurXNNJ6Xkbqnle\ns7HkULNxQdJLJbVKelhS4UZHRcTREbGzknMMDNaI2JWet6/yCs0OD4eajRf7geuBC0fz4nTuuppT\nq3WZ5cWhZuNCRLSnn/0r6zOLki5PlyX5pqTHgXdJeo6k5ZJ+LekRSddLOj7zmr+Q9GC67/9k73wk\nXSPpbzPHnpl+JnGwa8+TdJukHkkPSfpC+lmq/v0h6X2SfgX8KtP2IknT0i7D/sdT/Xemkv5A0i1p\nfQ9LujbzofRvACcB301fd4mkmel5J6THTJO0UdJeJcs6/eWA79f1kr4u6Qkly420lPm/x6xqHGpm\nQ1tIMn1SE3At8AFgEfAakrnsHgWuApB0CvAvwNtIPuz9XEY/kXYf8GFgCnA6yUwM/3vAMYuAVwKn\nZBsjoivtMjw6Io4mma5qbbpbwJVp7S8hmU7u8vR17yCZAeKN6Ws/PUhda0iml5oGvBn4e0lnZfaf\nn16riWQWny+M+Cs3q5BDzWxot0XEhog4GBG9JLM4/HVEdETE0ySB8Ob0TubNwHcj4sfpkkeXMsqJ\ntCNiS0TcHhEHIuIB4MskQZp1ZUTsTesalKS/Av4Q+F/peXdExM0R8XREdAOfHeS8Q51rBvCnwF9F\nxH9FxF0ky+K8I3PYjyNiU/oe3DeAl5f1BZtVkfvjbdyT9DaS4IBk0cjz0ue7Bxx6MslUUNnlYPpI\npoialj0+Ip6S9Mgo63kxSeC0kKxBNYFkIdasgbUNPMd5wAdJFpHsTdueB3we+G/AMSR/1D5aZlnT\ngL3p+nn9Hkxr7PebzPOngCMlTYiIA2Vew6xivlOzcS8irs102Z2X3TXg0N3AeRHRlHkcma4b9RDJ\nihDAMxMnn5B57e9IAqrfC4Yp6YskEy7PjohjgY/z7CWVhrwLTCfp/RrwlojIht+V6etelp737QPO\nO9ydZRdwfP+kxamTSFZfNqsZDjUbF5Q4EpiUbh8p6YgRnuZLwN9JOjk9x1RJ/auqrwPeKOnV6aCO\nT3BoYNwFvE7S8ZJeAHxomOscAzwOPCnpD4H3llugkhWd/w34m4j48SDnfRLokdQMLBuw/7cki2c+\nSxqOPwWuTL93LyMZSXptubWZHQ4ONRsvTgZ6+f3ox16gfYTn+CeSARA3SXoCuJ1ksAbpShDvJxko\n8RDwBMkq20+nr/0G8HPgAeAm4LphrvMx4K3pOb5S4tiBXgHMAT6bHQWZ7vtEuv8x4EaS9a2yrgT+\nJh11+bFBzr0EmEly13YDyWoSN4+gNrMx51n6zcaAklXBe0i6EO/Pux6z8cJ3amZVIumNShZEnAx8\nBthGcmdmZoeJQ82sehaSdM11AbOBxeGuELPDyt2PZmZWGL5TMzOzwqjJD19PmTIlZs6cmXcZZmZW\nI7Zs2fJwREwtdVxNhtrMmTNpa2vLuwwzM6sRkh4s5zh3P5qZWWE41MzMrDAcamZmVhgVhZqkBZLa\n0wUDlw+y/yRJP5C0VdLdkl5XyfXMrDZt2NrJGZ+6hVnLb+SMT93Chq2e59jyMepQk9RAskDieSQL\nFS5JF0rM+hvg+oiYCywmWUTRzApkw9ZOVqzfRmdPLwF09vSyYv02B5vlopI7tXnAjojYmS6KuJZk\nRoWsAI5Nnz+XZKYFMyuQla3t9O7vO6Std38fK1tHOl+0WeUqGdLfzKELFXaQzliecTnJjObvByYD\nZ1dwPTOrQV09gy++PVS72Viq5E5t4KKF8OxFBpcA10TEdOB1wDckDXpNSUsltUlq6+7urqAsMzuc\npjU1jqjdbCxVEmodwIzM9nSe3b14IXA9QETcBhwJTBnsZBGxKiJaIqJl6tSSHxo3sxqxbP4cGic2\nHNLWOLGBZfPn5FSRjWeVhNpmYLakWelKv4tJFlDM2gWcBSDpJSSh5tswswJZNLeZK990KpMakl8n\nzU2NXPmmU1k0tznnymw8GvV7ahFxQNLFQCvQAKyOiHslXQG0RcRG4KPAVyR9mKRr8l1eisOseBbN\nbWbNHbsAuO6i03OuxsaziuZ+jIhNwKYBbZdmnm8HzqjkGmZmZuXyjCJmZlYYDjUzMysMh5qZmRWG\nQ83MzArDoWZmZoXhUDMzs8JwqJmZWWE41MzMrDAcamZmVhgONTMzKwyHmpmZFYZDzczMCsOhZmZm\nheFQMzOzwnComZlZYTjUzMysMBxqZmZWGA41MzMrDIeamZkVhkPNzMwKw6FmZmaF4VAzM7PCcKiZ\nmVlhONTMzKwwHGpmZlYYDjUzMysMh5qZmRWGQ83MzArDoWZmZoXhUDMzs8JwqJmZWWFUFGqSFkhq\nl7RD0vIhjnmLpO2S7pX0rUquZ2ZmNpwJo32hpAbgKuAcoAPYLGljRGzPHDMbWAGcERGPSnpepQWb\nmZkNpZI7tXnAjojYGRH7gLV0k9lfAAANbklEQVTAwgHH/CVwVUQ8ChAReyq4npmZ2bAqCbVmYHdm\nuyNty3ox8GJJP5F0u6QFQ51M0lJJbZLauru7KyjLzMzGq0pCTYO0xYDtCcBs4ExgCXC1pKbBThYR\nqyKiJSJapk6dWkFZZmY2XlUSah3AjMz2dKBrkGP+LSL2R8T9QDtJyJmZmVVdJaG2GZgtaZakScBi\nYOOAYzYAfw4gaQpJd+TOCq5pZmY2pFGPfoyIA5IuBlqBBmB1RNwr6QqgLSI2pvvOlbQd6AOWRcQj\n1Si8SDZs7WRlaztdPb1Ma2pk2fw5LJo78O1JMzMrZdShBhARm4BNA9ouzTwP4CPpwwaxYWsnK9Zv\no3d/HwCdPb2sWL8NwMFmZjZCnlEkZytb258JtH69+/tY2dqeU0VmZvXLoZazrp7eEbWbmdnQHGo5\nm9bUOKJ2MzMbmkMtZ8vmz6FxYsMhbY0TG1g2f05OFZmZ1a+KBopY5foHg1yy7m729R2k2aMfzcxG\nzaFWAxbNbWbNHbsAuO6i03Ouxsysfrn70czMCsOhZmZmheFQMzOzwnComZlZYTjUzMysMBxqZmZW\nGA41MzMrDIeamZkVhkPNzMwKw6FmZmaF4VAzM7PCcKiZmVlhONTMzKwwHGpmZlYYDjUzMysMr6dm\nI7ZhaycrW9vp6ullmhc1NbMa4lCzEdmwtZMV67fRu78PgM6eXlas3wbgYDOz3Ln70UZkZWv7M4HW\nr3d/Hytb23OqyMzs9xxqNiJdPb0jajczO5wcajYi05oaR9RuZnY4OdRsRJbNn0PjxIZD2honNrBs\n/pycKjIz+z0PFLER6R8Mcsm6u9nXd5Bmj340sxriULMRWzS3mTV37ALguotOz7kaM7Pfc/ejmZkV\nRkWhJmmBpHZJOyQtH+a4N0sKSS2VXM/MzGw4ow41SQ3AVcB5wCnAEkmnDHLcMcAHgJ+N9lpmZmbl\nqORObR6wIyJ2RsQ+YC2wcJDjPgl8GvivCq5lZmZWUiWh1gzszmx3pG3PkDQXmBER3yt1MklLJbVJ\nauvu7q6gLDMzG68qCTUN0hbP7JSeA3wO+Gg5J4uIVRHREhEtU6dOraAsMzMbryoJtQ5gRmZ7OtCV\n2T4GeClwq6QHgFcBGz1YxMzMxkolobYZmC1plqRJwGJgY//OiHgsIqZExMyImAncDpwfEW0VVWxm\nZjaEUYdaRBwALgZagfuA6yPiXklXSDq/WgWamZmVq6IZRSJiE7BpQNulQxx7ZiXXMjMzK8UzipiZ\nWWE41MzMrDAcamZmVhgONTMzKwyHmpmZFYZDzczMCsOhZmZmheFQMzOzwnComZlZYTjUzMysMBxq\nZmZWGA41MzMrDIeamZkVhkPNzMwKw6FmZmaF4VAzM7PCcKiZmVlhONTMzKwwHGpmZlYYDjUzMysM\nh5qZmRWGQ83MzArDoWZmZoXhUDMzs8JwqJmZWWE41MzMrDAcamZmVhgONTMzKwyHmpmZFYZDzczM\nCqOiUJO0QFK7pB2Slg+y/yOStku6W9J/Sjq5kuuZmZkNZ8JoXyipAbgKOAfoADZL2hgR2zOHbQVa\nIuIpSe8FPg1cUEnBpWzY2snK1na6enqZ1tTIsvlzWDS3eSwvaWZmNaKSO7V5wI6I2BkR+4C1wMLs\nARHxg4h4Kt28HZhewfVK2rC1kxXrt9HZ00sAnT29rFi/jQ1bO8fysmZmViMqCbVmYHdmuyNtG8qF\nwPeH2ilpqaQ2SW3d3d2jKmhlazu9+/sOaevd38fK1vZRnc/MzOpLJaGmQdpi0AOltwMtwMqhThYR\nqyKiJSJapk6dOqqCunp6R9RuZmbFUkmodQAzMtvTga6BB0k6G/hr4PyIeLqC65U0ralxRO1mZlYs\nlYTaZmC2pFmSJgGLgY3ZAyTNBb5MEmh7KrhWWZbNn0PjxIZD2honNrBs/pyxvrSZmdWAUY9+jIgD\nki4GWoEGYHVE3CvpCqAtIjaSdDceDXxbEsCuiDi/CnUPqn+U4yXr7mZf30GaPfrRzGxcGXWoAUTE\nJmDTgLZLM8/PruT8o7FobjNr7tgFwHUXnX64L29mdcIf/ymmikLNzKwe9X/8p3+0dP/HfwAHW53z\nNFlmNu744z/F5VAzs3HHH/8pLoeamY07/vhPcTnUzGzc8cd/issDRcxs3PHHf4rLoWZm45I//lNM\n7n40M7PCcKiZmVlhONTMzKwwHGpmZlYYDjUzMysMh5qZmRWGQ83MzArDoWZmZoXhD1+bmdUBr/9W\nHoeamVmN8/pv5XP3o5lZjfP6b+VzqJmZ1Tiv/1Y+h5qZWY3z+m/lc6iZmdU4r/9WPg8UMTOrcV7/\nrXwONTOzOuD138rj7kczMysMh5qZmRWGQ83MzArDoWZmZoXhUDMzs8JwqJmZWWE41MzMrDAq+pya\npAXAPwENwNUR8akB+48Avg78MfAIcEFEPFDJNc1Gqh6X7KjHms2y8voZHnWoSWoArgLOATqAzZI2\nRsT2zGEXAo9GxIskLQb+AbigkoLNRqIel+yox5rNsvL8GVZEjO6F0unA5RExP91eARARV2aOaU2P\nuU3SBOA3wNQocdGWlpZoa2sbVV0A/7rk/bygezennHjsqM9xuG1/6HGAuqm5XurduquHpw/0Pav9\niAkNzD2pKYeKSqvHmqF+fiay6q3meqk3+zO887nNfPllCwFobmrkJ8tfO6pzStoSES2ljquk+7EZ\n2J3Z7gBeOdQxEXFA0mPACcDDA08maSmwFOCkk06qoCw4fvIRHPVYQ+kDa8hRk1zvWBgsHIZrrwX1\nWDPUz89EVr3VXC/1DvWzejiWyqkk1DRI28A7sHKOSRojVgGrILlTq6AuFl79mUpenouT8y5ghOql\n3rd+6hY6B/mH1NzUyAWj/ItxrNVjzVA/PxNZ9VZzvdQ71M/w4Vgqp5LRjx3AjMz2dKBrqGPS7sfn\nAnsruKbZiNTjkh31WLNZVp4/w5XcqW0GZkuaBXQCi4G3DjhmI/BO4DbgzcAtpd5PM6um/jel62kk\nYT3WbJaV58/wqAeKAEh6HfCPJEP6V0fE30m6AmiLiI2SjgS+AcwluUNbHBE7S5230oEiZmZWLIdj\noAgRsQnYNKDt0szz/wL+ZyXXMDMzK5dnFDEzs8JwqJmZWWFU9J7aWJHUDTxY4WmmMMjn4WpcvdXs\nesdevdVcb/VC/dU8Xus9OSKmljqoJkOtGiS1lfOmYi2pt5pd79irt5rrrV6ov5pd7/Dc/WhmZoXh\nUDMzs8IocqityruAUai3ml3v2Ku3muutXqi/ml3vMAr7npqZmY0/Rb5TMzOzccahZmZmhVHIUJO0\nQFK7pB2SluddTymSVkvaI+mevGsph6QZkn4g6T5J90r6YN41DUfSkZLukPTztN5P5F1TOSQ1SNoq\n6Xt511IOSQ9I2ibpLkk1P3mrpCZJ6yT9Iv1ZPj3vmoYjaU76ve1/PC7pQ3nXNRxJH07/zd0jaU06\nH/DYXrNo76lJagB+CZxDsvTNZmBJRGzPtbBhSPoz4Eng6xHx0rzrKUXSicCJEXGnpGOALcCiWv0e\nSxIwOSKelDQR+DHwwYi4PefShiXpI0ALcGxEvCHvekqR9ADQEhF18cFgSV8DfhQRV0uaBBwVET15\n11WO9PdcJ/DKiKh0oooxIamZ5N/aKRHRK+l6YFNEXDOW1y3indo8YEdE7IyIfcBaYGHONQ0rIn5I\nHa0zFxEPRcSd6fMngPtIVjmvSZF4Mt2cmD5q+q85SdOB1wNX511LEUk6Fvgz4KsAEbGvXgItdRbw\n61oNtIwJQGO6nuZRPHvNzaorYqg1A7sz2x3U8C/ceidpJsnSQj/Lt5LhpV15dwF7gJsjoqbrJVnS\n6RLgYN6FjEAAN0naImlp3sWU8EKgG/jXtIv3akmT8y5qBBYDa/IuYjgR0Ql8BtgFPAQ8FhE3jfV1\nixhqGqStpv8qr1eSjga+A3woIh7Pu57hRERfRJxGskL7PEk1280r6Q3AnojYknctI3RGRLwCOA94\nX9qtXqsmAK8AvhgRc4HfATX//jtA2lV6PvDtvGsZjqTjSHrJZgHTgMmS3j7W1y1iqHUAMzLb0zkM\nt7zjTfre1HeAayNifd71lCvtYroVWJBzKcM5Azg/fY9qLfBaSd/Mt6TSIqIr/e8e4AaStwJqVQfQ\nkbljX0cScvXgPODOiPht3oWUcDZwf0R0R8R+YD3w6rG+aBFDbTMwW9Ks9C+axcDGnGsqlHTgxVeB\n+yLis3nXU4qkqZKa0ueNJP/YfpFvVUOLiBURMT0iZpL8/N4SEWP+F24lJE1OBw2RduOdC9TsaN6I\n+A2wW9KctOksoCYHOg1iCTXe9ZjaBbxK0lHp74yzSN5/H1MVrXxdiyLigKSLgVagAVgdEffmXNaw\nJK0BzgSmSOoALouIr+Zb1bDOAN4BbEvfpwL4eLoSei06EfhaOmLsOcD1EVEXw+TryPOBG5LfXUwA\nvhUR/55vSSW9H7g2/eN3J/DunOspSdJRJCO7L8q7llIi4meS1gF3AgeArRyGKbMKN6TfzMzGryJ2\nP5qZ2TjlUDMzs8JwqJmZWWE41MzMrDAcamZmVhgONTMzKwyHmpmZFcb/ByLhT2Tigp4ZAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b796518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logreg = linear_model.LogisticRegression(C=C_opt,penalty='l1')\n",
    "logreg.C= C_opt\n",
    "logreg.fit(Xs,y)\n",
    "yhat = logreg.predict(Xs)\n",
    "acc = np.mean(yhat == y)\n",
    "print('Accuracy on the training data is {0:f}'.format(acc))\n",
    "W_l1 = logreg.coef_\n",
    "\n",
    "plt.figure(figsize=(7,7))\n",
    "plt.subplot(2,1,1)\n",
    "plt.stem(W[0,:])\n",
    "plt.title('No regularization')\n",
    "plt.subplot(2,1,2)\n",
    "plt.stem(W_l1[0,:])\n",
    "plt.title('l1-regularization')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy for the training data without regularization was 0.9692. So with regularization, we were able to improve the accuracy slightly. Note that actual results vary each time you run, because the optimal C depends on the cross validation split, which is random. When the nubmer of samples is small relative to the number of features, more and consistant gain is expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}