ml/slp-mlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Overview\n",
    "\n",
    "We implemented a [simple perception](perceptron.ipynb) that sums up all the inputs, weights, and bias to produce a scalar output. In this notebook, we will use a more sophisticated perceptron called single-layer perceptron (SLP) from the `sklearn` library.\n",
    "\n",
    "The SLP incorporates gradient descent to find the coefficients (components) of the function (the perceptron) perceptron, to minimize the cost function (objective). We then use the information for the error backpropagation algorithm in the perceptron.\n",
    "\n",
    "Finally we will apply an even more complex model multi-layer perceptron (MLP) that is specifically designed for classification using the `MLPClassifier` in the `sklearn` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will use the Iris sample dataset for this exercise."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "outputs": [],
   "source": [
    "iris = load_iris()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Preprocessing\n",
    "\n",
    "### Split the data into training and test\n",
    "\n",
    "First break the data into features X and labels y before splitting them to training and test datasets."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "outputs": [
    {
     "data": {
      "text/plain": "((112, 4), (38, 4))"
     },
     "execution_count": 233,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "\n",
    "# Split the data into training and test\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)\n",
    "X_train.shape, X_test.shape"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Standardization\n",
    "\n",
    "It's a good practice to standardize so that our feature data has a good Gaussian distribution ie. 0 mean and unit variance. This keeps the features consist and avoid unexpected behaviors as ML methods eg. logistic regression doesn't make assumption about data having a Gaussian distribution."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "sc.fit(X_train)\n",
    "X_train_std = sc.transform(X_train)\n",
    "X_test_std = sc.transform(X_test)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Train the SLP with the following parameters:\n",
    "\n",
    "* Learning rate `eta0` = 0.1\n",
    "* Epoch `max_iter` = 2000"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Perceptron\n",
    "\n",
    "epochs = 2000\n",
    "slp = Perceptron(random_state=1, eta0=0.1, max_iter=epochs)\n",
    "slp.fit(X_train_std, y_train)\n",
    "y_slp_predict = slp.predict(X_test_std)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Finally let's evaluate how accuracy our model:\n",
    "\n",
    "$$\n",
    "accuracy = 1 - \\frac{wrong\\ predications}{total\\ observations}\n",
    "$$\n",
    "\n",
    "We can use the convenient function `accuracy_score` to calculate accuracy."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "outputs": [
    {
     "data": {
      "text/plain": "0.868421052631579"
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_score(y_test, y_slp_predict)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now use a MLP classifier from the `sklearn` library with the following parameters:\n",
    "\n",
    "* Hidden Layers and neurons configuration `hidden_layer_sizes` = (5, 5, 5)\n",
    "* Epoch `max_iter` = 2000"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SLP Accuracy: 86.842%\n",
      "MLP Accuracy: 94.737%\n",
      "SLP Accuracy: 86.842%\n",
      "MLP Accuracy: 94.737%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "hidden_layers = (5, 5, 5)\n",
    "mlp = MLPClassifier(hidden_layer_sizes=hidden_layers, max_iter=epochs)\n",
    "mlp.fit(X_train_std, y_train)\n",
    "y_mlp_predict = mlp.predict(X_test_std)\n",
    "\n",
    "# Accuracy\n",
    "print('SLP Accuracy: %.3f%%' % (sum(y_slp_predict == y_test) / len(y_test) * 100))\n",
    "print('MLP Accuracy: %.3f%%' % (sum(y_mlp_predict == y_test) / len(y_test) * 100))\n",
    "\n",
    "# Alternatively we can use the convenience function `accuracy_score`.\n",
    "print('SLP Accuracy: %.3f%%' % (accuracy_score(y_test, y_slp_predict) * 100))\n",
    "print('MLP Accuracy: %.3f%%' % (accuracy_score(y_test, y_mlp_predict) * 100))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Reference\n",
    "\n",
    "* [How to Normalize and Standardize Your Machine Learning Data](https://machinelearningmastery.com/normalize-standardize-machine-learning-data-weka/)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}