deep_learning/08-dropout-and-strides-for-larger-models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "You've built a model to identify clothing types in the **MNIST for Fashion** dataset.  Now you will make your model bigger, specify larger stride lengths and apply dropout. These changes will make your model faster and more accurate.\n",
    "\n",
    "This is a last step in the **[Deep Learning Track](https://www.kaggle.com/learn/deep-learning)**.\n",
    "\n",
    "\n",
    "## Data Preparation\n",
    "**Run this cell of code.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow import keras\n",
    "\n",
    "# Set up code checking\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.deep_learning.exercise_8 import *\n",
    "print(\"Setup Complete\")\n",
    "\n",
    "img_rows, img_cols = 28, 28\n",
    "num_classes = 10\n",
    "\n",
    "def prep_data(raw):\n",
    "    y = raw[:, 0]\n",
    "    out_y = keras.utils.to_categorical(y, num_classes)\n",
    "    \n",
    "    x = raw[:,1:]\n",
    "    num_images = raw.shape[0]\n",
    "    out_x = x.reshape(num_images, img_rows, img_cols, 1)\n",
    "    out_x = out_x / 255\n",
    "    return out_x, out_y\n",
    "\n",
    "fashion_file = \"../input/fashionmnist/fashion-mnist_train.csv\"\n",
    "fashion_data = np.loadtxt(fashion_file, skiprows=1, delimiter=',')\n",
    "x, y = prep_data(fashion_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1) Increasing Stride Size in A Layer\n",
    "\n",
    "Below is a model without strides (or more accurately, with a stride length of 1)\n",
    "\n",
    "Run it. Notice it's accuracy and how long it takes per epoch. Then you will change the stride length in one of the layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 48000 samples, validate on 12000 samples\n",
      "Epoch 1/3\n",
      "48000/48000 [==============================] - 16s 328us/sample - loss: 0.3974 - accuracy: 0.8564 - val_loss: 0.3083 - val_accuracy: 0.8879\n",
      "Epoch 2/3\n",
      "48000/48000 [==============================] - 11s 236us/sample - loss: 0.2519 - accuracy: 0.9064 - val_loss: 0.2593 - val_accuracy: 0.9080\n",
      "Epoch 3/3\n",
      "48000/48000 [==============================] - 12s 248us/sample - loss: 0.1879 - accuracy: 0.9294 - val_loss: 0.2851 - val_accuracy: 0.9028\n",
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 16)        160       \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 24, 24, 16)        2320      \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 128)               1179776   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 1,183,546\n",
      "Trainable params: 1,183,546\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Flatten, Conv2D, Dropout\n",
    "\n",
    "batch_size = 16\n",
    "\n",
    "fashion_model = Sequential()\n",
    "fashion_model.add(Conv2D(16, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=(img_rows, img_cols, 1)))\n",
    "fashion_model.add(Conv2D(16, (3, 3), activation='relu'))\n",
    "fashion_model.add(Flatten())\n",
    "fashion_model.add(Dense(128, activation='relu'))\n",
    "fashion_model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "fashion_model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "fashion_model.fit(x, y,\n",
    "          batch_size=batch_size,\n",
    "          epochs=3,\n",
    "          validation_split = 0.2)\n",
    "fashion_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have the same code in the cell below, but the model is now called `fashion_model_1`.  Change the specification of `fashion_model_1` so the second convolutional layer has a stride length of 2.\n",
    "\n",
    "Run the cell after you have done that. How does the speed and accuracy change compared to the first model you ran above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 48000 samples, validate on 12000 samples\n",
      "Epoch 1/3\n",
      "48000/48000 [==============================] - 12s 248us/sample - loss: 0.4291 - accuracy: 0.8436 - val_loss: 0.3379 - val_accuracy: 0.8770\n",
      "Epoch 2/3\n",
      "48000/48000 [==============================] - 11s 234us/sample - loss: 0.2956 - accuracy: 0.8910 - val_loss: 0.2850 - val_accuracy: 0.8983\n",
      "Epoch 3/3\n",
      "48000/48000 [==============================] - 11s 233us/sample - loss: 0.2403 - accuracy: 0.9097 - val_loss: 0.2720 - val_accuracy: 0.9048\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 1.0, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_AddStrides\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fashion_model_1 = Sequential()\n",
    "fashion_model_1.add(Conv2D(16, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=(img_rows, img_cols, 1)))\n",
    "fashion_model_1.add(Conv2D(16, (3, 3), activation='relu', strides=2))\n",
    "fashion_model_1.add(Flatten())\n",
    "fashion_model_1.add(Dense(128, activation='relu'))\n",
    "fashion_model_1.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "fashion_model_1.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "fashion_model_1.fit(x, y,\n",
    "          batch_size=batch_size,\n",
    "          epochs=3,\n",
    "          validation_split = 0.2)\n",
    "q_1.check()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the solution, uncomment and run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should notice that your model training ran about twice as fast, but the accuracy change was trivial.  \n",
    "\n",
    "In addition to being faster to train, this model is also faster at making predictions. This is very important in many scenarios. In practice, you'll need to decide whether that type of speed is important in the applications where you eventually apply deep learning models.\n",
    "\n",
    "You could experiment with more layers or more convolutions in each layer. With some fine-tuning, you can build a model that is both faster and more accurate than the original model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 48000 samples, validate on 12000 samples\n",
      "Epoch 1/7\n",
      "48000/48000 [==============================] - 6s 128us/sample - loss: 0.5964 - accuracy: 0.7825 - val_loss: 0.4089 - val_accuracy: 0.8514\n",
      "Epoch 2/7\n",
      "48000/48000 [==============================] - 5s 105us/sample - loss: 0.3606 - accuracy: 0.8665 - val_loss: 0.3090 - val_accuracy: 0.8894\n",
      "Epoch 3/7\n",
      "48000/48000 [==============================] - 5s 106us/sample - loss: 0.2984 - accuracy: 0.8896 - val_loss: 0.2761 - val_accuracy: 0.9020\n",
      "Epoch 4/7\n",
      "48000/48000 [==============================] - 5s 108us/sample - loss: 0.2598 - accuracy: 0.9030 - val_loss: 0.2623 - val_accuracy: 0.9053\n",
      "Epoch 5/7\n",
      "48000/48000 [==============================] - 5s 114us/sample - loss: 0.2302 - accuracy: 0.9149 - val_loss: 0.2419 - val_accuracy: 0.9137\n",
      "Epoch 6/7\n",
      "48000/48000 [==============================] - 5s 108us/sample - loss: 0.2055 - accuracy: 0.9221 - val_loss: 0.2498 - val_accuracy: 0.9115\n",
      "Epoch 7/7\n",
      "48000/48000 [==============================] - 5s 110us/sample - loss: 0.1796 - accuracy: 0.9326 - val_loss: 0.2430 - val_accuracy: 0.9166\n",
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_4 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 26, 26, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 24, 24, 32)        9248      \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 24, 24, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 22, 22, 48)        13872     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 22, 22, 48)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 20, 20, 64)        27712     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 20, 20, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 18, 18, 64)        36928     \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 18, 18, 64)        0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 20736)             0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 128)               2654336   \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 2,743,706\n",
      "Trainable params: 2,743,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 strides=1,\n",
    "                 activation='relu',\n",
    "                 input_shape=(img_rows, img_cols, 1)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Conv2D(32, kernel_size=(3, 3), strides=1, activation='relu'))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Conv2D(48, kernel_size=(3, 3), strides=1, activation='relu'))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Conv2D(64, kernel_size=(3, 3), strides=1, activation='relu'))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Conv2D(64, kernel_size=(3, 3), strides=1, activation='relu'))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "model.fit(x, y,\n",
    "          batch_size=128,\n",
    "          epochs=7,\n",
    "          validation_split = 0.2)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congrats\n",
    "You've finished the Deep Learning course.  You have the tools to create and tune computer vision models.  \n",
    "\n",
    "If you feel like playing more with this dataset, you can open up a new code cell to experiment with different models (adding dropout, adding layers, etc.)  Or pick a new project and try out your skills.  \n",
    "\n",
    "A few fun datasets you might try include:\n",
    "- [Written letter recognition](https://www.kaggle.com/olgabelitskaya/classification-of-handwritten-letters)\n",
    "- [Flower Identification](https://www.kaggle.com/alxmamaev/flowers-recognition)\n",
    "- [Cats vs Dogs](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition)\n",
    "- [10 Monkeys](https://www.kaggle.com/slothkong/10-monkey-species)\n",
    "- [Predict Bone Age from X-Rays](https://www.kaggle.com/kmader/rsna-bone-age)\n",
    "\n",
    "You have learned a lot, and you'll learn it as you practice. Have fun with it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Deep Learning Home Page](https://www.kaggle.com/learn/deep-learning)**\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum) to chat with other Learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}