Introduction to PyTorch/Part 8 - Transfer Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L-iTUbNQEUTT"
   },
   "source": [
    "# Download Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s42XyPHG3KUK"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir('/content/drive/My Drive/Colab Notebooks')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "aXQWFX4J3NwS",
    "outputId": "ab1a3f4a-b90a-4071-9dbf-c382e456a7f7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-02-23 07:04:59--  https://s3.amazonaws.com/content.udacity-data.com/nd089/Cat_Dog_data.zip\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.207.197\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.207.197|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 580495262 (554M) [application/zip]\n",
      "Saving to: ‘Cat_Dog_data.zip.1’\n",
      "\n",
      "Cat_Dog_data.zip.1  100%[===================>] 553.60M  15.9MB/s    in 40s     \n",
      "\n",
      "2020-02-23 07:05:40 (13.9 MB/s) - ‘Cat_Dog_data.zip.1’ saved [580495262/580495262]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://s3.amazonaws.com/content.udacity-data.com/nd089/Cat_Dog_data.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ujEZZgDD3TiE"
   },
   "outputs": [],
   "source": [
    "!python -m zipfile -e Cat_Dog_data.zip ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "TmVzF1-O3T9a",
    "outputId": "fc8c660d-552f-438c-9799-e800d7251cbe"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25003"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([len(files) for r, d, files in os.walk(\"Cat_Dog_data\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Sdesbdfm2j1f"
   },
   "source": [
    "# Transfer Learning\n",
    "\n",
    "In this notebook, you'll learn how to use pre-trained networks to solved challenging problems in computer vision. Specifically, you'll use networks trained on [ImageNet](http://www.image-net.org/) [available from torchvision](http://pytorch.org/docs/0.3.0/torchvision/models.html). \n",
    "\n",
    "ImageNet is a massive dataset with over 1 million labeled images in 1000 categories. It's used to train deep neural networks using an architecture called convolutional layers. I'm not going to get into the details of convolutional networks here, but if you want to learn more about them, please [watch this](https://www.youtube.com/watch?v=2-Ol7ZB0MmU).\n",
    "\n",
    "Once trained, these models work astonishingly well as feature detectors for images they weren't trained on. Using a pre-trained network on images not in the training set is called transfer learning. Here we'll use transfer learning to train a network that can classify our cat and dog photos with near perfect accuracy.\n",
    "\n",
    "With `torchvision.models` you can download these pre-trained networks and use them in your applications. We'll include `models` in our imports now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kFt-twFq2j1i"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms, models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ANeJ5MIs2j1n"
   },
   "source": [
    "Most of the pretrained models require the input to be 224x224 images. Also, we'll need to match the normalization used when the models were trained. Each color channel was normalized separately, the means are `[0.485, 0.456, 0.406]` and the standard deviations are `[0.229, 0.224, 0.225]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t0cACtxF2j1o"
   },
   "outputs": [],
   "source": [
    "data_dir = 'Cat_Dog_data'\n",
    "\n",
    "# TODO: Define transforms for the training data and testing data\n",
    "train_transforms = transforms.Compose([transforms.RandomRotation(30),\n",
    "                                       transforms.RandomResizedCrop(224),\n",
    "                                       transforms.RandomHorizontalFlip(),\n",
    "                                       transforms.ToTensor(),\n",
    "                                       transforms.Normalize([0.485, 0.456, 0.406], \n",
    "                                                            [0.229, 0.224, 0.225])\n",
    "                                      ])\n",
    "\n",
    "test_transforms = transforms.Compose([transforms.Resize(255),\n",
    "                                      transforms.CenterCrop(224),\n",
    "                                      transforms.ToTensor(),\n",
    "                                      transforms.Normalize([0.485, 0.456, 0.406], \n",
    "                                                            [0.229, 0.224, 0.225])\n",
    "                                      ])\n",
    "\n",
    "# Pass transforms in here, then run the next cell to see how the transforms look\n",
    "train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)\n",
    "test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)\n",
    "\n",
    "trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)\n",
    "testloader = torch.utils.data.DataLoader(test_data, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xr-6KUlN2j1r"
   },
   "source": [
    "We can load in a model such as [DenseNet](http://pytorch.org/docs/0.3.0/torchvision/models.html#id5). Let's print out the model architecture so we can see what's going on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "c1f9c9aea0b04355a9575dd04117f141",
      "5b67c54257ab495791c12288c379919d",
      "5e83efe919b84b3a81999d61d05bf11d",
      "91263dcfdf8c4e1cbf541bd0d6d65270",
      "e771ca0ba45448889090688da9f771a5",
      "95783c39cd3047cb8c3ca442442dfbcc",
      "861840fe1dd44ecb994c92a129bbcb04",
      "af33422af70e4abbbda6a1a0c0ba53c6"
     ]
    },
    "colab_type": "code",
    "id": "NOFJVDjh2j1s",
    "outputId": "5be83707-1054-4082-a0ad-e7e879d3bcb7",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: \"https://download.pytorch.org/models/densenet121-a639ec97.pth\" to /root/.cache/torch/checkpoints/densenet121-a639ec97.pth\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1f9c9aea0b04355a9575dd04117f141",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=32342954), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DenseNet(\n",
       "  (features): Sequential(\n",
       "    (conv0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
       "    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (relu0): ReLU(inplace=True)\n",
       "    (pool0): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
       "    (denseblock1): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(96, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (transition1): _Transition(\n",
       "      (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
       "    )\n",
       "    (denseblock2): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer7): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer8): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer9): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer10): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer11): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer12): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (transition2): _Transition(\n",
       "      (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
       "    )\n",
       "    (denseblock3): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer7): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer8): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer9): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer10): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer11): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer12): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer13): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer14): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer15): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer16): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer17): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer18): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer19): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer20): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer21): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer22): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer23): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer24): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (transition3): _Transition(\n",
       "      (norm): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
       "    )\n",
       "    (denseblock4): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer7): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer8): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer9): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer10): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer11): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer12): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer13): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer14): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer15): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer16): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (norm5): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  )\n",
       "  (classifier): Linear(in_features=1024, out_features=1000, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models.densenet121(pretrained=True)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qgRoDeTJ2j10"
   },
   "source": [
    "This model is built out of two main parts, the features and the classifier. The features part is a stack of convolutional layers and overall works as a feature detector that can be fed into a classifier. The classifier part is a single fully-connected layer `(classifier): Linear(in_features=1024, out_features=1000)`. This layer was trained on the ImageNet dataset, so it won't work for our specific problem. That means we need to replace the classifier, but the features will work perfectly on their own. In general, I think about pre-trained networks as amazingly good feature detectors that can be used as the input for simple feed-forward classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "miP788Wt2j11"
   },
   "outputs": [],
   "source": [
    "# Freeze parameters so we don't backprop through them\n",
    "for param in model.parameters():\n",
    "    param.requires_grad = False\n",
    "\n",
    "from collections import OrderedDict\n",
    "classifier = nn.Sequential(OrderedDict([\n",
    "                          ('fc1', nn.Linear(1024, 500)),\n",
    "                          ('relu', nn.ReLU()),\n",
    "                          ('fc2', nn.Linear(500, 2)),\n",
    "                          ('output', nn.LogSoftmax(dim=1))\n",
    "                          ]))\n",
    "    \n",
    "model.classifier = classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9ogDN2Zq2j15"
   },
   "source": [
    "With our model built, we need to train the classifier. However, now we're using a **really deep** neural network. If you try to train this on a CPU like normal, it will take a long, long time. Instead, we're going to use the GPU to do the calculations. The linear algebra computations are done in parallel on the GPU leading to 100x increased training speeds. It's also possible to train on multiple GPUs, further decreasing training time.\n",
    "\n",
    "PyTorch, along with pretty much every other deep learning framework, uses [CUDA](https://developer.nvidia.com/cuda-zone) to efficiently compute the forward and backwards passes on the GPU. In PyTorch, you move your model parameters and other tensors to the GPU memory using `model.to('cuda')`. You can move them back from the GPU with `model.to('cpu')` which you'll commonly do when you need to operate on the network output outside of PyTorch. As a demonstration of the increased speed, I'll compare how long it takes to perform a forward and backward pass with and without a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OoUxI_732j16"
   },
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "cN4J7X3G2j1_",
    "outputId": "2c31b47a-7a17-43d5-d7dc-de225c4c415b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device = cpu; Time per batch: 4.676 seconds\n",
      "Device = cuda; Time per batch: 0.012 seconds\n"
     ]
    }
   ],
   "source": [
    "for device in ['cpu', 'cuda']:\n",
    "\n",
    "    criterion = nn.NLLLoss()\n",
    "    # Only train the classifier parameters, feature parameters are frozen\n",
    "    optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)\n",
    "\n",
    "    model.to(device)\n",
    "\n",
    "    for ii, (inputs, labels) in enumerate(trainloader):\n",
    "\n",
    "        # Move input and label tensors to the GPU\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "\n",
    "        start = time.time()\n",
    "\n",
    "        outputs = model.forward(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        if ii==3:\n",
    "            break\n",
    "        \n",
    "    print(f\"Device = {device}; Time per batch: {(time.time() - start)/3:.3f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RfZijj052j2D"
   },
   "source": [
    "You can write device agnostic code which will automatically use CUDA if it's enabled like so:\n",
    "```python\n",
    "# at beginning of the script\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "...\n",
    "\n",
    "# then whenever you get a new Tensor or Module\n",
    "# this won't copy if they are already on the desired device\n",
    "input = data.to(device)\n",
    "model = MyModule(...).to(device)\n",
    "```\n",
    "\n",
    "From here, I'll let you finish training the model. The process is the same as before except now your model is much more powerful. You should get better than 95% accuracy easily.\n",
    "\n",
    ">**Exercise:** Train a pretrained models to classify the cat and dog images. Continue with the DenseNet model, or try ResNet, it's also a good model to try out first. Make sure you are only training the classifier and the parameters for the features part are frozen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "uiHKuHIM2j2E",
    "outputId": "bc7ed49a-cfe4-40b3-eeef-c8012dffb5ad"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DenseNet(\n",
       "  (features): Sequential(\n",
       "    (conv0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
       "    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (relu0): ReLU(inplace=True)\n",
       "    (pool0): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
       "    (denseblock1): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(96, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (transition1): _Transition(\n",
       "      (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
       "    )\n",
       "    (denseblock2): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer7): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer8): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer9): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer10): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer11): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer12): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (transition2): _Transition(\n",
       "      (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
       "    )\n",
       "    (denseblock3): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer7): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer8): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer9): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer10): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer11): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer12): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer13): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer14): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer15): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer16): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer17): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer18): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer19): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer20): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer21): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer22): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer23): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer24): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (transition3): _Transition(\n",
       "      (norm): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
       "    )\n",
       "    (denseblock4): _DenseBlock(\n",
       "      (denselayer1): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer2): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer3): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer4): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer5): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer6): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer7): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer8): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer9): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer10): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer11): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer12): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer13): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer14): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer15): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "      (denselayer16): _DenseLayer(\n",
       "        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu1): ReLU(inplace=True)\n",
       "        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
       "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "        (relu2): ReLU(inplace=True)\n",
       "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      )\n",
       "    )\n",
       "    (norm5): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  )\n",
       "  (classifier): Sequential(\n",
       "    (0): Linear(in_features=1024, out_features=256, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Dropout(p=0.2, inplace=False)\n",
       "    (3): Linear(in_features=256, out_features=2, bias=True)\n",
       "    (4): LogSoftmax()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## TODO: Use a pretrained model to classify the cat and dog images\n",
    "# Use GPU if it's available\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model = models.densenet121(pretrained=True)\n",
    "\n",
    "# Freeze parameters so we don't backprop through them\n",
    "for param in model.parameters():\n",
    "    param.requires_grad = False\n",
    "    \n",
    "model.classifier = nn.Sequential(nn.Linear(1024, 256),\n",
    "                                 nn.ReLU(),\n",
    "                                 nn.Dropout(0.2),\n",
    "                                 nn.Linear(256, 2),\n",
    "                                 nn.LogSoftmax(dim=1))\n",
    "\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "# Only train the classifier parameters, feature parameters are frozen\n",
    "optimizer = optim.Adam(model.classifier.parameters(), lr=0.003)\n",
    "\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "fP6jc_7C6uTs",
    "outputId": "7219bc79-5e6d-4eb5-d6a7-edb00ade669b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1.. Train loss: 0.664.. Test loss: 0.293.. Test accuracy: 0.896\n",
      "Epoch 1/1.. Train loss: 0.325.. Test loss: 0.101.. Test accuracy: 0.972\n",
      "Epoch 1/1.. Train loss: 0.279.. Test loss: 0.121.. Test accuracy: 0.957\n",
      "Epoch 1/1.. Train loss: 0.212.. Test loss: 0.077.. Test accuracy: 0.971\n",
      "Epoch 1/1.. Train loss: 0.324.. Test loss: 0.063.. Test accuracy: 0.977\n",
      "Epoch 1/1.. Train loss: 0.189.. Test loss: 0.082.. Test accuracy: 0.968\n",
      "Epoch 1/1.. Train loss: 0.178.. Test loss: 0.057.. Test accuracy: 0.981\n",
      "Epoch 1/1.. Train loss: 0.198.. Test loss: 0.068.. Test accuracy: 0.975\n",
      "Epoch 1/1.. Train loss: 0.143.. Test loss: 0.076.. Test accuracy: 0.972\n",
      "Epoch 1/1.. Train loss: 0.215.. Test loss: 0.069.. Test accuracy: 0.974\n",
      "Epoch 1/1.. Train loss: 0.157.. Test loss: 0.069.. Test accuracy: 0.973\n",
      "Epoch 1/1.. Train loss: 0.186.. Test loss: 0.067.. Test accuracy: 0.977\n",
      "Epoch 1/1.. Train loss: 0.157.. Test loss: 0.051.. Test accuracy: 0.982\n",
      "Epoch 1/1.. Train loss: 0.237.. Test loss: 0.050.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.257.. Test loss: 0.051.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.217.. Test loss: 0.068.. Test accuracy: 0.975\n",
      "Epoch 1/1.. Train loss: 0.188.. Test loss: 0.052.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.178.. Test loss: 0.051.. Test accuracy: 0.982\n",
      "Epoch 1/1.. Train loss: 0.168.. Test loss: 0.047.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.171.. Test loss: 0.047.. Test accuracy: 0.983\n",
      "Epoch 1/1.. Train loss: 0.179.. Test loss: 0.053.. Test accuracy: 0.980\n",
      "Epoch 1/1.. Train loss: 0.159.. Test loss: 0.044.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.122.. Test loss: 0.058.. Test accuracy: 0.978\n",
      "Epoch 1/1.. Train loss: 0.099.. Test loss: 0.041.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.123.. Test loss: 0.056.. Test accuracy: 0.979\n",
      "Epoch 1/1.. Train loss: 0.187.. Test loss: 0.041.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.184.. Test loss: 0.074.. Test accuracy: 0.970\n",
      "Epoch 1/1.. Train loss: 0.176.. Test loss: 0.044.. Test accuracy: 0.982\n",
      "Epoch 1/1.. Train loss: 0.144.. Test loss: 0.041.. Test accuracy: 0.986\n",
      "Epoch 1/1.. Train loss: 0.169.. Test loss: 0.072.. Test accuracy: 0.971\n",
      "Epoch 1/1.. Train loss: 0.132.. Test loss: 0.043.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.128.. Test loss: 0.044.. Test accuracy: 0.986\n",
      "Epoch 1/1.. Train loss: 0.177.. Test loss: 0.078.. Test accuracy: 0.971\n",
      "Epoch 1/1.. Train loss: 0.165.. Test loss: 0.040.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.176.. Test loss: 0.041.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.181.. Test loss: 0.040.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.148.. Test loss: 0.045.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.153.. Test loss: 0.044.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.161.. Test loss: 0.047.. Test accuracy: 0.982\n",
      "Epoch 1/1.. Train loss: 0.144.. Test loss: 0.045.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.173.. Test loss: 0.055.. Test accuracy: 0.978\n",
      "Epoch 1/1.. Train loss: 0.113.. Test loss: 0.055.. Test accuracy: 0.979\n",
      "Epoch 1/1.. Train loss: 0.160.. Test loss: 0.069.. Test accuracy: 0.976\n",
      "Epoch 1/1.. Train loss: 0.206.. Test loss: 0.060.. Test accuracy: 0.979\n",
      "Epoch 1/1.. Train loss: 0.332.. Test loss: 0.044.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.223.. Test loss: 0.059.. Test accuracy: 0.986\n",
      "Epoch 1/1.. Train loss: 0.168.. Test loss: 0.055.. Test accuracy: 0.983\n",
      "Epoch 1/1.. Train loss: 0.136.. Test loss: 0.042.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.118.. Test loss: 0.040.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.166.. Test loss: 0.038.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.124.. Test loss: 0.037.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.188.. Test loss: 0.036.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.117.. Test loss: 0.058.. Test accuracy: 0.979\n",
      "Epoch 1/1.. Train loss: 0.173.. Test loss: 0.042.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.120.. Test loss: 0.039.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.102.. Test loss: 0.042.. Test accuracy: 0.983\n",
      "Epoch 1/1.. Train loss: 0.175.. Test loss: 0.043.. Test accuracy: 0.983\n",
      "Epoch 1/1.. Train loss: 0.142.. Test loss: 0.040.. Test accuracy: 0.987\n",
      "Epoch 1/1.. Train loss: 0.145.. Test loss: 0.049.. Test accuracy: 0.982\n",
      "Epoch 1/1.. Train loss: 0.162.. Test loss: 0.039.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.176.. Test loss: 0.040.. Test accuracy: 0.985\n",
      "Epoch 1/1.. Train loss: 0.155.. Test loss: 0.041.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.163.. Test loss: 0.048.. Test accuracy: 0.982\n",
      "Epoch 1/1.. Train loss: 0.144.. Test loss: 0.053.. Test accuracy: 0.981\n",
      "Epoch 1/1.. Train loss: 0.204.. Test loss: 0.041.. Test accuracy: 0.986\n",
      "Epoch 1/1.. Train loss: 0.144.. Test loss: 0.051.. Test accuracy: 0.983\n",
      "Epoch 1/1.. Train loss: 0.150.. Test loss: 0.071.. Test accuracy: 0.970\n",
      "Epoch 1/1.. Train loss: 0.195.. Test loss: 0.038.. Test accuracy: 0.984\n",
      "Epoch 1/1.. Train loss: 0.199.. Test loss: 0.048.. Test accuracy: 0.981\n"
     ]
    }
   ],
   "source": [
    "epochs = 1\n",
    "steps = 0\n",
    "running_loss = 0\n",
    "print_every = 5\n",
    "for epoch in range(epochs):\n",
    "    for inputs, labels in trainloader:\n",
    "        steps += 1\n",
    "        # Move input and label tensors to the default device\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        logps = model.forward(inputs)\n",
    "        loss = criterion(logps, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        if steps % print_every == 0:\n",
    "            test_loss = 0\n",
    "            accuracy = 0\n",
    "            model.eval()\n",
    "            with torch.no_grad():\n",
    "                for inputs, labels in testloader:\n",
    "                    inputs, labels = inputs.to(device), labels.to(device)\n",
    "                    logps = model.forward(inputs)\n",
    "                    batch_loss = criterion(logps, labels)\n",
    "                    \n",
    "                    test_loss += batch_loss.item()\n",
    "                    \n",
    "                    # Calculate accuracy\n",
    "                    ps = torch.exp(logps)\n",
    "                    top_p, top_class = ps.topk(1, dim=1)\n",
    "                    equals = top_class == labels.view(*top_class.shape)\n",
    "                    accuracy += torch.mean(equals.type(torch.FloatTensor)).item()\n",
    "                    \n",
    "            print(f\"Epoch {epoch+1}/{epochs}.. \"\n",
    "                  f\"Train loss: {running_loss/print_every:.3f}.. \"\n",
    "                  f\"Test loss: {test_loss/len(testloader):.3f}.. \"\n",
    "                  f\"Test accuracy: {accuracy/len(testloader):.3f}\")\n",
    "            running_loss = 0\n",
    "            model.train()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Part 8 - Transfer Learning.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "5b67c54257ab495791c12288c379919d": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5e83efe919b84b3a81999d61d05bf11d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "IntProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "IntProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_95783c39cd3047cb8c3ca442442dfbcc",
      "max": 32342954,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e771ca0ba45448889090688da9f771a5",
      "value": 32342954
     }
    },
    "861840fe1dd44ecb994c92a129bbcb04": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "91263dcfdf8c4e1cbf541bd0d6d65270": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_af33422af70e4abbbda6a1a0c0ba53c6",
      "placeholder": "​",
      "style": "IPY_MODEL_861840fe1dd44ecb994c92a129bbcb04",
      "value": "100% 30.8M/30.8M [00:04&lt;00:00, 7.99MB/s]"
     }
    },
    "95783c39cd3047cb8c3ca442442dfbcc": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "af33422af70e4abbbda6a1a0c0ba53c6": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c1f9c9aea0b04355a9575dd04117f141": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5e83efe919b84b3a81999d61d05bf11d",
       "IPY_MODEL_91263dcfdf8c4e1cbf541bd0d6d65270"
      ],
      "layout": "IPY_MODEL_5b67c54257ab495791c12288c379919d"
     }
    },
    "e771ca0ba45448889090688da9f771a5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}