assignments/assignment_yourname_class5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XKzF6dMaiLyP"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/assignments/assignment_yourname_class5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YDTXd8-Lmp8Q"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n",
    "\n",
    "**Module 5 Assignment: Computer Vision Neural Network**\n",
    "\n",
    "**Student Name: Your Name**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ncNrAEpzmp8S"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "This assignment will be most straightforward if you use Google CoLab, because it requires both PyTorch and YOLOv5 to be installed. It will be necessary to mount your GDrive so that you can send your notebook during the submit process. Running the following code will map your GDrive to ```/content/drive```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fU9UhAxTmp8S",
    "outputId": "7637a3dd-0799-46a7-e7a0-5510c487b7c8"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive', force_remount=True)\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QSKZqD1Mmp-C"
   },
   "source": [
    "# Assignment Submit Function\n",
    "\n",
    "You will submit the 10 programming assignments electronically.  The following submit function can be used to do this.  My server will perform a basic check of each assignment and let you know if it sees any basic problems.\n",
    "\n",
    "**It is unlikely that should need to modify this function.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7F2MhA7bjag8"
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "import PIL\n",
    "import PIL.Image\n",
    "import io\n",
    "\n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - List of pandas dataframes or images.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.\n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    payload = []\n",
    "    for item in data:\n",
    "        if type(item) is PIL.Image.Image:\n",
    "            buffered = BytesIO()\n",
    "            item.save(buffered, format=\"PNG\")\n",
    "            payload.append({'PNG':base64.b64encode(buffered.getvalue()).decode('ascii')})\n",
    "        elif type(item) is pd.core.frame.DataFrame:\n",
    "            payload.append({'CSV':base64.b64encode(item.to_csv(index=False).encode('ascii')).decode(\"ascii\")})\n",
    "    r= requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={ 'payload': payload,'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code==200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8fJKkSenqklH"
   },
   "source": [
    "# Assignment Instructions\n",
    "\n",
    "For this assignment, you will use YOLO running on Google CoLab.  I suggest that you run this assignment on CoLab because the example code below is already setup to get you started with the correct versions of  YOLO on PyTorch.\n",
    "\n",
    "For this assignment you are provided with 10 image files that contain 10 different webcam pictures taken at the [Venice Sidewalk Cafe](https://www.westland.net/beachcam/) a WebCam that has been in opration since 1996.  You can find the 10 images here:\n",
    "\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk1.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk2.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk3.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk4.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk5.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk6.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk7.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk8.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk9.jpg\n",
    "* https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk10.jpg\n",
    "\n",
    "You can see a sample of the WebCam here:\n",
    "\n",
    "![alt text](https://data.heatonresearch.com/data/t81-558/sidewalk/sidewalk1.jpg)\n",
    "\n",
    "YOLO does quite well-recognizing objects in this webcam, as the following image illustrates.\n",
    "\n",
    "![alt text](https://data.heatonresearch.com/data/t81-558/sidewalk/predictions.jpg)\n",
    "\n",
    "You are to write a script that counts the number of certain objects in each of the images.  Specifically, you are looking for:\n",
    "\n",
    "* person\n",
    "* car\n",
    "* bicycle\n",
    "* motorbike\n",
    "* umbrella\n",
    "* handbag\n",
    "\n",
    "It is essential that your use YOLO with a threshold of 10% if you want your results to match mine. The sample code below already contains this setting.  Your program can set this threshold with the following command.\n",
    "\n",
    "* conf_thres=0.1  # confidence threshold (use this value)\n",
    "* iou_thres=0.25  # NMS IOU threshold (use this value)\n",
    "\n",
    "Your submitted data frame should also contain a column that identifies which image generated each row.  This column should be named **image** and contain integer numbers between 1 and 10.  There should be 10 rows in total.  The complete data frame should look something like this (not necessarily exactly these numbers).\n",
    "\n",
    "|image|person|car|bicycle|motorbike|umbrella|handbag|\n",
    "|-|-|-|-|-|-|-|\n",
    "|1|23|0|3|4|0|0|\n",
    "|2|27|1|8|2|0|0|\n",
    "|3|29|0|0|0|3|0|\n",
    "|...|...|...|...|...|...|...|\n",
    "\n",
    "\n",
    "The following code sets up YOLO and then dumps the classification information for the first image.  This notebook only serves to get you started.  Read in all ten images and generate a data frame that looks like the following. Use the **submit** function as you did in previous assignments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ym5_juokofQl"
   },
   "source": [
    "### Installing YOLOv5\n",
    "\n",
    "YOLO is not available directly through either PIP or CONDA.  Additionally, YOLO is not installed in Google CoLab by default. Therefore, whether you wish to use YOLO through CoLab or run it locally, you need to go through several steps to install it.  This section describes the process of installing YOLO.  The same steps apply to either CoLab or a local install.  For CoLab, you must repeat these steps each time the system restarts your virtual environment.  You must perform these steps only once for your virtual Python environment for a local install.  If you are installing locally, make sure to install to the same virtual environment you created for this course.  The following commands install YOLO directly from its GitHub repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VuTjby5MzEre",
    "outputId": "3f479ee8-f376-4954-bf32-a85d0beec5d1"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "!git clone https://github.com/ultralytics/yolov5 --tag 6.2  # clone\n",
    "!mv /content/6.2 /content/yolov5\n",
    "%pip install -qr /content/yolov5/requirements.txt  # install\n",
    "sys.path.insert(0,'/content/yolov5')\n",
    "\n",
    "import torch\n",
    "import utils\n",
    "display = utils.notebook_init()  # checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qYOvD3M7ofQl"
   },
   "source": [
    "### Running YOLOv5\n",
    "\n",
    "In addition to the command line execution we just saw, the YOLO library can easily integrate with Python applications.  The following code adds the downloaded YOLOv5 to Python's environment, allowing **yolov5** to be imported like a regular Python library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MY3gVyidmp-K",
    "outputId": "8fa4b9ce-7f4b-410b-f717-245867535d79"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# Model\n",
    "yolo_model = torch.hub.load('ultralytics/yolov5', 'yolov5s')  # or yolov5n - yolov5x6, custom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sGbur-vdZWyz"
   },
   "source": [
    "I built the following function from the code presented in the course module. The function combines some of the code from the module to accept an image and return what YOLO recognizes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lCy_pvDXqYv4"
   },
   "source": [
    "### Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SFmqHkLbYQVm",
    "outputId": "537fc38d-fb0b-4f1e-ad97-6fcd83307318"
   },
   "outputs": [],
   "source": [
    "# Add your solution here, put your results into submit_df\n",
    "\n",
    "# This is your student key that I emailed to you at the beginnning of the semester.\n",
    "#key = \"5iuwhudihwiao6dsfw7dE2ml08iNfVOg6l0O3M06\"  # This is an example key and will not work.\n",
    "key = \"uTtH5yNbPs9tZhhhhdRWsBf9V99RU2iP5cL7F3zH\"\n",
    "\n",
    "# You must also identify your source file.  (modify for your local setup)\n",
    "file='/content/drive/MyDrive/Colab Notebooks/assignment_solution_class5.ipynb'  # Google CoLab\n",
    "\n",
    "submit(source_file=file,data=[submit_df],key=key,no=5)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}