Understanding_Videos_at_Scale.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Understanding_Videos_at_Scale.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/JasperLS/Understanding_Videos_at_Scale/blob/master/Understanding_Videos_at_Scale.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pW5CvsYJskRb"
      },
      "source": [
        "# **Understanding Videos at Scale**\n",
        "**_How to Extract Insights for Business Research_**\n",
        "\n",
        "- **Output**: This script extracts various video features from a given set of videos and stores these features in multiple .csv files, which can be opened with Excel or any text editor afterwards. It will store results in your Google Drive. This requires some free space (<1 MB).\n",
        "- **Input**: You need to provide a Google Drive Account with a folder with video files. Atlernatively, one example video is provided. \n",
        "- **Execution**: In order to execute this script, **run all cells from top to bottom and follow the instructions**\n",
        "\n",
        "_This script uses multiple external tools. Different licenses may apply depending on the specific video features you intend to extract. You can find all additional licenses in a licenses.zip file, which will be automatically downloaded._"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fHqjvh9mS82"
      },
      "source": [
        "## **1. Download Models**\n",
        "_You need a Google Account to verify your legitimate use; The models will be automatically downloaded from a public GDrive_\n",
        "_____________________________________________________________________________\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dlDsxEbfP891"
      },
      "source": [
        "!pip install -U pillow==6.1\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "okoFRhcK7n-d"
      },
      "source": [
        "**<font color='red'>After completion: Please click -> Runtime -> Change runtime type -> GPU in top menu</font>**\n",
        "\n",
        "If runtime is already set to GPU, please click Runtime -> Restart runtime"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install imageio_ffmpeg\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "metadata": {
        "id": "xf_pOGmxzCcJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zUUG2Emgx_ET"
      },
      "source": [
        "import os\n",
        "import shutil\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "from tqdm import tqdm\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkvXvCwDhBlU"
      },
      "source": [
        "Running the next cell will create a link. **You need to click this link and enter your Google Account credentials. Copy the code and enter it in the space below.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WhD5gLYbzkOR"
      },
      "source": [
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdNhcxUim2ta"
      },
      "source": [
        "Running the next cell will download required models from a public GDrive. After running it, you can verify success by clicking on the folder icon on the left side of this screen.There should be a 'downloads' folder containing various files (You may need to click 'Refresh')."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O2hky3wJF17G"
      },
      "source": [
        "os.makedirs('/content/downloads/', exist_ok=True)\n",
        "os.chdir('/content/downloads/')\n",
        "\n",
        "folder_id = '1e-UQc-ylzVOOvW2ZiOCpnP-EEziHA4cQ'\n",
        "file_list = drive.ListFile({'q': \"'{}' in parents and trashed=false\".format(folder_id)}).GetList()\n",
        "for file in tqdm(file_list):\n",
        "    file.GetContentFile(file['title'])\n",
        "\n",
        "os.chdir('/content/')\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4f_8XH-Bmmee"
      },
      "source": [
        "## **2. Provide Input & Select Features**\n",
        "_You need to provide a Google Drive Account, which includes a folder with video files for video mining._\n",
        "_____________________________________________________________\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1G3nRSNiHPi"
      },
      "source": [
        "Running the next cell will create a link. **You need to click this link and enter your Google Account credentials. Copy the code and enter it in the space below.** If you just want to test the example video, you can skip this step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "42Icp9RCnAYB"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7S84YwhciPYP"
      },
      "source": [
        "**<font color='red'>Please provide the correct path to your folder with videos (example in comment) </font>**\n",
        "\n",
        "If you just want to test an example video, leave the folder path to 'None'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Xc6ul7ETY2g"
      },
      "source": [
        "in_folder = None       # '/content/drive/My Drive/trailer/vids/'  folder with videos, names are used as IDs\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLOyesxvnRKy"
      },
      "source": [
        "If you did not provide an in_folder, the following cell will create a folder with one example_video for analysis\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "88LL5s2ZnRYU"
      },
      "source": [
        "if in_folder == '' or in_folder == None: \n",
        "  os.makedirs('/content/examples/',exist_ok=True)\n",
        "  shutil.copy('/content/downloads/example_video.mp4','/content/examples/example_video.mp4')\n",
        "  in_folder = '/content/examples/'\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mJuuNBDUkGee"
      },
      "source": [
        "You **can unselect features** you are not interested in **by writing 'False' instead of 'True'**. This may drastically increase processing speed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3BR9d_Kvui7m"
      },
      "source": [
        "extract_length           = True # get duration of video\n",
        "extract_cuts             = True # get scene cuts\n",
        "extract_colors           = True # get resolution, colorfulness, saturation, value, and 11 colors\n",
        "extract_faces            = True # get faces\n",
        "extract_emotions         = True # get 7 emotions per face \n",
        "extract_objects          = False # get humans, 8 animals, and 71 objects ATTENTION - THERE SEEMS TO BE AN INTERDEPENDENCE ERROR, PLEASE USE THIS OPTION SEPARATELY FROM EVERYTHING ELSE\n",
        "extract_variance         = True # get scene to scene and overall visual variance\n",
        "extract_quality          = True # get quality\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jnI2Vu0djci4"
      },
      "source": [
        "## **3. Provide Optional Input**\n",
        "_**Run all cells - no additional input required, however you may change things to customize execution**_\n",
        "_____________________________________________________________\n",
        "\n",
        "The following folders and the logfile will be created to store the results of your analysis. **You do not have to change them, unless you want to use your custom folder** structure to store results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "505rRLPFjeGG"
      },
      "source": [
        "out_folder = '/content/drive/My Drive/video_analysis/preds/' # folder to store extracted features\n",
        "log_name   = 'DATE_logfile.csv' # name of lofile, pls include .csv ending \n",
        "log_folder = '/content/drive/My Drive/video_analysis/logs/' # folder for logfile\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GXdB63WE40CN"
      },
      "source": [
        "Setting a start and end index allows you to only analyze a subset of the videos in your folder. **You do not have to change this, if you want to analyze all videos in your provided folder.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UNLRRRV_40K2"
      },
      "source": [
        "start_index = 0\n",
        "end_index = len(os.listdir(in_folder))\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYU3L3SU4h4m"
      },
      "source": [
        "## **4. Extract Selected Features**\n",
        "_**Run all cells - no additional input required**_\n",
        "_____________________________________________________________________________\n",
        "\n",
        "_This will extract your selected features by looping through your provided video files folder. (Depending on number and size of your videos, this may take some minutes or multiple hours)._"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sopyJEtc3fIP",
        "outputId": "61adeaf8-4333-4827-8792-61a271789328",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "variables = ''\n",
        "for variable in ['extract_length',\n",
        "                 'extract_cuts',\n",
        "                 'extract_colors',\n",
        "                 'extract_faces',\n",
        "                 'extract_emotions',\n",
        "                 'extract_variance',\n",
        "                 'extract_objects',\n",
        "                 'extract_quality',\n",
        "                 'start_index',\n",
        "                 'end_index']:\n",
        "  variables += ' --' + variable.replace('_','-') + ' ' + str(eval(variable))\n",
        "\n",
        "for variable in ['in_folder',\n",
        "                 'out_folder',\n",
        "                 'log_folder',\n",
        "                 'log_name']:\n",
        "  variables += ' --' + variable.replace('_','-') + ' \"' + str(eval(variable))+'\"'\n",
        "\n",
        "!python \"/content/downloads/feature_extraction_main.py\" {variables}\n",
        "print(f'Your feature extraction is completed, you will find your results in {out_folder} on the left side. \\\n",
        "\\nIf you had provided a Drive Account, results will have been automatically stored to this folder in your Drive account')"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\r100% 1/1 [09:49<00:00, 589.57s/it]\r100% 1/1 [09:49<00:00, 589.57s/it]\n",
            "Your feature extraction is completed, you will find your results in /content/drive/My Drive/video_analysis/preds/ on the left side. \n",
            "If you had provided a Drive Account, results will have been automatically stored to this folder in your Drive account\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ihG_fkTcaf_"
      },
      "source": [
        "## _**5. APPENDIX: Optional Extras**_\n",
        "_The following code snippets may be used for specific purposes._\n",
        "_____________________________________________________________________________"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5X0pYqZc8Vh"
      },
      "source": [
        "### **5.1 Analyzing Videos from Youtube or Similar Sources**\n",
        "Instead of a folder with videos, this code only requires you to provide a list of URLs. **Please provide this list in he following cell.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mmNgyNJl8K4M"
      },
      "source": [
        "urls = ['https://www.youtube.com/watch?v=TcMBFSGVi1c',\n",
        "        'https://www.youtube.com/watch?v=VyHV0BRtdxo'] #replace with your list of URLs\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sgN4JERn9Yf6"
      },
      "source": [
        "You **can unselect features** you are not interested in **by writing 'False' instead of 'True'**. This may drastically increase processing speed.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsmWZN-v7aGl"
      },
      "source": [
        "extract_length           = True # get length of video\n",
        "extract_cuts             = True # get scene cuts\n",
        "extract_colors           = True # get brightness and  color information\n",
        "extract_faces            = True # get faces\n",
        "extract_emotions         = True # get 8 different emotions per face \n",
        "extract_objects          = True # get 80 objects\n",
        "extract_variance         = True # get semantic variance\n",
        "extract_quality          = True # get quality\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nw-ThBr9l44"
      },
      "source": [
        "The following folders and the logfile will be created to store the results of your analysis. You do not have to change them, unless you want to use your custom folder structure to store results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pOXmP_pa9m4E"
      },
      "source": [
        "out_folder = '/content/drive/My Drive/video_analysis/preds/' # folder to store extracted features\n",
        "log_name   = '2020-06-09_logfile_downloads.csv' # name of lofile, pls include .csv ending \n",
        "log_folder = '/content/drive/My Drive/video_analysis/logs/' # folder for logfile\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L_qNW_WO8yC1"
      },
      "source": [
        "**Now you can run all cells to analyze the videos in the URLs provided.** The code will temporarily create a copy of the video in your Colab work environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "74nVk57wcj1S"
      },
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "!pip install -q youtube-dl\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V3ZaEl5o7S8O"
      },
      "source": [
        "os.makedirs('/content/download_vids/',exist_ok=True)\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3d8VQRmS-c4Z"
      },
      "source": [
        "in_folder = '/content/download_vids/'\n",
        "start_index = 0\n",
        "\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_mk5tokX_qMQ"
      },
      "source": [
        "for video in urls:\n",
        "  name = video[-11:]\n",
        "  print(name,video)\n",
        "\n",
        "  if not name+'.mp4' in os.listdir(in_folder):\n",
        "    !youtube-dl -f 22 --output $in_folder$name\".%(ext)s\" $video\n",
        "print('\\n successfully executed, please proceed with next step\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h__0t5XVqpXn"
      },
      "source": [
        "end_index = len(os.listdir(in_folder))\n",
        "variables = ''\n",
        "for variable in ['extract_length',\n",
        "                'extract_cuts',\n",
        "                'extract_colors',\n",
        "                'extract_faces',\n",
        "                'extract_emotions',\n",
        "                'extract_variance',\n",
        "                'extract_objects',\n",
        "                'extract_quality',\n",
        "                'start_index',\n",
        "                'end_index']:\n",
        "  variables += ' --' + variable.replace('_','-') + ' ' + str(eval(variable))\n",
        "\n",
        "for variable in ['in_folder',\n",
        "                'out_folder',\n",
        "                'log_folder',\n",
        "                'log_name']:\n",
        "  variables += ' --' + variable.replace('_','-') + ' \"' + str(eval(variable))+'\"'\n",
        "\n",
        "!python \"/content/downloads/feature_extraction_main.py\" {variables}\n",
        "print(f'Your feature extraction is completed, you will find your results in {out_folder} on the left side. \\\n",
        "\\nIf you had provided a Drive Account, results will have been automatically stored to this folder in your Drive account')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DhSU_pf1ghW5"
      },
      "source": [
        "### **5.2 Extracting Text from Videos or Images**\n",
        "**Please provide a folder with all frames/images** you want to run optical character recognition on. You can use the default '/content/image2text/images/' without any changes, which will automatically analyze 3 example images.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K_8qgQbXTZfq"
      },
      "source": [
        "ocr_image_folder   = '/content/scene_imgs/example_video/frames/'\n",
        "text_output_folder = '/content/scene_imgs/'\n",
        "print_text = True\n",
        "show_images = True\n",
        "\n",
        "!python \"/content/downloads/OCR.py\" --images-folder {ocr_image_folder} --text-folder {text_output_folder} --verbose {print_text}\n",
        "print(f'Your text recognition is completed, you will find your results in {text_output_folder} on the left side.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIVx-m_fLcwD"
      },
      "source": [
        "The next cell will show all images in given ocr_image_folder, allowing for comparison between detected text and image."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3VN32fE-K3cq"
      },
      "source": [
        "if show_images:\n",
        "    %matplotlib inline\n",
        "    import matplotlib.pyplot as plt\n",
        "    import cv2\n",
        "    for image in os.listdir(ocr_image_folder):\n",
        "      img = cv2.imread(ocr_image_folder + image)\n",
        "      plt.axis('off')\n",
        "      plt.imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB))\n",
        "      plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}