Inverted_Index_Implementation_and_full_text_search_with_pyspark_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": " Inverted Index Implementation and full text search with pyspark-.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZU1b1W2UVq8B"
      },
      "source": [
        "This project is an Implementaion of Full Text Search with Invereted Index which is stored in mysql database. This Project is implemented using pyspark\n",
        "\n",
        "<u>The entire project is divided into 6 steps</u>\n",
        "\n",
        "-Step 0 - Setup Environment and Import Packages\n",
        "\n",
        "-Step 1 - Create Invereted Index and doc magnitude and store it in a file\n",
        "\n",
        "-Step 2 - Store it in a mysql\n",
        "\n",
        "-Step 3 - Lookup Inverted Index and get metrics\n",
        "\n",
        "-Step 4 - Calculate Cosine Similarity\n",
        "\n",
        "-Step 5 - Document Ranking\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u_fSpzDi_FAI"
      },
      "source": [
        "#Step 0 - Setup Environment and Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ots6kZqYVtQp"
      },
      "source": [
        "!pip install -q wget\n",
        "import wget\n",
        "import sys\n",
        "import os\n",
        "import string\n",
        "import pandas as pd\n",
        "import json\n",
        "import math\n",
        "import sqlite3"
      ],
      "execution_count": 142,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8uA1hjXuW8QB"
      },
      "source": [
        "# utility functions\n",
        "def wget_pbar_if_not_exists(url, save_path):\n",
        "\n",
        "  def bar_progress(current, total, width=80):\n",
        "    progress_message = \"Downloading: %d%% [%d / %d] bytes\" % (current / total * 100, current, total)\n",
        "    sys.stdout.write(\"\\r\" + progress_message)\n",
        "    sys.stdout.flush()\n",
        "\n",
        "  if not os.path.isfile(f'{save_path}'):\n",
        "    print('downloading', save_path)\n",
        "    wget.download(url, save_path, bar=bar_progress)\n",
        "  else:\n",
        "    print('file ', save_path, 'already exists')"
      ],
      "execution_count": 143,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V7CqdE1VhI8_",
        "outputId": "35e70e47-2b2b-4621-fe39-18c98726178a"
      },
      "source": [
        "!apt-get update \n",
        "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n",
        "wget_pbar_if_not_exists(url='https://archive.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz', save_path=f'./spark-2.4.4-bin-hadoop2.7.tgz')\n",
        "\n",
        "!tar xf spark-2.4.4-bin-hadoop2.7.tgz\n",
        "!pip install -q findspark\n",
        "\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
        "os.environ[\"SPARK_HOME\"] = \"/content/spark-2.4.4-bin-hadoop2.7\"\n",
        "print('4')\n",
        "import findspark\n",
        "findspark.init(\"spark-2.4.4-bin-hadoop2.7\")# SPARK_HOME\n",
        "\n",
        "from pyspark import SparkContext\n",
        "sc = SparkContext.getOrCreate()\n",
        "\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 144,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\r0% [Working]\r            \rHit:1 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease\n",
            "\r0% [Connecting to archive.ubuntu.com (91.189.88.152)] [Connecting to security.u\r0% [1 InRelease gpgv 3,626 B] [Connecting to archive.ubuntu.com (91.189.88.152)\r                                                                               \rIgn:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Waiting for headers] [Wait\r                                                                               \rHit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease\n",
            "\r                                                                               \rHit:4 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease\n",
            "\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Connecting to ppa.launchpa\r                                                                               \rHit:5 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "\r0% [1 InRelease gpgv 3,626 B] [Waiting for headers] [Connecting to ppa.launchpa\r                                                                               \rIgn:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "Hit:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
            "Hit:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "Hit:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease\n",
            "Hit:10 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n",
            "Hit:11 http://archive.ubuntu.com/ubuntu bionic-backports InRelease\n",
            "Hit:12 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease\n",
            "Hit:13 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
            "Reading package lists... Done\n",
            "file  ./spark-2.4.4-bin-hadoop2.7.tgz already exists\n",
            "4\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 144
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nmzp1HtM-6r2"
      },
      "source": [
        "#Step 1 - *Create* Invereted Index and doc magnitude and store it in a file\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tFQa1qU-Ml-B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "15a4410b-52ee-4122-80cd-691c3a41a5df"
      },
      "source": [
        "!unzip input_docs.zip > /dev/null\n",
        "!ls input_docs/ | wc -l"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "replace __MACOSX/._input_docs? [y]es, [n]o, [A]ll, [N]one, [r]ename: "
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "usmMrr5Uhipf"
      },
      "source": [
        "**Create an RDD from a text file**\n",
        "\n",
        "Each line of the text file becomes an element of the RDD."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TVfPs26GjAal"
      },
      "source": [
        "# wholeTextFiles generates an RDD of pair values, \n",
        "# where the key is the full path of each file, the value is the content of each file\n",
        "input = sc.wholeTextFiles(\"input_docs\");\n",
        "\n",
        "# Now we strip the prefix of filenames and leave only the basename. \n",
        "# e.g. 'file:/content/drive/My Drive/Colab Notebooks/data_spark/input_docs/3.html'\n",
        "# becomes '3.html' \n",
        "\n",
        "input2 = input.map(lambda x: (int(os.path.basename(x[0]).split(\".\")[0]), x[1]))\n",
        "\n",
        "for x in input2.take(5):\n",
        "  print(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AdFPY9uqgq1P"
      },
      "source": [
        "# Doc to wordlist function\n",
        "# The output will be a list of tuples such as \n",
        "# (\"apple\", (3,10,10/20)), \n",
        "# where 3 is docid, \n",
        "# 10 is frequency of \"apple\" in this doc, \n",
        "# 20 is maxf in in this doc.\n",
        "\n",
        "from bs4 import BeautifulSoup\n",
        "from collections import Counter\n",
        "import re\n",
        "\n",
        "from nltk.corpus import stopwords\n",
        "stop_words = set(stopwords.words('english'))\n",
        "\n",
        "# for a given doc return a list of tuples of the form (w, (docid, freq, freq/maxfreq))\n",
        "def dw(docid, htmltext):\n",
        "  cleantext = BeautifulSoup(htmltext).get_text()\n",
        "  result = list()\n",
        "  word_freq = dict()\n",
        "  max_freq = 0\n",
        "\n",
        "  for word in re.findall('[a-zA-Z]+', cleantext):\n",
        "    word = word.lower()\n",
        "    if word not in stop_words and word not in string.punctuation:\n",
        "      if word not in word_freq:\n",
        "        word_freq[word] = 1\n",
        "      else:\n",
        "        word_freq[word] += 1\n",
        "      if word_freq[word] > max_freq:\n",
        "        max_freq = word_freq[word]\n",
        "\n",
        "  for word, frequency in word_freq.items():\n",
        "    word_attrs = (word, (docid, frequency, frequency/max_freq))\n",
        "    result.append(word_attrs)\n",
        "\n",
        "  return result\n",
        "\n",
        "word_docid_freq_tf = input2.flatMap(lambda x: dw(x[0], x[1]))\n",
        "\n",
        "\n",
        "for r in word_docid_freq_tf.take(5):\n",
        "  print(r)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ZiKzsibkj92"
      },
      "source": [
        "# Now create an RDD as follows \n",
        "# (word, [(did1,freq1,tf1), (did2,freq2,tf2), ...])\n",
        "\n",
        "word_postinglist_freq_tf_grouped = word_docid_freq_tf.groupByKey().mapValues(list)\n",
        "\n",
        "for r in word_postinglist_freq_tf_grouped.take(5):\n",
        "  print(r)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y_8KGpQvlLGp"
      },
      "source": [
        "# (word, [(did,freq,tfidf), ...])\n",
        "# We easily obtain idf as 1/len(postinglist_tf)\n",
        "# idf = 1/len(postinglist_tf)\n",
        "\n",
        "def get_tfidf(word, postinglist):\n",
        "  idf  = 1/len(postinglist)\n",
        "  new_posting_list = list()\n",
        "  for element in postinglist:\n",
        "    tf = element[2]\n",
        "    tfidf = tf * idf\n",
        "    result_element = (element[0], element[1], tfidf)\n",
        "    new_posting_list.append(result_element)\n",
        "\n",
        "  return (word, new_posting_list)\n",
        "\n",
        "\n",
        "word_postinglist_freq_tfidf = word_postinglist_freq_tf_grouped.map(lambda x: get_tfidf(x[0], x[1]))\n",
        "\n",
        "\n",
        "for r in word_postinglist_freq_tfidf.take(5):\n",
        "  print(r)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U8jahG9q2SVf"
      },
      "source": [
        "# Now, we would like to obtain the magnitude of each doc.\n",
        "# First, produce (did, (freq,tfidf)) for each word of doc did; \n",
        "# We do don't need the word itself, just its (freq,tfidf). \n",
        "# Then, do reduceByKey on these tuples and obtain maxfreq and \n",
        "# magnitude (squared) for each document. \n",
        "\n",
        "\n",
        "import itertools\n",
        "import functools\n",
        "import operator\n",
        "\n",
        "from collections import Iterable\n",
        "\n",
        "\n",
        "def flatten(items):\n",
        "    \"\"\"Yield items from any nested iterable; see Reference.\"\"\"\n",
        "    for x in items:\n",
        "        if isinstance(x, Iterable) and not isinstance(x, (str, bytes)):\n",
        "            for sub_x in flatten(x):\n",
        "                yield sub_x\n",
        "        else:\n",
        "            yield x\n",
        "\n",
        "def tfidfsq(word, posting_list):\n",
        "  result = list()\n",
        "  for element in posting_list:\n",
        "    new_element = (element[0], (element[1], element[2]))\n",
        "    result.append(new_element)\n",
        "  return result\n",
        "\n",
        "\n",
        "def find_mag(docid, max_freq_mag_list):\n",
        "  max_freq = max_freq_mag_list[0]\n",
        "  mag_list = max_freq_mag_list[1]\n",
        "\n",
        "  mag_squared = 0\n",
        "  for ele in mag_list:\n",
        "    mag_squared += ele*ele\n",
        "  return (docid, (max_freq, mag_squared))\n",
        "\n",
        "did_freq_tfidfsq_rdd = word_postinglist_freq_tfidf.flatMap(lambda x: tfidfsq(x[0], x[1]))\n",
        "\n",
        "\n",
        "doc_maxf_mag = did_freq_tfidfsq_rdd.reduceByKey(lambda a,b: (max(a[0], b[0]), list(flatten([a[1], b[1]]))))\n",
        "\n",
        "doc_maxf_mag = doc_maxf_mag.map(lambda a: find_mag(a[0], a[1]))\n",
        "\n",
        "for r in doc_maxf_mag.take(5):\n",
        "  print(r)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zBByinGhlu7-"
      },
      "source": [
        "!rm -rf inv_idx\n",
        "word_postinglist_freq_tfidf.saveAsTextFile(\"inv_idx\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "szlsKC-RmWvd"
      },
      "source": [
        "!rm -rf doc_mag\n",
        "doc_maxf_mag.saveAsTextFile(\"doc_mag\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vNplByM4fsvF"
      },
      "source": [
        "!ls -lrt inv_idx\n",
        "!head inv_idx/part-00001\n",
        "!wc -l inv_idx/part-00000\n",
        "!wc -l inv_idx/part-00001\n",
        "!cat inv_idx/part-00000 inv_idx/part-00001 > /content/drive/MyDrive/inv_idx.txt\n",
        "!wc -l /content/drive/MyDrive/inv_idx.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Islv0-ONjyF1"
      },
      "source": [
        "!ls -lrt doc_mag\n",
        "!head doc_mag/part-00000\n",
        "!wc -l doc_mag/part-00000\n",
        "!wc -l doc_mag/part-00001\n",
        "!cat doc_mag/part-00000 doc_mag/part-00001 > /content/drive/MyDrive/doc_mag.txt\n",
        "!wc -l /content/drive/MyDrive/doc_mag.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGPrWl2tV5AU"
      },
      "source": [
        "\n",
        "\n",
        "#Step 2: Store it in a mysql\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZVjjpF8Ggli"
      },
      "source": [
        "sql_stmt_create_postings = \"CREATE TABLE IF NOT EXISTS postings ( word VARCHAR(100) PRIMARY KEY, postinglist_freq_tfidf TEXT );\"\n",
        "sql_stmt_create_docmag = \"CREATE TABLE IF NOT EXISTS docmag ( docid INT PRIMARY KEY, maxf INT, mag FLOAT );\"\n",
        "\n",
        "def create_table(conn, create_table_sql):\n",
        "    \"\"\" create a table from the create_table_sql statement\n",
        "    :param conn: Connection object\n",
        "    :param create_table_sql: a CREATE TABLE statement\n",
        "    :return:\n",
        "    \"\"\"\n",
        "    try:\n",
        "        c = conn.cursor()\n",
        "        c.execute(create_table_sql)\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "\n",
        "def create_connection(db_file):\n",
        "    \"\"\" create a database connection to the SQLite database\n",
        "        specified by db_file\n",
        "    :param db_file: database file\n",
        "    :return: Connection object or None\n",
        "    \"\"\"\n",
        "    conn = None\n",
        "    try:\n",
        "        conn = sqlite3.connect(db_file)\n",
        "        return conn\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    return conn\n",
        "\n",
        "os.remove('inverted_index_db.db') if os.path.exists('inverted_index_db.db') else None\n",
        "my_conn = create_connection('inverted_index_db.db')\n",
        "create_table(my_conn, sql_stmt_create_postings)\n",
        "create_table(my_conn, sql_stmt_create_docmag)\n",
        "\n",
        "#convert doc_mag.txt to dataframe\n",
        "df_doc_mag = pd.read_csv('/content/drive/MyDrive/doc_mag.txt', delimiter = \"\\t\", names=['raw_text'], index_col=False)\n",
        "df_doc_mag['raw_text'] = df_doc_mag['raw_text'].apply(lambda x: x.replace('(', '').replace(')', ''))\n",
        "df_doc_mag[['docid', 'maxf', 'mag']] = df_doc_mag['raw_text'].str.split(',', 2, expand=True)\n",
        "del df_doc_mag['raw_text']\n",
        "df_doc_mag = df_doc_mag.set_index('docid')\n",
        "\n",
        "#convert inv_idx.txt to dataframe\n",
        "df_doc_inv_idx = pd.read_csv('/content/drive/MyDrive/inv_idx.txt', delimiter = \"\\t\", names=['raw_text'], index_col=False)\n",
        "df_doc_inv_idx['raw_text'] = df_doc_inv_idx['raw_text'].apply(lambda x: x[1:-1])\n",
        "df_doc_inv_idx[['word', 'postinglist_freq_tfidf']] = df_doc_inv_idx['raw_text'].str.split(',', 1, expand=True)\n",
        "\n",
        "df_doc_inv_idx['word'] = df_doc_inv_idx['word'].apply(lambda x: x.replace(\"'\", '').replace('\"', ''))\n",
        "\n",
        "del df_doc_inv_idx['raw_text']\n",
        "df_doc_inv_idx = df_doc_inv_idx.set_index('word')\n",
        "\n",
        "\n",
        "#dataframe to sql\n",
        "df_doc_mag.to_sql('docmag', my_conn, if_exists='append')\n",
        "df_doc_inv_idx.to_sql('postings', my_conn, if_exists='append')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eUhXovqOXmsj"
      },
      "source": [
        "###verify if if the sql database was correctly populated"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dh7_f0SrOj3q"
      },
      "source": [
        "df_doc_mag = pd.read_sql('select * from docmag', my_conn, index_col='docid')\n",
        "print('df_doc_mag')\n",
        "print(df_doc_mag.dtypes)\n",
        "print(df_doc_mag.head())\n",
        "\n",
        "\n",
        "df_doc_inv_idx = pd.read_sql('select * from postings', my_conn, index_col='word')\n",
        "print('\\n\\ndf_doc_inv_idx')\n",
        "print(df_doc_inv_idx.dtypes)\n",
        "print(df_doc_inv_idx.head())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JurCxhxIYJna"
      },
      "source": [
        "# Step 3: Lookup Inverted Index and get metrics"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6d7UBiBFOj6E"
      },
      "source": [
        "\n",
        "def find_inv_idx(words):\n",
        "  quoted_words = [f\"'{word}'\" for word in words]\n",
        "  sql_tatement = f\"SELECT * from postings where word IN ({ ', '.join(quoted_words) })\"\n",
        "  df_posting_list = pd.read_sql(sql_tatement, my_conn, index_col='word')\n",
        "\n",
        "  data = []\n",
        "\n",
        "  for i in df_posting_list.itertuples():\n",
        "    row_list = re.findall(r'\\(.*?\\)', i[1].strip())\n",
        "    for row in row_list:\n",
        "      row = row[1:-1].split(',')\n",
        "      data.append([i[0], row[0], row[1], row[2]])\n",
        "\n",
        "  #unstack posting\n",
        "  df_posting_list_unstacked = pd.DataFrame(data=data, columns=['word', 'docid', 'freq', 'tfidf'])\n",
        "  df_posting_list_unstacked[\"docid\"] = pd.to_numeric(df_posting_list_unstacked[\"docid\"])\n",
        "  df_posting_list_unstacked[\"freq\"] = pd.to_numeric(df_posting_list_unstacked[\"freq\"])\n",
        "  df_posting_list_unstacked[\"tfidf\"] = pd.to_numeric(df_posting_list_unstacked[\"tfidf\"])\n",
        "\n",
        "  final_df = df_posting_list_unstacked.join(df_doc_mag, on='docid', how='left', lsuffix=\"_left\")\n",
        "  del final_df['mag']\n",
        "\n",
        "  return final_df\n",
        "\n",
        "search_words = ['business', 'relationships']\n",
        "inv_index_metrics = find_inv_idx(search_words)\n",
        "print(inv_index_metrics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NL0ce8znOj8Z"
      },
      "source": [
        "def get_query_metrics(words, search_inverted_index_metrics):\n",
        "  word_freq = dict()\n",
        "  word_idf = dict()\n",
        "  for word in words:\n",
        "    if word not in word_freq:\n",
        "      word_freq[word] = 1\n",
        "    else:\n",
        "      word_freq[word] += 1\n",
        "    \n",
        "  max_freq = 0\n",
        "  for freq in word_freq.values():\n",
        "    if max_freq < freq:\n",
        "      max_freq = freq\n",
        "\n",
        "  for word, df in search_inverted_index_metrics.groupby('word'):\n",
        "    word_idf[word] = 1/df.shape[0]\n",
        "  \n",
        "  for word in word_freq:\n",
        "    if word not in word_idf:\n",
        "      word_idf[word] = 0\n",
        "\n",
        "  data = list()\n",
        "  for word in word_freq:\n",
        "    data.append([word, word_freq[word], max_freq, word_idf[word]])\n",
        "  \n",
        "  df_query_metrics = pd.DataFrame(data=data, columns=['word', 'freq', 'maxf', 'idf'])\n",
        "  return df_query_metrics\n",
        "\n",
        "search_words = ['business', 'relationships']\n",
        "search_query_metrics = get_query_metrics(search_words, search_inverted_index_metrics)\n",
        "print(search_query_metrics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EgllmuE_AGJY"
      },
      "source": [
        "# Step 4 Calculate Cosine Similarity"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C__uubdrOj-0"
      },
      "source": [
        "def cosine_similarity(query, document):\n",
        "  query = query.set_index('word').T.to_dict()\n",
        "  document = document.set_index('word').T.to_dict()\n",
        "\n",
        "  numerator = 0\n",
        "\n",
        "  #calculate query[word]['tfidf']\n",
        "  for word in query:\n",
        "      query[word]['tfidf'] = ((query[word]['freq']/query[word]['maxf']) * query[word]['idf'])\n",
        "\n",
        "  #calculate numerator\n",
        "  for word in query:\n",
        "    if word in document:\n",
        "      numerator += query[word]['tfidf'] * document[word]['tfidf']\n",
        "\n",
        "  #calculate denominator\n",
        "  mag_query = 0\n",
        "  for word in query:\n",
        "    mag_query += query[word]['tfidf'] * query[word]['tfidf']\n",
        "\n",
        "  mag_document = 0\n",
        "  for word in document:\n",
        "    mag_document += document[word]['tfidf'] * document[word]['tfidf']\n",
        "\n",
        "  denominator =  math.sqrt(mag_query) * math.sqrt(mag_document)\n",
        "\n",
        " \n",
        "  cosine_similarity = numerator/denominator\n",
        "  return cosine_similarity\n",
        "\n",
        "cosine_similarity(search_query_metrics, inv_index_metrics.loc[inv_index_metrics['docid']==1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c8Umym0bAKI8"
      },
      "source": [
        "#Step 5 Document Ranking"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tS0L1-OoAOhh"
      },
      "source": [
        "def rank_documents(df_query_metrics, inv_index_metrics, show_top=20):\n",
        "  doc_score = dict()\n",
        "  for docid, df_doc_metrics in inv_index_metrics.groupby('docid'):\n",
        "    doc_score[docid] = cosine_similarity(df_query_metrics, df_doc_metrics)\n",
        "  \n",
        "  if not doc_score:\n",
        "    print('No matches found')\n",
        "  res_count = 0\n",
        "  for w in sorted(doc_score, key=doc_score.get, reverse=True):\n",
        "      res_count += 1\n",
        "      if res_count > show_top:\n",
        "        break\n",
        "      print('doc {:>8}   {:<20}'.format(w, doc_score[w]))\n",
        "      \n",
        "\n",
        "rank_documents(search_query_metrics, inv_index_metrics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OjOVpPekEHXz"
      },
      "source": [
        "# User Interface"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jRPoiYodElih"
      },
      "source": [
        "Enter words here"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "J5Jr6XmdEhur"
      },
      "source": [
        "#@title Enter Search Query Here\n",
        "\n",
        "#type your search query here\n",
        "input_words = 'business relationships' #@param (type:\"string\")\n",
        "# input_words = 'interest payments' #@param (type:\"string\")\n",
        "# input_words = 'would' #@param (type:\"string\")\n",
        "\n",
        "#remove stop words\n",
        "search_words = [word.lower().strip() for word in input_words.split() if word not in stop_words and word not in string.punctuation]\n",
        "\n",
        "search_inverted_index_metrics = find_inv_idx(search_words)\n",
        "search_query_metrics = get_query_metrics(search_words, search_inverted_index_metrics)\n",
        "\n",
        "rank_documents(search_query_metrics, search_inverted_index_metrics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OpOm7jeYRUpX"
      },
      "source": [
        "below output shows some meta data to understand document scoring"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XMOAk57ZRMVv"
      },
      "source": [
        "print('META DATA')\n",
        "print('\\n\\n')\n",
        "print('inverted_index_metrics')\n",
        "print(search_inverted_index_metrics)\n",
        "print('\\n\\n')\n",
        "print('query_metrics')\n",
        "print(search_query_metrics)\n",
        "print('\\n\\n')\n",
        "print('cosine similarity with doc 3')\n",
        "cosine_similarity(search_query_metrics, search_inverted_index_metrics.loc[search_inverted_index_metrics['docid']==3])"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}