what_is_X_up_to.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "761dd70c-7376-479f-ac32-f4f5e63ff8c1",
   "metadata": {},
   "source": [
    "# About the Notebook\n",
    "\n",
    "In this notebook, we'll inspect how we can obtain structured information about a particular entity, using various types of data, and ML methods.\n",
    "\n",
    "Sections:\n",
    "1. Getting the Data Ready\n",
    "\n",
    "    - We'll collect some data on the GME stock from Reddit.\n",
    "    - We'll parse the data, and assume we push it into an ES index.\n",
    "    - We'll query actual data about GME, from an ES index.\n",
    "    \n",
    "2. Running NER and RelEx on the Data\n",
    "\n",
    "    - We'll \n",
    "    \n",
    "3. Parsing and Saving the results\n",
    "\n",
    "    - This time, we'll assume we have NER + RelEx applied data on \"where Elon Musk went\" recently.\n",
    "    - We'll parse our data to nodes and edges, and save the data.\n",
    "    - We'll both print the data to see it in the notebook, and also run a Dash App to see it as a graph.\n",
    "    \n",
    "\n",
    "\n",
    "## Note:\n",
    "\n",
    "This is not a tutorial notebook, so it does not implement the process end-to-end.\n",
    "\n",
    "However it gathers most of the components we need together, to do the analysis. It should give us an idea on how to form a complete pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58599c78-cf10-4e32-b0a0-28712a585a22",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd /Users/pydata_pres"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7b8c2b-24e3-40a3-9563-6b3d099fd71b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# 1- Getting the Data Ready \n",
    "\n",
    "- We'll collect some data on the GME stock from Reddit.\n",
    "- We'll parse the data, and assume we push it into an ES index.\n",
    "- We'll query actual data about GME, from an ES index."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5319888-a0bd-4a04-b367-c54c933513b7",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Scrape Data from the Identified Data Source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f161bd6-5e19-4401-84bf-395d596d7ea9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.geeksforgeeks.org/scraping-reddit-with-python-and-beautifulsoup/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09bef0c2-a04b-42cc-8adc-ad19e5e812e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "from datetime import date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47ac9900-8e09-4710-9982-1c528a8ebe42",
   "metadata": {},
   "outputs": [],
   "source": [
    "def getdata(url):\n",
    "    r = requests.get(url)\n",
    "    return r.text\n",
    "\n",
    "url = \"https://www.reddit.com/r/GME/comments/mfv9we/just_posted_on_sec_%D0%BEver_500000_awarded_to/\"\n",
    "  \n",
    "htmldata = getdata(url)\n",
    "soup = BeautifulSoup(htmldata, 'html.parser')\n",
    "    \n",
    "# print(soup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72d707c4-2d0c-440f-90f8-f1058c12dfb5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "page = []\n",
    "for i,item in enumerate(soup.find_all(\"p\")):\n",
    "    page.append(item.get_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15a9136f-1cd1-46d5-bf67-a67dfce70309",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a3b2be3-c50b-41a8-b605-72cafd04a6a1",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# [\"Link to the Press Release on SEC's website:\",\n",
    "#  'https://www.sec.gov/news/press-release/2021-54',\n",
    "#  'From the release:',\n",
    "#  'FOR IMMEDIATE RELEASE2021-54',\n",
    "#  'Washington D.C., March 29, 2021 —',\n",
    "#  \"The Securities and Exchange Commission awarded more than $500,000 to a whistleblower who raised concerns internally before submitting a tip to the Commission.\\xa0The whistleblower's information and assistance allowed the Commission and another agency to quickly file actions, shutting down an ongoing fraudulent scheme.\",\n",
    "#  'The whistleblower\\'s information prompted an internal investigation by the company, which then reported to an outside agency, which in turn provided the information to the SEC.\\xa0Separately, the whistleblower also reported to the SEC within 120 days of reporting the violations internally to the company.\\xa0Under the \"safe harbor\"\\xa0provision of the SEC\\'s whistleblower rules, the SEC treats the whistleblower\\'s information as though it had been submitted to the SEC at the same time it was internally reported as long as the whistleblower also reports the information to the SEC within 120 days of the internal report.',"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54f092b4-60f1-4bc9-8d83-c2dd628c9f65",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Parse Scraped Data to a Common Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98e76fa2-7560-4ec4-8c1f-bb0d6682f411",
   "metadata": {},
   "outputs": [],
   "source": [
    "datastore = {}\n",
    "page_parsed_into_a_common_format = {\"id\":123, \n",
    "                                    \"url\":url,\n",
    "                                    \"date_obtained\":date.today(),\n",
    "                                    \"content\":[]}\n",
    "for i,par in enumerate(page):\n",
    "    page_parsed_into_a_common_format[\"content\"].append(par)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c1effe-b351-4cfb-9405-55ea375e2d82",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "page_parsed_into_a_common_format"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d231938-affd-4f66-b307-984b8142ed9d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Store the Common Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d997957f-70fa-48ff-bffc-22d4ed08b121",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from: https://stackoverflow.com/questions/66049377/insert-new-document-using-python-elastic-client-raises-illegal-argument-exceptio\n",
    "# res = es.index(index='reddit_pages', doc_type=\"_doc\", id=page_parsed_into_a_common_format['id'], body=page_parsed_into_a_common_format)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e3d813f-99ac-46e3-9312-dc433199fc7d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Query your Data-Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b99a792f-3f57-4c05-8f0a-f5e8725769bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from elasticsearch.helpers import scan\n",
    "from elasticsearch import Elasticsearch\n",
    "from pydata_pres_vars import MY_ELASTICSEARCH_URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0af3b30-b26a-4ccb-a37a-10ae99932636",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Elasticsearch(MY_ELASTICSEARCH_URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "659fe138-9eff-4f6a-8bab-64bbc8a4b60b",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = {\n",
    "        \"bool\": {\n",
    "            \"filter\": [\n",
    "                {\"range\": {\"date\": {\"gte\": \"2022-06-18\", \"lt\": \"2022-06-19\"}}},\n",
    "                {\"query_string\": {\"query\": '\"elon musk\"', \"default_field\": \"content\"}}\n",
    "            ],\n",
    "        }\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "109c3b6b-960a-4b5e-b827-05858e0bb2c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.count(index=\"en_search\", body={\"query\":query})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baab7f89-2cb0-47c5-bbf0-2ccbe187b2c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydata_pres_vars import MY_INDEX_NAME\n",
    "\n",
    "scanner = scan(\n",
    "    client=client,\n",
    "    index=\"en_search\",\n",
    "    query={\n",
    "        \"query\": query,\n",
    "        \"_source\": [\n",
    "            \"date\",\n",
    "            \"title\",\n",
    "            \"entities.organizations\",\n",
    "            \"content\",\n",
    "            \"spans\",\n",
    "        ],\n",
    "    },\n",
    ")\n",
    "\n",
    "docs_full = [{\"id\": d[\"_id\"], **d[\"_source\"]} for d in scanner]\n",
    "docs = [d['content'] for d in docs_full]\n",
    "\n",
    "len(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c152bd0-c868-4e7d-8690-c8d00c0f682b",
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in docs[:3]:\n",
    "    print(doc[:175]+'..','\\n','_____________________')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "becaaaf6-024b-4916-9a58-0a9907523331",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# THREE SpaceX employees have been sacked after calling chief executive Elon Musk a \"source of distraction and embarrassment\" in an open letter.\n",
    "\n",
    "# The letter, which was first cir.. \n",
    "#  _____________________\n",
    "# Now the crypto industry is grappling with an even grimmer prospect: The worst may be yet to come.\n",
    "\n",
    "# Written by David Yaffe-Bellany\n",
    "\n",
    "# Cryptocurrency prices are plummeting. A so-c.. \n",
    "#  _____________________\n",
    "# Elon Musk warned Twitter staffers its business needed to \"get healthy\" and undergo a \"rationalisation of headcount and expenses\" as he addressed the social media platform's em.. \n",
    "#  _____________________"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1423a646-f173-41a5-865c-7bdb3bffaea2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# 2- Running NER and RelEx on the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913d14f4-1810-4601-82de-9586480b6864",
   "metadata": {},
   "source": [
    "![title](imgs/ner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad820f80-5830-43e1-b4be-e7d266e847ad",
   "metadata": {},
   "source": [
    "![title](imgs/relex.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15215377-fb2f-4d8d-b2e1-ae87af9f1c4c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "# 3- Parsing and Saving Results\n",
    "\n",
    "- This time, we'll assume we have NER + RelEx applied data on \"where Elon Musk went\" recently.\n",
    "- We'll parse our data to nodes and edges, and save the data.\n",
    "- We'll both print the data to see it in the notebook, and also run a Dash App to see it as a graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d27d4117-62bc-4999-8dd7-fd4c4082f1c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4ba992c-8521-4817-b65c-49a58ca96147",
   "metadata": {},
   "outputs": [],
   "source": [
    "RESULTS_PATH = \"elon_dataset.json\" \n",
    "\n",
    "with open(RESULTS_PATH,\"r\") as f:\n",
    "    elon_dataset = json.loads(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e94b2f4-24ef-409b-a746-daa48d3768b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "extracted_information = []\n",
    "\n",
    "for doc in elon_dataset:\n",
    "\n",
    "    visits_in_a_doc = list(set([visit['answer'] for visit in doc[\"visits\"] if visit != None]))\n",
    "    \n",
    "    if len(visits_in_a_doc)>0:\n",
    "        assert len(visits_in_a_doc)==1\n",
    "        \n",
    "        if visits_in_a_doc[0]!='[SEP]':\n",
    "            extracted_information.append({'title':doc['_source']['title'], 'visits':visits_in_a_doc})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "325a7d43-267e-41d7-9d5b-b9739f4a903a",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(extracted_information)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e526b3bf-513a-419f-8aaf-84f5855e4912",
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_to_titles = {}\n",
    "\n",
    "for doc in extracted_information:\n",
    "    for loc in doc['visits']:\n",
    "        if loc in nodes_to_titles.keys():\n",
    "            nodes_to_titles[loc].append(doc['title'])\n",
    "        else:\n",
    "            nodes_to_titles[loc] = [doc['title']]\n",
    "            \n",
    "nodes = set(nodes_to_titles.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52c2f595-65be-4fa9-afcc-b02de9175081",
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_nodes = [{'data':{'id':node,'label':node}}\n",
    "                for node in nodes]\n",
    "parsed_nodes.append({'data':{'id':'Elon Musk','label':'Elon Musk'}})\n",
    "parsed_nodes[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bacd0d38-9f36-4970-8f04-30e415a52064",
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_nodes.append({'data':{'id':'Elon Musk','label':'Elon Musk'}})\n",
    "parsed_nodes[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4799a267-1134-43a3-a56e-9fd2d932c931",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import math \n",
    "visualized_coordinates = []\n",
    "for i in range(len(parsed_nodes)):\n",
    "    angle = i*((2*math.pi)/len(parsed_nodes))\n",
    "    visualized_coordinates.append((math.cos(angle), math.sin(angle)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "409fdd69-f405-495a-95fd-132052c58894",
   "metadata": {},
   "outputs": [],
   "source": [
    "visualized_coordinates[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a55d6acf-dae7-40d8-9dc8-69999d515249",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(parsed_nodes)==len(visualized_coordinates)\n",
    "\n",
    "for i,n in enumerate(parsed_nodes):\n",
    "    parsed_nodes[i].update({'position':{'x':visualized_coordinates[i][0]*2000,'y':visualized_coordinates[i][1]*2000}})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3f01651-66c7-4d04-a8fb-38a5965bc46a",
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_nodes[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39bb29fc-2948-4b4a-8875-169db57df8a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_edges = [{'data':{'source':'Elon Musk', 'target':loc, 'label':title}} \n",
    "                for loc, titles in nodes_to_titles.items() \n",
    "                for title in titles]\n",
    "\n",
    "parsed_edges[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1ac72c7-f317-4d37-8a15-f4b7a6301fa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "extracted_information_str = json.dumps(parsed_nodes+parsed_edges)\n",
    "\n",
    "with open(\"elon_musk_results_to_visualize.json\",\"w\") as f:\n",
    "    f.write(extracted_information_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad0db9a6-b354-4c30-93b6-0306a8df766c",
   "metadata": {},
   "outputs": [],
   "source": [
    "for edge in parsed_edges[:3]:\n",
    "    print(edge)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}