MPQA.ipynb

{
 "metadata": {
  "name": "MPQA"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h1>Subjectivity Lexicon</h1>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>General Information:</h3>\n",
      "Made available under the terms of GNU General Public License. They are distributed without any warranty.\n",
      "\n",
      "The Subjectivity Lexicon (list of subjectivity clues) that is part of OpinionFinder is also available for separate download. These clues were compiled from several sources (see the enclosed README). This is the version of the lexicon used in:\n",
      "\n",
      "Theresa Wilson, Janyce Wiebe, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proc. of HLT-EMNLP-2005.\n",
      "\n",
      "<h3>Download:</h3>\n",
      "http://mpqa.cs.pitt.edu/lexicons/subj_lexicon/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Example:</h3>\n",
      "<table class=\"narrowcol\">\n",
      "      <tbody><tr class=\"hdr\"><th></th>  <th>Strength</th>     <th>Length</th> <th>Word</th> <th>Part-of-speech</th><th>Stemmed</th><th>Polarity</th></tr>\n",
      "      <tr><td>1.</td><td>type=weaksubj</td><td>len=1</td><td>word1=abandoned</td><td>pos1=adj</td><td>stemmed1=n</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr class=\"even\"><td>2.</td><td>type=weaksubj</td><td>len=1</td><td>word1=abandonment</td><td>pos1=noun</td><td>stemmed1=n</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr><td>3.</td><td>type=weaksubj</td><td>len=1</td><td>word1=abandon</td><td>pos1=verb</td><td>stemmed1=y</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr class=\"even\"><td>4.</td><td>type=strongsubj</td><td>len=1</td><td>word1=abase</td><td>pos1=verb</td><td>stemmed1=y</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr><td>5.</td><td>type=strongsubj</td><td>len=1</td><td>word1=abasement</td><td>pos1=anypos</td><td>stemmed1=y</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr class=\"even\"><td>6.</td><td>type=strongsubj</td><td>len=1</td><td>word1=abash</td><td>pos1=verb</td><td>stemmed1=y</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr><td>7.</td><td>type=weaksubj</td><td>len=1</td><td>word1=abate</td><td>pos1=verb</td><td>stemmed1=y</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr class=\"even\"><td>8.</td><td>type=weaksubj</td><td>len=1</td><td>word1=abdicate</td><td>pos1=verb</td><td>stemmed1=y</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr><td>9.</td><td>type=strongsubj</td><td>len=1</td><td>word1=aberration</td><td>pos1=adj</td><td>stemmed1=n</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr class=\"even\"><td>10.</td><td>type=strongsubj</td><td>len=1</td><td>word1=aberration</td><td>pos1=noun</td><td>stemmed1=n</td><td>priorpolarity=negative</td></tr>\n",
      "      <tr><td colspan=\"7\">...</td></tr>\n",
      "      <tr class=\"even\"><td>8221.</td><td>type=strongsubj</td><td>len=1</td><td>word1=zest</td><td>pos1=noun</td><td>stemmed1=n</td><td>priorpolarity=positive</td></tr>\n",
      "    </tbody></table>\n",
      "http://sentiment.christopherpotts.net/lexicons.html"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h2>Code</h2>\n",
      "<h3>Load MPQA</h3>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import sys, nltk"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Load the MPQA file\n",
      "fswn=open('/Users/robwilliams/Dropbox/Capstone/Final/SentimentLibraries/MPQA/subjclueslen1-HLTEMNLP05.tff')\n",
      "mpqaLines = [line for line in fswn.readlines()]\n",
      "fswn.close"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "<function close>"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#ArrayList that will hold all dictionaries of word, POS & rating\n",
      "mpqaArrList = []\n",
      "\n",
      "#Cycle through each of the physical lines of the MPQA File\n",
      "for line in mpqaLines:\n",
      "    #Split on the space to create the Key=Value with the = sign\n",
      "    data = line.strip().split(\" \") #Result Ex: len=1\n",
      "    \n",
      "    #Initialize Dictionary for each line that will hold the key=value\n",
      "    word_dict = {}\n",
      "    #Split on the = to get the Key and Value split up\n",
      "    for d in data:\n",
      "        try:\n",
      "            keyValue = d.strip().split(\"=\")\n",
      "            key = keyValue[0]\n",
      "            value = keyValue[1]\n",
      "            #Store the key value pair into a dictionary\n",
      "            word_dict[key] = value\n",
      "        except IndexError:\n",
      "            print \"Error\"\n",
      "            \n",
      "    #Add To The MPQA ArrayList\n",
      "    mpqaArrList.append(word_dict)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print \"MPQA Finished Loading\"\n",
      "print \"Total Entries: \" + str(len(mpqaArrList))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "MPQA Finished Loading\n",
        "Total Entries: 8225\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "9/21/13 - Added the following words. Suck = Neg, Sucking = Neg, Loving = Pos"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Load CMU Parser</h3>\n",
      "<ul>\n",
      "    <li>N - Common Noun</li>\n",
      "    <li>O - Pronoun</li>\n",
      "    <li>S - Nominal + Posessive</li>\n",
      "    <li>^ - Proper Noun</li>\n",
      "    <li>Z - Proper Noun + Posessive</li>\n",
      "    <li>L - Nominal + Verbal</li>\n",
      "    <li>M - Proper Noun + Verbal</li>\n",
      "    <li>V - Verb</li>\n",
      "    <li>A - Adjective</li>\n",
      "    <li>R - Adverb</li>\n",
      "    <li>! - Interjection</li>\n",
      "    <li>D - Determiner</li>\n",
      "    <li>P - Pre or postposition</li>\n",
      "    <li>& - Coordinating Conjuntion</li>\n",
      "    <li>T - Verb Particle</li>\n",
      "    <li>X - Existential there, predeterminers</li>\n",
      "    <li>Y - X + Verbal</li>\n",
      "    <li># - Hashtag</li>\n",
      "    <li>@ - User Mention</li>\n",
      "    <li>~ - Discourse Marker (RT)</li>\n",
      "    <li>U - URL</li>\n",
      "    <li>E - Emoticon</li>\n",
      "    <li>$ - Numeral</li>\n",
      "    <li>, - Punctuation</li>\n",
      "    <li>G - Other abbreviations, garbage</li>\n",
      "</ul>\n",
      "http://www.ark.cs.cmu.edu/TweetNLP/gimpel+etal.acl11.pdf"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Append the path to the CMU Tweet Python Wrapper\n",
      "sys.path.append('/Users/robwilliams/Dropbox/Capstone/CMU')\n",
      "\n",
      "#Import the CMU Tweet Tagger Wrapper\n",
      "import CMUTweetTagger"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 20
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Load vocabulary\n",
      "def load_vocab(source=\"all\"):\n",
      "    vocab = set([])\n",
      "    if source in [\"nltk_words\", \"all\"]:\n",
      "        vocab = vocab | set(w.lower() for w in nltk.corpus.words.words())\n",
      "    if source in [\"wordsEn\", \"all\"]:\n",
      "        #source: http://www-01.sil.org/linguistics/wordlists/english/wordlist/wordsEn.txt\n",
      "        input_file = '/Users/robwilliams/Dropbox/Capstone/CMU/wordsEn.txt'\n",
      "        fd = open(input_file)\n",
      "        vocab = vocab | set(str.strip(w.lower()) for w in fd)\n",
      "        fd.close()\n",
      "    return vocab\n",
      "\n",
      "vocab = load_vocab()\n",
      "print \"vocab size: {}\".format(len(vocab))\n",
      "print 'done'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "vocab size: 287704\n",
        "done\n"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Test Sentences</h3>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "testSen1 = \"I really hate waking up early on Mondays\"\n",
      "testSen2 = \"The Bills really suck this season\"\n",
      "testSen3 = \"I'm loving this new iPhone\"\n",
      "testSen4 = \"my cats face is very funny looking\"\n",
      "testSen5 = \"I love cherrios\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 22
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Check Sentiment Function</h3>\n",
      "Takes in the word to be checked and the parts of speech. Checks the word to see if it exists in the MPQA wordbase"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#NOTE: currently not doing anything with POS, I want to do something incase the tempArray length is > 1 \n",
      "#which means there's more than 1 result\n",
      "\n",
      "def checkWordSentiment(word, pos):\n",
      "    tempArray = []\n",
      "    #Check to see if the word provided is located in the MPQA & return any matches (could be multiple)\n",
      "    for dict in mpqaArrList:\n",
      "        #If a match is found\n",
      "        if dict['word1'].upper() == word.upper():\n",
      "            #add it to a temp array list (as there may be more than one match)\n",
      "            tempArray.append(dict)\n",
      "    #when done checking, return all matches\n",
      "    return tempArray\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Print Results Function</h3>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Takes in an Array of the tweet which has been seperated at the spaces for each word\n",
      "def printResults(splitTweet):\n",
      "    #Iterate through each of the individual words in the sentence\n",
      "    for posWord in splitTweet[0]:\n",
      "        #Check the individual word against the CheckSentiment function to xRef the word if it exists in MPQA\n",
      "        word = posWord[0]\n",
      "        pos = posWord[1]\n",
      "        \n",
      "        result = checkWordSentiment(word, pos)\n",
      "        #If the word is actually listed grab the details\n",
      "        if len(result) != 0:\n",
      "            print result"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Pre-Processing</h3>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def preProcessing(sentence):\n",
      "    CMUResult = CMUTweetTagger.runtagger_parse([sentence])\n",
      "\n",
      "    #return sentence.strip().split(' ')\n",
      "    return CMUResult"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<h3>Testing</h3>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Pre Process a bunch of the test sentences created earlier\n",
      "t1 = preProcessing(testSen1)\n",
      "t2 = preProcessing(testSen2)\n",
      "t3 = preProcessing(testSen3)\n",
      "t4 = preProcessing(testSen4)\n",
      "t5 = preProcessing(testSen5)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print \"Sentence: \" \n",
      "print testSen1\n",
      "print \"\"\n",
      "print \"CMU POS:\"\n",
      "print t1\n",
      "print \"\"\n",
      "printResults(t1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Sentence: \n",
        "I really hate waking up early on Mondays\n",
        "\n",
        "CMU POS:\n",
        "[[('I', 'O', 0.9962), ('really', 'R', 0.9885), ('hate', 'V', 0.999), ('waking', 'V', 0.9997), ('up', 'T', 0.8815), ('early', 'A', 0.6792), ('on', 'P', 0.9958), ('Mondays', '^', 0.7839)]]\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print \"Sentence: \" \n",
      "print testSen2\n",
      "print \"\"\n",
      "print \"CMU POS:\"\n",
      "print t2\n",
      "print \"\"\n",
      "printResults(t2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Sentence: \n",
        "The Bills really suck this season\n",
        "\n",
        "CMU POS:\n",
        "[[('The', 'D', 0.9963), ('Bills', 'N', 0.9549), ('really', 'R', 0.9986), ('suck', 'V', 0.9987), ('this', 'D', 0.9772), ('season', 'N', 0.9979)]]\n",
        "\n",
        "[{'stemmed1': 'n', 'len': '1', 'word1': 'really', 'priorpolarity': 'neutral', 'type': 'strongsubj', 'pos1': 'anypos'}]\n",
        "[{'stemmed1': 'n', 'len': '1', 'word1': 'suck', 'priorpolarity': 'negative', 'type': 'strongsubj', 'pos1': 'verb'}]"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#Individually Check A Word\n",
      "print checkWordSentiment(\"love\", \"n\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[{'stemmed1': 'n', 'len': '1', 'word1': 'love', 'priorpolarity': 'positive', 'type': 'strongsubj', 'pos1': 'noun'}, {'stemmed1': 'y', 'len': '1', 'word1': 'love', 'priorpolarity': 'positive', 'type': 'strongsubj', 'pos1': 'verb'}]\n"
       ]
      }
     ],
     "prompt_number": 30
    }
   ],
   "metadata": {}
  }
 ]
}