Classwork - 2 TFIDF

[edaraa2]

TFIDF

• In information retrieval, the “term frequency – inverse document frequency” (also called TFIDF),is a well know method to evaluate how important is a word in adocument. TFIDF comes up a lot in research work because it’sboth a corpus exploration method and a pre-processing step formany other text-mining measures and models.

• To get TFIDF you need 4 data structures:

1. A list of all keywords in the corpus (K). This list is initially empty and growsare tweets are read and processed.
2. A one-dimensional list of all words in a certain document (W). Thisstructure is used for storing the words of a document as it is beingprocessed.
3. The two-dimensional sparse array (TF) for storing the TF and/or TFIDF:the number of rows is the number of documents, and the columnsrepresent the set of all words used in the corpus.
4. A one-dimentional list of document frequencies (DF), which is a parallellist of K and has the same size, it has the count of how many documentcontain that word.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB