word2vec example

The word2vec example is an algorithm for computing continuous distributed representations of words. According to the word2vec repository it provides a provides an efficient implementation of the continuous bag-of-words and skip-gram architectures for computing vector representations of words. These representations can be subsequently used in many natural language processing applications and for further research.

The code is based on a the paper Distributed Representations of Words and Phrases and their Compositionality by Tomas Mikolov et al. and a detailed explanation is covered in the Word2Vec TF tutorial.

---

The installation is best done in a docker image or with a full bazel installation. In the docker image or main execute the following code listed below. The wget command will load the text8 corpus (30 MByte/100 MByte extracted) which starts with anarchism originated as a term of abuse. The file contains 17,005,207 words in 100,000,000 characters. The file questions-words.txt contains roughly 20,000 manually curated word relationships (ngrams and shingles) including capital-common-countries (Athens Greece Baghdad Iraq), capital-world (Abuja Nigeria Accra Ghana), currency (Algeria dinar Argentina peso), city-in-state, family, gram1-adjective-to-adverb, gram2-opposite, gram3-comparative, gram4-superlative (bad worst big biggest), gram5-present-participle, gram6-nationality-adjective, gram7-past-tense, gram8-plural, gram9-plural-verbs.

cd tensorflow
wget http://mattmahoney.net/dc/text8.zip -O text8.gz
gzip -d text8.gz -f
wget https://word2vec.googlecode.com/svn/trunk/questions-words.txt
bazel build -c opt tensorflow/models/embedding:all

which results in

root@fb729273837c:/tensorflow# bazel build -c opt tensorflow/models/embedding:all
INFO: Reading 'startup' options from /root/.bazelrc: --batch
INFO: Found 10 targets...
INFO: Elapsed time: 10.615s, Critical Path: 2.25s
```

After that we can start the example python file by using the manual command from the readme.

bazel-bin/tensorflow/models/embedding/word2vec_optimized
--train_data=text8
--eval_data=questions-words.txt
--save_path=/tmp/

which will then drizzle into

```
root@fb729273837c:/tensorflow# bazel-bin/tensorflow/models/embedding/word2vec_optimized \
>   --train_data=text8 \
>   --eval_data=questions-words.txt \
>   --save_path=/tmp/
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 8
I tensorflow/core/common_runtime/local_session.cc:45] Local session inter op parallelism threads: 8
I tensorflow/models/embedding/word2vec_kernels.cc:134] Data file: text8 contains 100000000 bytes, 17005207 words, 253854 unique words, 71290 unique frequent words.
Data file:  text8
Vocab size:  71290  + UNK
Words per epoch:  17005207
Eval analogy file:  questions-words.txt
Questions:  17827
Skipped:  1717
Epoch    1 Step   151319: lr = 0.023 words/sec =    30761
Eval 1535/17827 accuracy =  8.6%
Epoch    2 Step   302672: lr = 0.022 words/sec =   130612
Eval 2333/17827 accuracy = 13.1%
Epoch    3 Step   454007: lr = 0.020 words/sec =     4570
Eval 3057/17827 accuracy = 17.1%
Epoch    4 Step   605354: lr = 0.018 words/sec =   107753
Eval 3628/17827 accuracy = 20.4%
Epoch    5 Step   756681: lr = 0.017 words/sec =    82295
Eval 4003/17827 accuracy = 22.5%
Epoch    6 Step   907984: lr = 0.015 words/sec =    13700
Eval 4489/17827 accuracy = 25.2%
Epoch    7 Step  1059339: lr = 0.013 words/sec =    41209
Eval 4650/17827 accuracy = 26.1%
Epoch    8 Step  1210660: lr = 0.012 words/sec =    55382
Eval 4921/17827 accuracy = 27.6%
Epoch    9 Step  1361993: lr = 0.010 words/sec =    30143
Eval 5267/17827 accuracy = 29.5%
Epoch   10 Step  1513356: lr = 0.008 words/sec =    96452
Eval 5467/17827 accuracy = 30.7%
Epoch   11 Step  1664704: lr = 0.007 words/sec =    50088
Eval 5669/17827 accuracy = 31.8%
Epoch   12 Step  1816013: lr = 0.005 words/sec =    27675
Eval 5862/17827 accuracy = 32.9%
Epoch   13 Step  1967318: lr = 0.003 words/sec =   114565
Eval 6022/17827 accuracy = 33.8%
Epoch   14 Step  2118684: lr = 0.002 words/sec =    46359
Eval 6160/17827 accuracy = 34.6%
Epoch   15 Step  2270008: lr = 0.000 words/sec =   100920
Eval 6234/17827 accuracy = 35.0%
root@fb729273837c:/tensorflow# 

```



### Links

* [text8](http://mattmahoney.net/dc/textdata) - text8 corpus by Matt Mahoney
* [word2vec](https://code.google.com/p/word2vec/) - computing continuous distributed representations of words
* [word2vec@chalow](http://chalow.net/2014-05-21-1.html) - 手持ちの MacBook Air (OS X 10.9.2) で word2vec を動かしてみる
* [word2vec@cnblogs](http://www.cnblogs.com/wowarsenal/p/3293586.html) - 用中文把玩Google开源的Deep-Learning项目word2vec
* [Word2Vec&GloVe](http://textminingonline.com/tag/word2vec-in-python) - Getting Started with Word2Vec and GloVe in Python
* [Books&ngrams](https://books.google.com/ngrams) - Google Books ngram viewer