Add GloVe

[leezu]

Description

This adds an implementation of GloVe. Unlike the original C implementation with asynchronous AdaGrad we used synchronous batched AdaGrad to make use of GPU. This also adds optimized C++ tools vocab_count and cooccur to construct the input for the train_glove.py script. Both tools make use of a (single-machine) map-reduce pattern to scale better than the single-threaded versions of https://github.com/stanfordnlp/GloVe .

Checklist

Essentials

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Add GloVe
• Scalable vocab_count and cooccur tools

Comments

• The C++ tools require a compiler with basic C++17 support. They require a few few sub-modules for compilation: 1) CNPY to write the output in numpy format. 2) range-v3 Implementation of Ranges-TS for C++ >= 11, 3) sparsepp a faster (less cache misses) and more memory efficient hashmap 4) CLI11 Command line interface. CMake is set up to download all dependencies automatically when preparing the compilation of the tools.

[mli]

Job PR-359/3 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/3/index.html

[codecov]

Codecov Report

Merging #359 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #359   +/-   ##
=======================================
  Coverage   84.25%   84.25%           
=======================================
  Files          94       94           
  Lines        8313     8313           
=======================================
  Hits         7004     7004           
  Misses       1309     1309

Flag	Coverage Δ
#PR359	83.19% <0%> (ø)	⬆️
#PR416	84.2% <0%> (ø)	⬆️
#master	83.19% <0%> (ø)	⬆️
#notserial	58.04% <0%> (ø)	⬆️
#py2	83.92% <0%> (ø)	⬆️
#py3	84.03% <0%> (ø)	⬆️
#serial	67.98% <0%> (ø)	⬆️

[mli]

Job PR-359/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/4/index.html

[mli]

Job PR-359/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/5/index.html

[mli]

Job PR-359/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/6/index.html

[eric-haibin-lin]

I think we need a readme.md or some rst file explaining how to use this training script (steps for compilation and cmd to run) and the expected result

[mli]

Job PR-359/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/7/index.html

[eric-haibin-lin]

Great work! I haven't gone through all the code in tools/cooccur.cc. Some comments so far

[leezu]

Will update this PR (rebase) to follow the structure adopted in #384 once merged.

[mli]

Job PR-359/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/8/index.html

[eric-haibin-lin]

Some minor comments

[mli]

Job PR-359/9 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/9/index.html

[mli]

Job PR-359/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/10/index.html

[mli]

Job PR-359/12 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-359/12/index.html