Sparkdeduplication#429

Open

axnow wants to merge 21 commits into

CeON:masterfrom

axnow:sparkdeduplication

Commits on Mar 17, 2017

Initial version of spark-based document deduplication. It contains
axnow
committed

Commits on Apr 4, 2017

Work on the algorithm which splits large clusters among the cluster
axnow
committed

Commits on Apr 7, 2017

Complete version with tiled comparison task.
axnow
committed

Commits on Apr 14, 2017

Work on tiled optimalization.
axnow
committed

Commits on Jun 23, 2017

Stable version, does proper job within 2.5h on full data set.
axnow
committed

Commits on Jun 26, 2017

Added options parsing from command line to control app behaviour.
axnow
committed

Commits on Jul 11, 2017

Added dependency for the scopt.
axnow
committed

Commits on Jul 14, 2017

Task tiling class rewritten to scala, with tests.
axnow
committed

Commits on Jul 23, 2017

Scala version.
axnow
committed

Commits on Jul 24, 2017

Commits on Jul 25, 2017