GettingStarted

Getting Started #

Requirements

Spark 0.7.0
Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.

sbt 0.11.3
A build tool for Scala and Java projects. It requires Java 1.6 or later.

Scala 2.9.2
Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way.

Usage

Run runKmeans.sh with parameters:

• path to SPARK root
• 0: to only run, 1: to package in jar and run
• sparkConf file
• parameters File

E.g.

./runKmeans.sh /user/spark/spark-0.7.0 0 spark.conf parameter.conf 

The output is in direcorty out with result Centroids and log file of job.

Configuration files

spark.conf

• host: master URL passed to Spark

Master URL	Meaning
local	Run Spark locally with one worker thread (i.e. no parallelism at all)
local[K]	Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine)
spark://HOST:PORT	Connect to the given Spark standalone cluster master

• appName: name of job
• inputfile: a path to input json dataset, either a local one on the machine or a hdfs://

This is an example:

{
	"host": "local",
	"appName": "SparkKmeans",
	"inputFile": "pathToInput"
}

parameter.conf

• InitialCentroids: number of initial Centroids, you can set to sqrt(inputSize/2)
• ConvergeDist: Stopping condition: old and new centroids between iterations don't change more than this value.
• numSamplesForMedoid: num of Elements in a cluster that will be used to update centroid. TODO
• weights: weights for features, values between 0 and 1. Sum of weights must be 1.

This is an example:

{
	"initialCentroids" : 100,
	"convergeDist" : 0.001,
	"numSamplesForMedoid" : 3,
	"weights" : {
		"space" : 0.20, 
		"time" : 0.20,
		"IP" : 0.20, 
		"uri" : 0.20, 
		"botname" : 0.20 
	}
}