Home

K-means clustering algorithm on Spark system#

Introduction

One of the big challenges in the security field, today, comes from the sheer volume of data that security analysts have to deal with on a daily basis. Security companies like Symantec collect every day several millions of events related to malicious activities in the Internet, such as spam, phishing or malware attacks. The main objective of this project is to study the suitability and performance of a data-intensive scalable computing framework to enable Big Data analytics for security analysis applications.

The focus of this project is on Spark, which is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write. To run programs faster, Spark provides primitives for in-memory parallel computing: a data analysis job can load data into memory and query it repeatedly (and iteratively) much quicker than with disk-based systems like Hadoop MapReduce. To make programming faster, Spark provides clean, concise APIs in Scala, Java and Python. In addition, it is possible to use Spark interactively from the Scala console to rapidly query big datasets.

Spark was initially developed at UC Berkeley for two applications where keeping data in memory helps: iterative algorithms, which are common in machine learning, and interactive data mining. However, while Spark is a new engine, it can access any data source supported by Hadoop (like HDFS), making it easy to run over existing data

Project Goals

The underlying goal of this project is to use the Spark API to implement a k-means clustering algorithm for multi-feature classification of a SPAM email dataset. The implementation of k-means includes the proper definition of email features and the proper definition and implementation of distance metrics. After the implementation of the K-means algorithm an experimental study will be carried out on the BigFoot private cloud platform to dissect the performance achieved.

Project Steps

For the theory dealing with the algorithm, you can refer to Overview on algorithm
Various steps have been considered as intermediate targets:

1. Initially a standard Kmeans has been implemented for two-dimensional problem, in this case for points on plane. A first implementation in Scala using a functional paradigm has been done using functional programming patterns. Then an implementation using Spark Scala API has been defined and tested.