Home

K-means clustering algorithm on Spark system#

Introduction

One of the big challenges in the security field, today, comes from the sheer volume of data that security analysts have to deal with on a daily basis. Security companies like Symantec collect every day several millions of events related to malicious activities in the Internet, such as spam, phishing or malware attacks. The main objective of this project is to study the suitability and performance of a data-intensive scalable computing framework to enable Big Data analytics for security analysis applications.

The focus of this project is on Spark, which is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write. To run programs faster, Spark provides primitives for in-memory parallel computing: a data analysis job can load data into memory and query it repeatedly (and iteratively) much quicker than with disk-based systems like Hadoop MapReduce. To make programming faster, Spark provides clean, concise APIs in Scala, Java and Python. In addition, it is possible to use Spark interactively from the Scala console to rapidly query big datasets.

Spark was initially developed at UC Berkeley for two applications where keeping data in memory helps: iterative algorithms, which are common in machine learning, and interactive data mining. However, while Spark is a new engine, it can access any data source supported by Hadoop (like HDFS), making it easy to run over existing data

Project Goals

The underlying goal of this project is to use the Spark API to implement a k-means clustering algorithm for multi-feature classification of a SPAM email dataset. The implementation of k-means includes the proper definition of email features and the proper definition and implementation of distance metrics. After the implementation of the K-means algorithm an experimental study will be carried out on the BigFoot private cloud platform to dissect the performance achieved.

Project Steps

For the theory dealing with the algorithm, you can refer to Overview on algorithm
Various steps have been considered as intermediate targets:

1. Implementation of a standard Kmeans for two-dimensional problem in Scala.
2. Implementation using Spark Scala API has been defined and tested for Kmeans for points on plane.
3. Study Spam Dataset identifying possible types for Features (Numeric and Categorical has been considered)
4. Definition of Algorithm that use KMeans and KMedoids structures to manage different types of features. (Improvement for Categorical types can be KModes)
5. Design and Implementation of the Algorithm
6. Evaluation with different parameters

References and Links

Spam Clustering Using Wave Oriented
Layout Based Spam Filtering
Spam Detection using Clustering, Random Forests, and Active Learning
Internet Geolocation and Evasion
An Empirical Study of Clustering Behavior of Spammers and Group-based Anti-Spam Strategies
Clustering Spam domains and hosts: anti-spam forensics with data mining String metric library