Features

Mail Features

To generalize the clustering algorithm we need to define a function
f : Email -> Point
that maps email on a point. PM this is a point on a multi-dimensional space.

So we must first determine each email's representation in the feature space, and then compute the distance between the resulting points.

According to current algorithm we have to define semantic for

• closestCentroid
• + operator
• / operator
• distance

Features categories

According to mean computation:

• Feature Space n:
  numeric features on which you can compute Average
  metrics: Euclidean, Manhattan
  use k-means

• Feature Space m: string features without a computable "mean"
  metrics: levenshtein, edit distance use k-medoids

According to approach:

• content and text clustering
• ip and networks topology of senders
• urls and domain names
• sending data: volume of sending, forwarding, replying, type and quantities of attachments
• layout string also for MIME part string

Text clustering Kmeans

Input vector

• boolean vector
• frequency vector
• tf/idf vector
• reweighted tf/idf vector

Similarity measure

• cosine
• jaccard
• dice coefficient
• simple matching coefficient
• pearson correlation coefficient
• avg kullback leibler divergeance

Tokenization

• convert char to lower case
• separate on symbolic chars or space
• prefix tokens with header Tokens must be at least in 1% and upon to 99 %

Ip based clustering

Problem: spammers steal Ip addresses on the same local network or use botnet. URLs are virtually untraceable, there are also services, like tinyurl.com, that generate hash url that are alias. So the sending source is difficult to identify, but we can recognize some sending patterns.

• time series: inter-arrival, trasnsforms, burst email
• geographic data: Ip address, country, connection type, real IP address of the spam source, the route information, the TCP SYN fingerprint, which can be used to identify the OS information of the spam source.

Note:
The retrieval of hosting IP requires three steps: extracting URLs from emails; extracting the domain name portion of the URL; fetching the IP address of the domain name.

Layout structure

We have focused on two ways to characterize an email’s layout. The first uses directly quantifiable properties such as its size, the total number of new lines, blank lines, links, addresses or parts. The set of values can be regarded as a feature vector that represents the email’s position in the n- dimensional feature space Hn So the text structure of the body of a message like: “Hi, read this: \n http://www.a.com \n a@b.com \n\n\n” could be represented by the string "1 1 3 4 0 0 0" – considering 1 – short line, 3 – link, 4- email address, 0 – blank line. The MIME part structure is treated in the same manner, only with different items and indexes.

Mail features

• senders'ID address
• senders'mail
• Received date
• Time stamp
• subject
• length
• content
• word count
• embedded URLs
• attachments filenames
• attachment size

Spam features

Spam is a mail that has same sender and same content, we have typical patterns:

• company logo
• links to site
• from is good one, but reply is different

Pre-processing

• normalization: 0,1 values
• standardization: mean=0 and variance=1, data follows Gaussian Law?
• outliers management