RecordLinkage.Rmd

---
title: "getPairs-Funktion im RecordLinkage-Package"
author: "Omar Besic, Thierry Schmidt, Chantal Zbinden"
date: "5/25/2019"
output:
  html_document:
    theme: united
    highlight: tango  
    toc: true
    toc_float:
      collapsed: false
---
<span style="color:blue">
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(RecordLinkage, quietly = TRUE)
```

### RecordLinkage package
Provides functions for linking and de-duplicating data sets. <br>
Methods based on a stochastic approach are implemented as well as classification algorithms <br>
from the machine learning domain.

###Data analysis
dataframe of 500 persons <br>
with firstname, second firstname, lastname, second lastname, birthyear, birthmonth and birthday

```{r dataframe}
data(RLdata500)
head(RLdata500)
```

### Building pairs on RLdata500 dataset
using compare.dedup to generate record pairs for deduplication or linkage. <br>
The summary shows 124'750 combinations based on the 500 datasets. <br>
There are no matching statuses because weight was not setted at this moment.

```{r generate Pairs Summary}
summary(compare.dedup(RLdata500))
```

blockfld aliviates the problem of n*(n-1)/2 and compares at least one of the field of 1,3,5,6,7 <br>
(compare columns 1, 3 or, 5 up to 7 for two records to  qualify to become a pair) <br>
with help of levenshteinSim formula we can compare the strings.
```{r comparededup}
rec.pairs <- compare.dedup(RLdata500
                           ,blockfld = list(1, 3, 5:7)
                           ,strcmp = c(2,3,4)
                           ,strcmpfun = levenshteinSim)
```

rec.pairs is list and there is dataframe of name pairs which is created in compare.dedup <br>
assign dataframe rec.pairs$pairs to matches 
```{r assign}
matches <- rec.pairs$pairs
```

give me the matches of all rows on which the restriction had be defined in blockfld <br>
for 1 = same | 0  = not same
```{r matches}

head(matches)


```

####Example 1 of pairs generated
show pair in row 2 (id1: 1, id2: 64) <br>
show patient in row 1 <br>
show patient in row 64
```{r example1}
matches[2,]
RLdata500[1,]
RLdata500[64,]
```

####Example 2 of pairs generated
show pair in row 13 (id1: 2, id2: 43) <br>
show patient in row 2 <br>
show patient in row 43
```{r example2}
matches[13,]
RLdata500[2,]
RLdata500[43,]
```


```{r compare}
#?compare.dedup()
rec.pairs <- compare.dedup(RLdata500,blockfld = list(1, 3, 5:7),strcmp = c(2,3,4),strcmpfun = levenshteinSim)
summary(rec.pairs)
```

### Calculate weights
Variant with epiWeight:"Calculates weights for record paris based on the EpiLink approach"
<li> Values from 0 to 1</li>

Values with emWeight:"Calculates weights for Record Linkage based on an EM algorithm"
<li> Values from -10 to 35</li>

#### epiWeight in example
Calculating for each record pair the weight based on the EpiLink approch
```{r weight}
rec.pairs <- epiWeights(rec.pairs)
length(rec.pairs$pairs$id1) #Anzahl Pairs
length(rec.pairs$Wdata) #Anzahl Weights
head(rec.pairs$Wdata) #Overview of data rparis - weights
```


#### Histogramm of epiWeight
The histogram shows how the weights are distributed.
```{r histo}
hist(rec.pairs$Wdata)
```

### getPairs function
Extracts record pairs from the previously created data list
It shows a pair and the corresponding weight within three rows.
The pairs can be filtered with:
<li>min.weight</li>
<li>max.weight</li>

```{r getPairs}
head(getPairs(rec.pairs, min.weight=0.5, max.weight=0.6)) #show all record pairs with weights between 0.5 and 0.6
tail(getPairs(rec.pairs, min.weight=0.5, max.weight=0.6))
rpairs <- getPairs(rec.pairs, single.rows=FALSE) #generate pairs of data.frame rec.pairs

```

### Classification

<div style="background-color: #eaeded ">
#### Theorie
To use the attribut <b>show=["links", "nonlinks", "possible""]</b> in the getPairs function, you need first to classify your dataset with epiClassify() or emClassify() depending your approach (epiWeight or emWeight)"

Two general approaches are implemented. The classical procedure by Fellegi and Sunter minimizes the number of possible links with given error levels for false links (my) and false non-links (ny).

The second approach requires threholds for links and possible links to be set by the user. A pair with weight w is classified as a link if w >= threshold.upper, as a possible link if threhold.upper >= w >= threshold.lower and as a non-link if w < threshold.lower.

If threshold.upper or threshold.lower is given, the threshold-based approach is used, otherwise, if one of the error bounds is given, the Fellegi-Sunter model. If only my is supplied, links are chosen to meet the error bound and all other pairs are classified as non-links (the equivalent case holds if only ny is specified). If no further arguments than rpairs are given, a single threshold of 0 is used.

The quality of classification of the Fellegi-Sunter method relies strongly on resonable estimations of m- and u-probabilities. The results should be evaluated critically.

####Threshold
<li>Threshold.upper --> Threshold value from when pairs become links</li>
<li>Threshold.lower --> Threshold value from when non-links become possible links</li></br>
If no threshold.lower is defined; treshold.lower = treshold.upper. In this case there are no possible</br>
</div>

#### emClassify
Classifies data pairs to which weights were assigned by emWeights. Based on user-defined threholds or predefined error rates

####epiClassify
Classifies data pairs to which weights were assigned by epiWeights. Based on user-defined threholds or predefined error rates

There are no possible, because no treshold.lower was defined
```{r classify1}
cldata <- epiClassify(rec.pairs, 0.5) #threshold.lower = threshold.upper if not defined
summary(cldata$prediction)
```


Now (with treshold.lower) there are some pairs which are classified as possible.
```{r classify2}
cldata <- epiClassify(rec.pairs, 0.6, 0.4)
summary(cldata$prediction)
```

Because we had classified the record pairs, we can use the "show" attribut in the getPairs() function.
```{r classify3}
head(getPairs(cldata, show="links", single.rows=FALSE))
```


###False matches and false non-matches
Now we can check which of the pairs are false-negative and which are false-positive. 
In other words, which of these pairs form synonyms (false-negative) or homonyms (false-positive)?

Therefore we use the RLBigDataDedup method, add weights with the epi aproach and weight them by 0.5 

```{r Synonym, warning=FALSE}
data(RLdata500)
testdata <- RLdata500
data <- RLBigDataDedup(testdata, identity = identity.RLdata500,
                       blockfld=list(1,3,5:7),
                       strcmp = c(2,3,4),
                       strcmpfun = "levenshtein")
data <- epiWeights(data)
result <- epiClassify(data, 0.5)
```

#### Synonym
Example: Martin and Maeddae: same person is meant, but the pair does not form a link.<br>
Give me all synonyms based on my weight of 0.5

```{r getFalseNeg}
 getFalseNeg(result)
```

#### Homonym
Example: Jan and Jan: not the same person, but the pair forms a link because string is the same.
```{r Homonym}
 getFalsePos(result)
```

### Fragestellung
<div style="background-color: #d2b4de">
"Dear Professor Sariyar: I am using your great tools to do matching. I got a quick question: In getPairs (matcher or links), we have filter.match = c("match", "unknown", "nonmatch"), filter.link = c("nonlink", "possible", "link"), do we have any ways to calculate theri confidence levels related? Thanks and best regards..."
</div>

### Threshold determination

#### 1. Clerical Review

```{r threshold cl}

```
#### 2. getParetoThreshold
Calculates a classification threshold based on a generalized Pareto distribution fitted to the weights distribution of the given data pairs
```{r threshold parteo}

```
#### 3. optimalThreshold
Calculates the optimal threshold for weight-based RecordLinkage
<li>my = error bound for false positives</li>
<li>ny = error bound for false negatives</li>

```{r threshold optimal}

```