segment-f-measure/README at master · rosannamilner/segment-f-measure · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
##########################################
# CONTACT
##########################################

Rosanna Milner, Thomas Hain @ SPandH, University of Sheffield
https://mini.dcs.shef.ac.uk/resources/speaker-diarisation-evaluation/
[ICASSP 2016] "Segment-oriented evaluation of Speaker Diarisation performance"

@inproceedings{Milner2016icassp,
  author    = {Rosanna Milner and
               Thomas Hain},
  title     = {Segment-oriented evaluation of speaker diarisation performance},
  booktitle = {{ICASSP}},
  pages     = {5460--5464},
  publisher = {{IEEE}},
  year      = {2016}
}

##########################################
# USAGE
##########################################

This script takes a reference RTTM file (or list of RTTM files) and a hypothesis RTTM file (or a list of RTTM files) and calculates the segment F-measure for speaker diarisation Evaluation.

For using list examples and collar 0.1:

$ ./segmentfmeasure_v1.0.py ./rttm/list.ref.rttm ./rttm/list.sys.rttm -c 0.1 --list

For using concatenated examples (or single files) and collar 0.1:

$ ./segmentfmeasure_v1.0.py ./rttm/example1+example2.ref.rttm ./rttm/example1+example2.sys.rttm -c 0.1


##########################################
# HELP (./segmentfmeasure_v1.0.py -h)
##########################################

usage: segmentfmeasure_v1.0.py [-h] [-u UEM] [-g GAP] [-c COLLAR]
                               [-d {uniform,triangular,Gaussian}]
                               [-t THRESHOLD] [-p PADDING] [-cs COLLAR_SCALE]
                               [-f FOLDER] [--sad] [-m] [--list] [--save]
                               ref hyp

positional arguments:
  ref                   Reference RTTM file (or list with flag --list)
  hyp                   Hypothesis RTTM file (or list with flag --list)

optional arguments:
  -h, --help            show this help message and exit
  -u UEM, --uem UEM     UEM file (can be single file or list)
  -g GAP, --gap GAP     Smoothing gap (seconds)
  -c COLLAR, --collar COLLAR
                        Collar around reference boundaries (+/- seconds)
  -d {uniform,triangular,Gaussian}, --distribution {uniform,triangular,Gaussian}
                        Distribution around reference boundaries
  -t THRESHOLD, --threshold THRESHOLD
                        Threshold for segment match/no match decision (not
                        uniform distribution)
  -p PADDING, --padding PADDING
                        Padding around hypothesis boundary
  -cs COLLAR_SCALE, --collar-scale COLLAR_SCALE
                        Scale to multiply collar if boundary type NONSPEECH
  -f FOLDER, --folder FOLDER
                        Folder in which to save smoothed RTTM files
  --sad                 Score Speech Activity Detection only - ignore speaker
                        labels
  -m, --spkr-map        Display speaker mapping information
  --list                REF and HYP are lists of RTTMs
  --save                Save smoothed RTTM files


##########################################
# OUPUT
##########################################

SE:	  speaker error (sum of impure reference speakers and missed reference speakers
SSE:	speaker segment error (hypothesis segments assigned to the incorrect speaker)
MAT:	reference-hypothesis segment matches
INS:	hypothesis segment insertions (unmatched to a reference segment)
DEL:	reference segment deletions (unmatched to a hypothesis segment)
PRC:	segment precision
RCL:	segment recall
F:	  segment F-measure

The overall score for SE is weighted by the number of reference speakers, SSE is weighted by the number of matched segments used for the speaker mapping stage, and the rest are weighted by the number of reference segments. For example:

$ ./segmentfmeasure_v1.3.py rttm/example1.ref.rttm ./rttm/example1.sys.rttm -u uem/example1.uem -c 0.1
--------------------------------
FILE:	example1
EVAL TIME:	50.00 TO 250.00
REF SEGMENTS:	72
HYP SEGMENTS:	110
REF SEGMENTS (SMOOTHED GAP 0.25):	60
COLLAR:	 0.10
--------
REF SPEAKERS:			4
HYP CLUSTERS:			4
BOUNDARY MATCHED SEGMENTS:	36
MAPPED SPEAKER-CLUSTER PAIRS:	4
UNASSIGNED SPEAKERS:		0
UNASSIGNED CLUSTERS:		0
PURE SPEAKER-CLUSTER PAIRS:	2
IMPURE SPEAKER-CLUSTER PAIRS:	2
SPEAKER ERROR (SE):		50.00 % ( (2 + 0) / 4 )
SPEAKER SEGMENT ERROR (SSE):	5.56 % ( 2 / 36 )
--------
SE:	  50.0 %
SSE:	5.6 %
MAT:	56.7 %
INS:	95.0 %
DEL:	43.3 %
PRC:	37.4 %
RCL:	56.7 %
F:	  45.0 %