clef-2018/raw_corpus_stat_NV.txt at master · ddenz/clef-2018 · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
--- FR raw dataset ---
training: 65843 docs
dev: 27850 docs
test: 31682 docs

training: avg number of codes per doc (including 'no code'): 4.025591179016752
dev: avg number of codes per doc (including 'no code'): 3.9740394973070017
test: avg number of codes per doc (including 'no code'): 4.164004797676914

unique codes in training: 3302
unique codes in dev: 2407
unique codes in test: 2579

unique unseen codes in dev: 229
unique unseen codes in test: 351

dev: avg number of unseen codes per doc: 1.043010752688172
test: avg number of unseen codes per doc: 1.0337387210670852


--- IT raw dataset ---
training: 14501 docs

training: avg number of codes per doc: 4.139507620164126

unique codes in training: 1442


--- Internal dataset division ---
IT: DOC division in 33%, 33%, 33% (use random function and check if statistics are still harmonized)
FR: count DOC per year, and take either 2 year with 5K each, or 1 year with 10K (then divide by two)

### IT raw data ###

training: 4835 docs
dev: 4833 docs
test: 4833 docs

training  - avg number of codes per doc (including 'no code'): 4.11664943123061
dev  - avg number of codes per doc (including 'no code'): 4.1578729567556385
test  - avg number of codes per doc (including 'no code'): 4.144009931719429

unique codes in training: 966
unique codes in dev: 972
unique codes in test: 1013

unseen codes in dev: 274
unseen codes in test: 308

dev: avg number of unseen codes per doc: 1.1282798833819243
test: avg number of unseen codes per doc: 1.138586956521739

### FR raw data ###

training (2006,2007,2008,2011,2012): 56544
dev (half 2009): 4649
test (half 2009): 4650

   YearCoded  counts
0       2006       3
1       2007    1370
2       2008    4603
3       2009    9299
4       2011   24146
5       2012   26422

training  - avg number of codes per doc (including 'no code'): 4.027483022071308
dev   - avg number of codes per doc (including 'no code'): 4.00408690040869
test  - avg number of codes per doc (including 'no code'): 4.024086021505377

unique codes in training: 3116
unique codes in dev: 1450
unique codes in test: 1457

unseen codes in dev: 89
unseen codes in test: 105

dev: avg number of unseen codes per doc: 1.0303030303030303
test: avg number of unseen codes per doc: 1.0442477876106195


--- scripts ---
FOLDER: T:\Natalia Viani\clef2018\scripts

read_data_brutes_FR.py (run on training, dev, test):
- read Brutes CSV file
- save distinct doc ids in external pickle

read_data_calculees_FR.py (run on training):
- read Calculees CSV file and check if documents are the same listed in the Brutes CSV file (remove additional lines)
- count lines with NO ICD code (optional)
- find unique ICD codes and save them in external pickle
- count distinct ICD codes (including the "no code" label) per document
- compute average number of distinct ICD codes per document

count_unseen_codes_FR.py (run on dev, test):
- read Calculees CSV file and check if documents are the same listed in the Brutes CSV file (remove additional lines)
- count distinct ICD codes (including the "no code" label) per document
- compute average number of distinct ICD codes per document
- count unique unseen codes
- compute average number of distinct unseen ICD codes per document

split_data_calculees_FR.py (run on training):
- read Calculees CSV file
- search for year with around 10K documents
- split these documents in 50% and 50% to create dev/test
- prepare output pickle with: DocID, SetName, YearCoded