Curious about the F1-score

[ljy19940805]

Hi, I just run the socring.py using dataset blogcatalog.mat according to readme. I found that the Macro-F1 is about 99% which is much higher than presented in readme. I'm curious about this. Is it the benefit of the new version of gensim?

[GTmac]

99% sounds suspicious -- what are the F1 scores you got with different ratio of training data (10%, 20%, etc.)? What version of sklearn are you using? Do you get the same result using the sklearn version as in requirements.txt?

[ljy19940805]

My sklearn version is 0.20.1 The outputs of the scoring.py are list below:
Train percent: 0.1
Shuffle #1: {'micro': 0.9970089730807578, 'macro': 0.9916461459510894}
Shuffle #2: {'micro': 0.9966273187183811, 'macro': 0.9907274953432105}
Shuffle #3: {'micro': 0.9964596321095974, 'macro': 0.9695754837021547}
Average score: {'micro': 0.9966986413029121, 'macro': 0.9839830416654848}

Train percent: 0.2
Shuffle #1: {'micro': 0.9972406656894024, 'macro': 0.9920512494705712}
Shuffle #2: {'micro': 0.9964615517390178, 'macro': 0.9875400204101393}
Shuffle #3: {'micro': 0.9971431044931175, 'macro': 0.9922137039235661}
Average score: {'micro': 0.9969484406405126, 'macro': 0.990601657934759}

Train percent: 0.30000000000000004
Shuffle #1: {'micro': 0.9969409907242944, 'macro': 0.9916935918529385}
Shuffle #2: {'micro': 0.9962635201573254, 'macro': 0.9913935399738298}
Shuffle #3: {'micro': 0.9970402525651144, 'macro': 0.9916241811476084}
Average score: {'micro': 0.9967482544822448, 'macro': 0.9915704376581256}

Train percent: 0.4
Shuffle #1: {'micro': 0.9966659002069441, 'macro': 0.9913201214257845}
Shuffle #2: {'micro': 0.996324373994946, 'macro': 0.9906089037661286}
Shuffle #3: {'micro': 0.9968879668049793, 'macro': 0.9911254998364157}
Average score: {'micro': 0.9966260803356232, 'macro': 0.9910181750094429}

Train percent: 0.5
Shuffle #1: {'micro': 0.9969575439081731, 'macro': 0.989652714060495}
Shuffle #2: {'micro': 0.996420704845815, 'macro': 0.9936082954918549}
Shuffle #3: {'micro': 0.9964132983859842, 'macro': 0.9919155403472311}
Average score: {'micro': 0.9965971823799907, 'macro': 0.9917255166331937}

Train percent: 0.6000000000000001
Shuffle #1: {'micro': 0.9970598408855067, 'macro': 0.9895625028975495}
Shuffle #2: {'micro': 0.9960440316477468, 'macro': 0.9936684600443851}
Shuffle #3: {'micro': 0.9968841959494548, 'macro': 0.991798614301317}
Average score: {'micro': 0.9966626894942361, 'macro': 0.9916765257477506}

Train percent: 0.7000000000000001
Shuffle #1: {'micro': 0.9969893469198703, 'macro': 0.990299789887283}
Shuffle #2: {'micro': 0.9952054794520548, 'macro': 0.9663837903244641}
Shuffle #3: {'micro': 0.9962937224924716, 'macro': 0.9858222894360839}
Average score: {'micro': 0.9961628496214656, 'macro': 0.9808352898826103}

Train percent: 0.8
Shuffle #1: {'micro': 0.9968553459119497, 'macro': 0.9903054138665885}
Shuffle #2: {'micro': 0.9969188634029442, 'macro': 0.9683922442652436}
Shuffle #3: {'micro': 0.9955063947459385, 'macro': 0.9855743755396478}
Average score: {'micro': 0.9964268680202775, 'macro': 0.9814240112238265}

Train percent: 0.9
Shuffle #1: {'micro': 0.997179125528914, 'macro': 0.9721175807333114}
Shuffle #2: {'micro': 0.9957716701902748, 'macro': 0.9598804271796423}
Shuffle #3: {'micro': 0.9958932238193019, 'macro': 0.9692336236702406}
Average score: {'micro': 0.9962813398461635, 'macro': 0.9670772105277314}

[GTmac]

This is most likely a bug -- what if you start from scratch, use gensim 3.1.0 and sklearn 0.19.1 and check the classification result again? I can reproduce the numbers in the README file using gensim and sklearn of the above version.

[ljy19940805]

Thanks for your advice. I have reproduce the result using gensim 3.1.0.