Provided checkpoints produce high EER

[w-hc]

Hi thank you for providing the code and checkpoints.

I am running the code on the updated final data release downloaded from https://datashare.is.ed.ac.uk/handle/10283/3336

I modified the dataset feeding layer a little bit for the changed file name and formats. But overall the logic is kept as is.

python model_main.py --eval  --eval_output=./cm.txt --model_path=./sample_model/SPEC/spec_logical.pth

This checkpoint produces ERR at 43.01%

python model_main.py --eval  --eval_output=./cm.txt --model_path=./sample_model/SPEC/spec.pth

This checkpoint produces ERR at 49.05%

python model_main.py --eval  --eval_output=./cm.txt --model_path=./sample_model/SPEC/spect_finetune.pth

This checkpoint produces ERR at 56.51%

python model_main.py --eval --features mfcc  --eval_output=./cm.txt --model_path=./sample_model/MFCC/mfcc_logical.pth

This checkpoint fails to run.

RuntimeError: size mismatch, m1: [32 x 32], m2: [480 x 128] at /opt/conda/conda-bld/pytorch_1587428207430/work/aten/src/THC/generic/THCTensorMathBlas.cu:283

The dimensions of the output is 32x32 before fc1 and there is a shape mismatch.

I guess my question is are these checkpoints supposed to perform much better?
If so then it might be an error on my part and I'll doublecheck.

Thank you!

[malzantot]

Hi @w-hc ,

Thanks for your interest in our project.

Yeah, the. number you reported looks far from the expected result of model performance. In order, to help you find what is the cause of this mismatch: can you please let me know which dataset are you using for evaluation ? Please notice that there are separate models for each task (i.e. logical access and physical access attack scenarios). if you use a model which is trained for one task on the other, you will likely end with near 50% accuracy as is the case.

Also, please notice that the evaluation number that we have in our paper are according to the competitions formal evaluation results which was carried out by the competition organizers, not by us, but the result was also consistent with our own evaluation on the development dataset. We could-not run evaluation by ourselves on the eval dataset by ourselves because it was kept private by the competition organizers. The full set of formal evaluation results can be found here and our team entry name was (team name: uclanesl, team code=T016).

When we released the code, we provided. all code we used for model training and evaluation but didn't anticipate there will be a need for adding pre-trained models. Therefore, we didn't snapshot our best models or add them to the repo. But later as we got many requests to add checkpoints. My co-author @wangziqi000 has uploaded some sample checkpoints which are not necessary the best performing models. As clearly stated in his commit, the recommended approach is to train your own models using the code.

In response to your issue, we are currently investigating the performance of the available checkpoints and if needed, we will , if needed, train new models and provide new checkpoints or update the instructions on how they can be used and on which datasets.

Regarding the MFCC model, there is a mis-match between the code and the provided checkpoint that you can either fix by training a new model or by changing the number of neurons in layer fc1 of MFCC model. Again, we are currently doing investigation of the checkpoints and will be updating the repo soon. But please allow us sometime as we are no longer actively working on this project and no longer have access to the machines that had the models we trained during the competition.

After the competition ended, we decided to release code as open-source to help the community making progress in this field. It is also intended to be starter-framework for other researchers to build and evaluate their models by extending our code. As we have spent days to build the data loading, feature extraction, model training, and evaluation. We wanted to save others at a least a part of this overhead. Nevertheless, as an open-source project if there is any bugs or a potential enhancement, feel free to let us know by contacting us or by contributing to the project through adding a pull request which we will happily merge into the project.