Conditional Multimodal Biometrics Embedding Learning For Periocular and Face in the Wild (ICPR 2022)
On the Representation Learning of Conditional Biometrics for Flexible Deployment (IEEE Access 2023)
- Environment: Check
requirements.txt
file which was generated usingpip list --format=freeze > requirements.txt
command for the environment requirement. These files are slightly filtered manually, so there may be redundant packages. - Dataset: Download dataset (training and testing) from this link. Password is conditional_biometrics.
Ensure that datasets are located in
data
directory. Configuredatasets_config.py
file to point to this data directory by changing main path. - Pre-trained models: (Optional) The pre-trained MobileFaceNet model for fine-tuning or testing can be downloaded from this link.
- Change hyperparameters accordingly in
params.py
file. The set values used are the default, but it is possible to alternatively change them when running the python file. - Run
python training/main.py
. The training should start immediately. - Testing will be performed automatically after training is done, but it is possible to perform testing on an already trained model (see next section).
- Based on the (pre-)trained models in the
models(/pretrained)
directory, load the correct model and the architecture (innetwork
directory) usingload_model.py
file. Change the file accordingly in case of different layer names, etc. - Evaluation:
- Identification / Cumulative Matching Characteristic (CMC) curve: Run
cmc_eval_identification.py
. Based on the generated.pt
files indata
directory, runplot_cmc_roc_sota.ipynb
to generate CMC graph. - Verification / Receiver Operating Characteristic (ROC) curve: Run
roc_eval_verification.py
. Based on the generated.pt
files indata
directory, runplot_cmc_roc_sota.ipynb
to generate ROC graph.
- Identification / Cumulative Matching Characteristic (CMC) curve: Run
- Visualization;
- Decidability Index: Run
decidability_index.ipynb
to get the histogram distribution. The decidability index$d'$ will be displayed on the graph. - Pearson Correlation: Run
pearson_correlation.ipynb
to get the correlation curve between periocular and face. The correlation coefficient will be displayed on the graph.
- Decidability Index: Run
Method | Rank-1 IR (%) (Periocular) |
Rank-1 EER (%) (Periocular) |
Cross-Modal IR (%) (Periocular Gallery) |
Cross-Modal EER (%) (Periocular-Face) |
---|---|---|---|---|
PF-GLSR |
79.03 | 15.56 | - | - |
86.96 | 9.62 | 77.26 | 9.80 |
├── configs: Contains configuration files and hyperparameters to run the codes │ ├── datasets_config.py - Contains directory path for dataset files. Change 'main' in 'main_path' dictionary to point to dataset, e.g.,/home/cb_net/data
(without slash). │ └── params.py - Hyperparameters and arguments for training. ├── data: Directory for dataset preprocessing, and folder to insert data based onconfig.py
files. │ ├── [INSERT DATASET HERE.] │ ├── The.pt
files to plot the CMC and ROC graphs will be generated in this directory. │ └── data_loader.py - Generate training and testing PyTorch dataloader. Adjust the augmentations etc. in this file. Batch size of data is also determined here, based on the values set inparams.py
. ├── eval: Evaluation metrics - Identification and Verification, also contains.ipynb
files to plot CMC and ROC graphs. │ ├── cmc_eval_identification.py - Evaluates Rank-1 Identification Rate (IR) and generates Cumulative Matching Characteristic (CMC) curve, which are saved as.pt
files indata
directory. Use these.pt
files to generate CMC curves. │ ├── decidability_index.ipynb - Notebook to plot Decidability Index (d') histogram. │ ├── identification.py - Rank-1 Identification Rate (IR) evaluation. │ ├── lbp_extract.py - Local Binary Pattern (LBP) feature extraction and identification calculation. │ ├── pearson_correlation.ipynb - Notebook to plot Pearson Correlation (ρ) graph. │ ├── plot_cmc_roc.ipynb - Notebook to plot CMC and ROC side-by-side simulatenously. │ └── roc_eval_verification.py - Evaluates Verification Equal Error Rate (EER) and generates Receiver Operating Characteristic (ROC) curve, which are saved as.pt
files indata
directory. Use these.pt
files to generate ROC curves. ├── graphs: Directory where graphs and visualization evaluations are generated. │ └── CMC and ROC curve file is generated in this directory. Some evaluation images are also generated in this directory. ├── logs: Directory where logs are generated. │ └── Logs will be generated in this directory. Each log folder will contain backups of training files with network files and hyperparameters used. ├── models: Directory to store pretrained models, and also where models are generated. │ ├── [INSERT PRE-TRAINED MODELS HERE.] │ ├── The base MobileFaceNet for fine-tuning the CB-Net can be downloaded in this link. │ └── Trained models will be generated in this directory. ├── network: Contains loss functions, and network related files. │ ├── cb_net.py - CB-Net model file. │ ├── load_model.py - Loads pre-trained weights based on a given model. │ ├── logits.py - Contains loss functions. │ └── mobilefacenet.py - MobileFaceNet model file. ├── training: Main files for training. │ ├── main.py - Main file to run for training. Settings and hyperparameters are based on the files inconfigs
directory. │ └── train.py - Training file that is called from `main.py`. Gets batch of dataloader and contains criterion for loss back-propagation. └── utils: Miscellaneous utility functions. ├── utils_cb_net.py - Conditional Biometrics (CB) Loss function (IEEE Access). ├── utils_cmb_net.py - Conditional Multimodal Biometrics (CMB) Loss function (ICPR). └── utils.py - Utility functions.
CMB-Net:
@INPROCEEDINGS{cmb_net,
author={Ng, Tiong-Sik and Low, Cheng-Yaw and Long Chai, Jacky Chen and Beng Jin Teoh, Andrew},
booktitle={2022 26th International Conference on Pattern Recognition (ICPR)},
title={Conditional Multimodal Biometrics Embedding Learning For Periocular and Face in the Wild},
year={2022},
volume={},
number={},
pages={812-818},
keywords={Training;Biometrics (access control);Face recognition;Biological system modeling;Neural networks;Transforms;Network architecture;Conditional Biometrics;Multimodal Biometrics;Periocular;Face;Deep Embedding Learning},
doi={10.1109/ICPR56361.2022.9956636}}
CB-Net:
@ARTICLE{cb_net,
author={Ng, Tiong-Sik and Low, Cheng-Yaw and Chai, Jacky Chen Long and Teoh, Andrew Beng Jin},
journal={IEEE Access},
title={On the Representation Learning of Conditional Biometrics for Flexible Deployment},
year={2023},
volume={11},
number={},
pages={82338-82350},
keywords={Biometrics (access control);Face recognition;Correlation;Representation learning;Performance gain;Iris recognition;Conditional biometrics;face;flexible matching;periocular;representation learning},
doi={10.1109/ACCESS.2023.3301150}}