Conditional Biometrics for Periocular and Face Images

Conditional Multimodal Biometrics Embedding Learning For Periocular and Face in the Wild (ICPR 2022)

On the Representation Learning of Conditional Biometrics for Flexible Deployment (IEEE Access 2023)

Pre-requisites:

• Environment: Check requirements.txt file which was generated using pip list --format=freeze > requirements.txt command for the environment requirement. These files are slightly filtered manually, so there may be redundant packages.
• Dataset: Download dataset (training and testing) from this link. Password is conditional_biometrics. Ensure that datasets are located in data directory. Configure datasets_config.py file to point to this data directory by changing main path.
• Pre-trained models: (Optional) The pre-trained MobileFaceNet model for fine-tuning or testing can be downloaded from this link.

Training:

1. Change hyperparameters accordingly in params.py file. The set values used are the default, but it is possible to alternatively change them when running the python file.
2. Run python training/main.py. The training should start immediately.
3. Testing will be performed automatically after training is done, but it is possible to perform testing on an already trained model (see next section).

Testing:

1. Based on the (pre-)trained models in the models(/pretrained) directory, load the correct model and the architecture (in network directory) using load_model.py file. Change the file accordingly in case of different layer names, etc.
2. Evaluation:
   • Identification / Cumulative Matching Characteristic (CMC) curve: Run cmc_eval_identification.py. Based on the generated .pt files in data directory, run plot_cmc_roc_sota.ipynb to generate CMC graph.
   • Verification / Receiver Operating Characteristic (ROC) curve: Run roc_eval_verification.py. Based on the generated .pt files in data directory, run plot_cmc_roc_sota.ipynb to generate ROC graph.
3. Visualization;
   • Decidability Index: Run decidability_index.ipynb to get the histogram distribution. The decidability index $d'$ will be displayed on the graph.
   • Pearson Correlation: Run pearson_correlation.ipynb to get the correlation curve between periocular and face. The correlation coefficient will be displayed on the graph.

Comparison with State-of-the-Art (SOTA) models (Periocular)

Method	Rank-1 IR (%) (Periocular)	Rank-1 EER (%) (Periocular)	Cross-Modal IR (%) (Periocular Gallery)	Cross-Modal EER (%) (Periocular-Face)
PF-GLSR	79.03	15.56	-	-
CB-Net	86.96	9.62	77.26	9.80

The project directories are as follows:

├── configs: Contains configuration files and hyperparameters to run the codes
│   ├── datasets_config.py - Contains directory path for dataset files. Change 'main' in 'main_path' dictionary to point to dataset, e.g., /home/cb_net/data (without slash).
│   └── params.py - Hyperparameters and arguments for training.
├── data: Directory for dataset preprocessing, and folder to insert data based on config.py files.
│   ├── [INSERT DATASET HERE.]
│   ├── The .pt files to plot the CMC and ROC graphs will be generated in this directory.
│   └── data_loader.py - Generate training and testing PyTorch dataloader. Adjust the augmentations etc. in this file. Batch size of data is also determined here, based on the values set in params.py.
├── eval: Evaluation metrics - Identification and Verification, also contains .ipynb files to plot CMC and ROC graphs.
│   ├── cmc_eval_identification.py - Evaluates Rank-1 Identification Rate (IR) and generates Cumulative Matching Characteristic (CMC) curve, which are saved as .pt files in data directory. Use these .pt files to generate CMC curves.
│   ├── decidability_index.ipynb - Notebook to plot Decidability Index (d') histogram.
│   ├── identification.py - Rank-1 Identification Rate (IR) evaluation.
│   ├── lbp_extract.py - Local Binary Pattern (LBP) feature extraction and identification calculation.
│   ├── pearson_correlation.ipynb - Notebook to plot Pearson Correlation (ρ) graph.
│   ├── plot_cmc_roc.ipynb - Notebook to plot CMC and ROC side-by-side simulatenously.
│   └── roc_eval_verification.py - Evaluates Verification Equal Error Rate (EER) and generates Receiver Operating Characteristic (ROC) curve, which are saved as .pt files in data directory. Use these .pt files to generate ROC curves.
├── graphs: Directory where graphs and visualization evaluations are generated.
│   └── CMC and ROC curve file is generated in this directory. Some evaluation images are also generated in this directory.
├── logs: Directory where logs are generated.
│   └── Logs will be generated in this directory. Each log folder will contain backups of training files with network files and hyperparameters used.
├── models: Directory to store pretrained models, and also where models are generated.
│   ├── [INSERT PRE-TRAINED MODELS HERE.]
│   ├── The base MobileFaceNet for fine-tuning the CB-Net can be downloaded in this link.
│   └── Trained models will be generated in this directory.
├── network: Contains loss functions, and network related files.
│   ├── cb_net.py - CB-Net model file.
│   ├── load_model.py - Loads pre-trained weights based on a given model.
│   ├── logits.py - Contains loss functions.
│   └── mobilefacenet.py - MobileFaceNet model file.
├── training: Main files for training.
│   ├── main.py - Main file to run for training. Settings and hyperparameters are based on the files in configs directory.
│   └── train.py - Training file that is called from `main.py`. Gets batch of dataloader and contains criterion for loss back-propagation.
└── utils: Miscellaneous utility functions.
    ├── utils_cb_net.py - Conditional Biometrics (CB) Loss function (IEEE Access).
    ├── utils_cmb_net.py - Conditional Multimodal Biometrics (CMB) Loss function (ICPR).
    └── utils.py - Utility functions.

Citation for this work:

CMB-Net:

@INPROCEEDINGS{cmb_net,
  author={Ng, Tiong-Sik and Low, Cheng-Yaw and Long Chai, Jacky Chen and Beng Jin Teoh, Andrew},
  booktitle={2022 26th International Conference on Pattern Recognition (ICPR)}, 
  title={Conditional Multimodal Biometrics Embedding Learning For Periocular and Face in the Wild}, 
  year={2022},
  volume={},
  number={},
  pages={812-818},
  keywords={Training;Biometrics (access control);Face recognition;Biological system modeling;Neural networks;Transforms;Network architecture;Conditional Biometrics;Multimodal Biometrics;Periocular;Face;Deep Embedding Learning},
  doi={10.1109/ICPR56361.2022.9956636}}

CB-Net:

@ARTICLE{cb_net,
  author={Ng, Tiong-Sik and Low, Cheng-Yaw and Chai, Jacky Chen Long and Teoh, Andrew Beng Jin},
  journal={IEEE Access}, 
  title={On the Representation Learning of Conditional Biometrics for Flexible Deployment}, 
  year={2023},
  volume={11},
  number={},
  pages={82338-82350},
  keywords={Biometrics (access control);Face recognition;Correlation;Representation learning;Performance gain;Iris recognition;Conditional biometrics;face;flexible matching;periocular;representation learning},
  doi={10.1109/ACCESS.2023.3301150}}