Framework for Test Dataset Quality Assessment in Deep Neural Networks using various Test Adequacy Criteria and MS
This framework is used to calculate traiditional test adequacy criteria (Structural and Non-structural), Latent Space Class Dispersion, and Mutation Scores for test dataset quality evaluation. Using this framework, mutant models using pre-training MOs can be trained. Later, LSCD, MS and distance-based Surprise Coverage scores are calculated and their correlation is studied thereby.
- Required python version: 3.10
- OS: Linux, Windows
python3.10 -m venv venv
source venv/bin/activate
python -m pip install -r requirements.txt
This folder contains pre-processed datasets that can be used to train mutant models in (/mutagen/).
This folder contains dataloading scripts that use base_dataset.py as a base class for loading the MNIST, SVHN, GTSRB and CIFAR-100 datasets. This can be extended to use a new dataset. The "corner_case_dataset.py" file contains a data loader for corner case images from fuzzing.
This folder contains original trained DNN models and their PyTorch implementations.
Use this folder and associated files to create mutant models and calculate LSCD, MS, and their correlation using the Pearson Correlation coefficient. A detailed guide can be found in this folder.
This folder should contain results obtained from mutation generation.
Due to the large size of our corner cases datasets used in our empirical study, we are sharing them via a cloud download link. Please download the necessary files using the following link: https://syncandshare.lrz.de/getlink/fiByBb2QhAViwhFXrjXJfr/