Coverage guided fuzz testing framework for DNNs

This is a framework -

1. To generate test dataset using Coverage-Guided Fuzzing (CGF) and
2. To evaluate effectivness of various coverge criteria and seed selection strategies. The current framework supports classification network on various DNN architectures, MNIST, SVHN, GTSRB datasets.

How to use:

Parts:

1. main.py: Use this main script to run fuzzing. Various input arguments are set to default. They should be changed based on requirements.

General Notes

1. Each model to be tested would need a saved state dictionary of trained model, a model profile and a set of initial seeds.
2. The model profile and initial seeds can be constructed using fuzzer by setting respective flags to 1.
3. The trained model need to be saved under - models/dataset/weights/<model_name>/
4. For new networks, the functions intermediate_outputs and good_seed_detection need to be implemented.
5. A configuration file needed for each model-dataset combination and saved in config/. Need to be thorough when creating these.

Installation:

Run requirements.txt file for installing necessary Python packages. (Use of virtual enviornment is highly recommended)

MacOS

1. Install Python 3.10 (preferably using pyenv)
2. Activate a virtual environment python -m venv .env && source .env/bin/activate
3. Install requirements: pip install -r requirements_mac.txt
4. Install brew install freetype imagemagick (needs Homebrew package manager). Note the installation path of imagemagick, which will be printed towards the end of the installation. Depending on the exact version installed, add a line similar to this to your ~/.bashrc or ~/.zshrc: export MAGICK_HOME=/opt/homebrew/Cellar/imagemagick/7.1.1-21
5. Run git submodule init && git submodule update to check out the flann submodule.
6. Build and install the FLANN library:
   1. brew install cmake
   2. cd flann && mkdir build && cd build
   3. cmake ..
   4. make
   5. sudo make install (enter your system password)
7. Test by running: python main.py --help

Accessing Datasets (Download Link)

Due to the large size of our corner cases datasets used in our empirical study, we are sharing them via a cloud download link. Please download the necessary files using the following link: https://www.dropbox.com/scl/fi/a2dws2hjpf8qslqc2riqx/fuzzing_data_eq_samples.zip?rlkey=fy22mwm4c1j73rczq0ag4mgbp&st=thm1w4wy&dl=0