How to create supermutants and run evaluation for an existing corpus?

[voidptr127]

I have two questions regarding your benchmark:

• Is it possible to perform a mutation analysis for already finished fuzzing campaigns?
• How do you compile the supermutants and run the second phase?

We have 5 (programs) x 20 (campaigns) x 24h (campaign duration) for our fuzzer evaluation. We already have the seed corpus for each finished campaign available. It would be good if we could obtain the total killed and total covered mutant numbers (or phase I covered and phase I killed numbers) for these campaigns as an additional metric.

I see that you use two phases for analysis where 1) you create a seed corpus with maximum coverage over 24h (that would be the equivalent to our existing seed corpus) and 2) re-run fuzzing for 1h against all supermutants of a program. (Hence my question: How do you compile the supermutants and run the fuzzer for phase II?)

For example, we compile lua as follows:

git clone https://github.com/lua/lua.git
cd lua/
sed -i 's/^CC=.*$/CC=\~\/AFLplusplus\/afl-cc/' makefile

CC=~/AFLplusplus/afl-cc \
  CXX=~/AFLplusplus/afl-c++ \
  AFL_CC_COMPILER=LLVM \
  AFL_LLVM_INSTRUMENT=PCGUARD \
  AFL_USE_ASAN=1 \
  make -j$(nproc)

# this is the final executable/harness used by the fuzzer
~/lua/lua -v

How would you create the respective supermutants and obtain the survived and covered values for a given seed corpus?

[phi-go]

Hey, happy to help but let me start this off with a warning. Make sure you can create a single bitcode file for your programs, otherwise this benchmark will not work. See here for some examples: https://github.com/CISPA-SysSec/mua_fuzzer_bench/tree/main/dockerfiles/programs

To answer your question, you have already done step one (https://github.com/CISPA-SysSec/mua_fuzzer_bench/tree/main/dockerfiles/programs). You would now need to continue with step two (https://github.com/CISPA-SysSec/mua_fuzzer_bench?tab=readme-ov-file#supermutant-evaluation-stage-2), here the seed-dir would point to your seed corpus. It probably is easiest to just integrate the programs using docker as described here: https://github.com/CISPA-SysSec/mua_fuzzer_bench?tab=readme-ov-file#adding-new-programs. If you only want the data for the seed corpus, I'm afraid the best way would be to use a one minute fuzzing time, doing this right would require some code changes. The database will contain info on if this was found by the seed or during the run.

[voidptr127]

Thank you for the quick response!

Can you please elaborate a bit more on how to continue here and maybe give a code example (list of commands)?

Let's take a look at the re2_harness/build.sh :

$CXX $CXXFLAGS -std=c++11 -static -c -I. \
    -c re2/fuzzing/re2_fuzzer.cc \
    -o re2_fuzzer
get-bc re2_fuzzer

llvm-link \
    re2_fuzzer.bc \
    obj/libre2.a.bc \
    -o $OUT/re2.bc

$CXX $CXXFLAGS -std=c++11 \
    $LIB_FUZZING_ENGINE \
    $OUT/re2.bc \
    -lpthread \
    -o $OUT/re2_fuzzer

I Assume you create the bitcode file here with the get-bc command? What is the get-bc command? Where does it come from?

I would expect to create bitcode files with the -emit-llvm compiler flag. Afterwards link and compile the bitcode file similarly as in your example to a final executable:

clang -emit-llvm -c file1.c -o file1.bc
clang -emit-llvm -c file2.c -o file2.bc
llvm-link file1.bc file2.bc -o fuzz_target.bc
clang fuzz_target.bc -o fuzz_target

I can create this for the lua executable under one condition: neither compile this with afl-cc nor with the -fsanitize-coverage=edge flag (have not tried ASAN, yet). The afl-cc command results in a crash (core dump) and the clang command results in a fatal error with undefined reference to __sanitizer_cov_trace_pc_guard_init messages.

Do you need instrumentation enabled to just get the results for the mutation analysis?

Would I also need to integrate the other fuzzers (like Nautilus) or could I just re-use the default AFL++ runner to obtain the results?

[phi-go]

Hey, the get-bc command comes from here: https://github.com/SRI-CSL/gllvm, I think we install it in the base image. It doesn't really matter how you get the bitcode file, this is just a convenient way as it can be used as a compiler replacement (CC).

You do not need to compile with afl-cc or other coverage to get the bitcode file. The instrumentation step is done at later point. (The bitcode file contains the full program so for stage one we compile that with instrumentation and for stage 2 we search through the bitcode to find places to mutate, the result of that mutation is again a bitcode file which can again be compiled.)

You do not need to add any fuzzers, as I understand, you do not want to fuzz individual mutants (as we do for stage 2) and already did stage 1. You can use any fuzzer but you would need to manage / combine the results yourself, so run this command once to get a stable list of mutations:

src/mua_fuzzer_benchmark/eval.py eval \
    --fuzzers aflpp \
    --progs <prog> \
    --fuzz-time 1 \
    --seed-dir <any dir with a file> \
    --result-path data/base/stats_all.db

The bitcode file is part of the db, no need to worry about that part if this command works.

Then generate a rerun file (not quite sure these arguments are correct but I don't have this project in active use atm, the help should be informative though):

src/mua_fuzzer_benchmark/eval.py generate_rerun_file \
    --db data/base/stats_all.db \
    --out-file rerun_file.json \
    --untried no \
    --covered yes \
    --skip-timeout yes \
    --skip-killed no \
    --skip-crashed yes \
    --mode keep

Then you can eval your seed corpus for each fuzzer with this command (with ASAN as an example):

MUT_BUILD_ASAN=1 src/mua_fuzzer_benchmark/eval.py eval \
    --fuzzers aflpp \
    --progs <prog> \
    --fuzz-time 1 \
    --seed-dir <corpus dir> \
    --rerun data/base/stats_all.db \
    --rerun-mutations rerun_file.json \
    --result-path data/<fuzzer you are evaluating>/stats_all.db

Then you can collect and combine the results from the databases.