ARASH is named after the heroic archer in Persian mythology. It can also be read as an acronym for Analysis of cRASH.

Usage guide

Running an experiment requires three steps:

1. Install dependencies.
2. Setting up LLM access.
3. Launch experiment.

Prerequisites

Dependencies

You must install:

1. Python 3.11
2. pip
3. python3.11-venv
4. Git
5. Docker

Python Dependencies

Install required dependencies in a Python virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

LLM Access

Setup OpenAI or Vertex AI with the following steps.

OpenAI

There are two ways to access OpenAI models.

1. OpenAI API Key on OpenAI: This is the default way for using OpenAI models.

2. OpenAI API Key on Azure: Please refer to this section if you are using OpenAI models on Azure.

OpenAI API Key on OpenAI

OpenAI requires an API key.

Then set it as an ENV variable:

export OPENAI_API_KEY='<your-api-key>'

OpenAI API Key on Azure

If your OpenAI API key is hosted on Azure, you need the specified Endpoint, API key, and the API version (optional).

Then set them as ENV variables:

export AZURE_OPENAI_API_KEY='<your-azure-api-key>'
export AZURE_OPENAI_ENDPOINT='<your-azure-endpoint>'
export AZURE_OPENAI_API_VERSION='<your-azure-api-version>'

Tip: To distinguish between the two ways of accessing OpenAI models, you need to add -azure to the model name when using OpenAI on Azure. For example, gpt-3.5-turbo-azure will use OpenAI on Azure, while gpt-3.5-turbo will use OpenAI on OpenAI.

Vertex AI

Accessing Vertex AI models require a Google Cloud Project (GCP) with Vertex AI enabled.

Then auth to GCP:

gcloud auth login
gcloud auth application-default login
gcloud auth application-default set-quota-project <your-project>

You'll also need to specify the GCP projects and locations where you have Vertex AI quota (comma delimited):

export CLOUD_ML_PROJECT_ID=<gcp-project-id>
export VERTEX_AI_LOCATIONS=us-west1,us-west4,us-east4,us-central1,northamerica-northeast1

Running experiments

To generate and evaluate the fuzz targets in a benchmark set via local experiments:

python ./arash_run_all.py \
    --model=<model-name> \
    --benchmark-directory=./benchmarks/libaom-av1_highbd_dr_prediction_z3_avx2 \
    [--ai-binary=<llm-access-binary>] \
    [--work-dir=results-dir] \
    [...] \

where the <model-name> must be the name of one of the supported models.

Benchmarks

Configure Benchmark

A crash benchmark should contains:

1. fuzz target source code (named as fuzz_target.c or fuzz_target.cc)
2. testcase to trigger crash (named as testcase)
3. original crash output (named as run.log)
4. project information (named as project.yaml)

Each file cannot be empty. project.yaml should include project name, project folder name in docker, language(c or c++), sanitizer, commit, the built-in target_name and target_path of OSS-Fuzz, extra compiler flag (optional, for compiling fuzz target), index (optional, to distinguish benchmarks with the same API), header path (optional, for compiling fuzz target), static link input file path (optional, for compiling fuzz target). Here is an example. Note that the project under test needs to be integrated into OSS-Fuzz to build. Currently only libfuzzer is supported. Please make sure that the network can connect to the official source.

Clang Static Analysis (CSA)

Arash supports Clang Static Analysis (CSA) to analyze the fuzz target code. Current CSA rules are stored in csa folder.

Development

Auto Format / Lint

You can install a Git pre-push hook to auto-format/-lint your code:

./helper/add_pre-push_hook

Or manually run the formater/linter by running:

.github/helper/presubmit

Updating Dependencies

We use https://github.com/jazzband/pip-tools to manage our Python dependencies.

pip install pip-tools
pip-compile requirements.in > requirements.txt
pip install -r requirements.txt