Onweer

Onweer is a prototype research tool which combines API fuzzing and fault injection to test the resilience of microservice systems.

Install

Running and building Onweer requires Poetry. Installation instructions for Poetry can be found here.

Clone the Onweer source code repository with Git or download the source code, then run the following command in the repository’s root directory:

$ poetry install

This will download all dependencies and install them in a venv for use.

Alternatively, if you wish to analyse the reports generated by Onweer to verify the results or reproduce the graphs from our paper, also install the dependencies required for that with the following command:

$ poetry install --with graph

Running Onweer

Instrumentation

Onweer requires instrumenting the application under test in order to gather coverage information, trace requests and inject faults. Please refer to the Onweer Agent documentation for details on how to instrument applications. Once your application has been instrumented, you must tell Onweer where it can contact the Onweer agents using a services.yml file. This is a YAML file containing a list of objects with a name property, containing the name to be used for the service, and an endpoint property, containing a HTTP URL where Onweer can reach the Onweer Agent. For example:

- name: yis
  endpoint: http://localhost:8070
- name: un
  endpoint: http://localhost:8071

API specification

Onweer uses an OpenAPI specification to know which endpoints and parameters are available. Ensure that a specification is available locally in YAML format.

Running Onweer

Onweer can be ran with the following command, assuming that services.yml contains the specification of Onweer Agents, openapi.yml contains the OpenAPI specification and your application can be reached at http://localhost:8080.

$ poetry run fuzz --services services.yml openapi.yml http://localhost:8080

Several additional options are available to configure Onweer’s behavior:

--stop-on-error

Onweer will stop on the first error found instead of running to the timeout.

--timeout SECONDS

Run for SECONDS seconds and then stop.

--nofail

Onweer will continue running instead of stopping when test execution fails (due to timeout, a bug in Onweer, etc.)

--trials TRIALS

Apply TRIALS mutators on each testing iteration

--initpop POP

Start fuzzing with POP tests for each API endpoint.

--enable-mut MUT, --disable-mut MUT

Enable or disable a mutator. By default, all mutators are enabled.

Coverage-only runs

Onweer can be ran in a “coverage-only” mode, where it will only collect coverage from the Onweer agents every second. This is useful to compare it to other tools which may not collect code coverage.

Run this mode using:

$ poetry run covonly --services services.yml

The output report will have its name prefixed with covonly- and contain an array of timestamps and coverage achieved at that timestamp.

Reports

After a run finishes, a JSON report of the run is written under the runs/ directory. It contains a detailed description of the parameters used, some general statistics about the run, and detailed information about every sequence added to the population as well as every sequence that resulted in an error.

Graphs and statistics

Under the graph/ directory, we include a number of scripts to perform statistical analysis on the reports generated by Onweer and to create graphs of Onweer’s performance.

poetry run graph [-p] REPORT...

Draw a stair graph of the coverage achieved by the given reports, if -p is passed it is shown as a percentage of the maximum possible coverage estimated by Onweer.

• Reports with the fault injection mutator disabled will be shown in a different color.
• Also prints some statistical information about the given reports.

poetry run graphnice [-p] [--covonly REPORT] REPORT...

Same as graph, but draw a single stair representing the average as well as a shaded area representing the 10th and 90th percentiles. Add any number of coverage-only reports with --covonly options.

• Reports with the fault injection mutator disabled will be shown separately.

poetry run classcov [-g] REPORT...

Create a bar graph showing the class coverage difference between reports with and without fault injection. If -g is passed, group classes with the same name on different services together.

poetry run vispop [-e] REPORT

Create a GraphViz DOT file which may be rendered into a visualization of the population. Shows the population with edges indicating which previous population member every sequence was mutated from as well as details of every sequence in the node itself. If -e is passed, draw errors as full nodes instead of aggregating them into a single node only containing the number of errors.

poetry run uerrs REPORT...

Create a report containing all the unique errors found in every REPORT, split by with fault injection and without fault injection.

Scripts

For convenience, several scripts wrapping Onweer and other tools are included in the scripts/ directory:

./scripts/runexperiment.sh

Automatically runs Onweer on an application specified as a docker-compose project ten times with and without fault injection. Can be configured by changing variables inside the script.

./scripts/dottopng.fish DIR [-e]

Requires fish and dot. Creates a graphviz file and renders it to PNG for every Onweer report in DIR. If -e is passed, displays errors as full nodes instead of aggregating them.

./scripts/truncateimagedata.fish REPORT...

Requires fish and sed. Replaces any base64-encoded image data in every REPORT with DATATRUNCATED in order to save disk space.

Contact

Onweer was developed at the Vrije Universiteit Brussel as part of a research grant by the Research Foundation - Flanders (FWO), grant number 1SHFL24N. .

If you wish to contribute or collaborate, please contact Gilles Coremans <[email protected]> or Coen De Roover <[email protected]>