Safety-Critical Tool Submission: mutest-rs (Mutation testing)

[zalanlevai]

What is the name of the tool?

mutest-rs

Add a concise description of the tool. Please refrain from marketing claims and stick to what the tool is, does, and how it would be used?

mutest-rs is an open-source mutation testing tool for Rust projects that I have been developing for the past four years, with the aim of making it fast and accessible for even larger projects. Mutation testing is a technique for finding testing gaps through generated bugs. To my knowledge, there are some safety-critical standards and certifications that either require it, or at least strongly suggest its use. I have ongoing work on mutest-rs for adding support for mutating embedded firmware written in Rust, and testing mutated code directly on embedded devices, that I am expecting to release in the next 2–4 months. I expect this to be of particular interest to a lot of safety-critical applications.

Link to the tool repo or homepage

https://mutest.rs

License of the tool

Apache-2.0 OR MIT

What is your role/connection to this tool?

Creator/Maintainer (open source)

Optional name of the tool vendor

No response

Optional liability provided by the tool vendor

No response

List any standards the tool is qualified for

No response

Tool category

test-runner

[zalanlevai]

Please note, that the category test-runner, is not an exact fit, and I could see how it could be put under other categories as well.

[mhatzl]

Hi @zalanlevai,
thank you for the proposal and looking forward to your demo in the next meeting.
We are currently in the process to moving the tools list around to be displayed on the arewesafetycriticalyet website. I'd create a PR to add your tool once this move is done to prevent the chance to miss any tool during migration.

[mhatzl]

What could a new tool category be named that this tool would fit in?
or should we rename test-runner to testing?

[zalanlevai]

Hi @mhatzl, the website update sounds great! As for the category, it depends on how specific you want them to be. You could place mutest-rs into a new test-strengthening/test-hardening/test-improvement category as the most specific option. Another option is simply putting it in the more generic coverage category, although mutation testing provides a much stronger notion of "functional coverage" of potential faulty programs. Of course it could always be put under a generic testing category as well. I would vouch for the first option of course, but I can see arguments for all of them. I suppose any of the first options could complement the existing test-runner category well.

[felix91gr]

For context, the category for tool submissions is currently selected from the tool submission issue template:

  - type: dropdown
    id: tool_type
    attributes:
      label: Tool category
      description: Specific use and type of this tool
      options:
        - "code-coverage"
        - "requirements-traceability"
        - "compiler"
        - "test-runner"
        - "formal-verification"
        - "static-analysis"
        - "tests-generation"
        - "package-manager"
        - "profiler"
        - "debugger"
    validations:
      required: false

Of those, the following definitely do not fit the bill:

• compiler: the tool is not a compiler.
• test-runner: test running is not its core function.
• formal-verification: not a FV tool.
• static-analysis: MT is a dynamic analysis tool.
• package-manager: not a PM.
• profiler: profiling is about performance. MT isn't.
• debugger: it's not a debugger either.

Of the remaining categories:

• code-coverage: MT is way more than just code coverage, as Zalán points out:

  Another option is simply putting it in the more generic coverage category, although mutation testing provides a much stronger notion of "functional coverage" of potential faulty programs.

  To categorize it under CC would miss 98% of what the tool is.

• requirements-traceability: RT is a related concern, but while Mutation Testing can be used to help in Requirements Traceability, it's not the core of what the tool does.

• tests-generation: not a test generation tool. A property testing framework would be more along those lines.

So I think the only reasonable option is to add new categories.

---

You could place mutest-rs into a new test-strengthening/test-hardening/test-improvement category as the most specific option.

All of those sound good.

Though test strengthening, hardening and improvement are all goals of someone using MT, they are not exactly what MT does, right? @zalanlevai. Maybe test-quality-measurement or something along those lines could describe it even better?

I suppose any of the first options could complement the existing test-runner category well.

I agree. The existing categories are definitely leaving some holes.

Of course it could always be put under a generic testing category as well.

Indeed. Though we can do much better than that, and I'd say that mutation testing is a useful enough tool to warrant a better-suiting category :)

[mhatzl]

test-quality-measurement sounds fitting to me for mutest-rs.

However, the problem with such fine grained categories is that not many other tools will match this. So you get many detailed categories each with 1-2 tools.
That is why I prefer to stick with the generic testing category for now, because we simply don't have a lot of tools listed at the moment. If a category becomes too big (say >15 tools) then we can always decide to split up the category in like two new ones that ~evenly distribute the tools.

Fewer more general categories also makes it easier for people adding new tools.

[felix91gr]

@mhatzl I would use something different from just testing in this case though, because a MT tool doesn't really test the project itself. It gives a metric of the quality of its test suite.

testing would be misleading in that sense.

---

Re: the rest

not many other tools will match this. So you get many detailed categories each with 1-2 tools. (...) Fewer more general categories also makes it easier for people adding new tools.

Maybe categories (a partition of the tool space) is not the right structure. Maybe what we want is tags: something that can be attached to a tool to, e.g. say "this tool contains / pertains to ".

On the other hand, currently some categories can be interpreted as containing parts of one another: static-analysis is a foundational part of compiler and formal-verification. And code-coverage is usually required for requirements-traceability.

That is why I prefer to stick with the generic testing category for now, because we simply don't have a lot of tools listed at the moment.

I don't know if that's a good criterion for picking categories. Maybe it is, but I'm not sure if doing that is going to bite us in the future.

If a category becomes too big (say >15 tools) then we can always decide to split up the category in like two new ones that ~evenly distribute the tools.

I'm going to go out on a limb and say that ~evenly distributing the tools is almost never going to be possible. Categories are rarely evenly distributed. For example, in the top libraries of the testing section of lib.rs, we see that most libraries belong to one of two categories:

• Enhancements to vanilla tools: pretty_assertions, serial_test, test-case, test-log, etc
• Mock testing frameworks: wiremock, httpmock, mockito, mockall, etc

Whereas the rest (about 3/5ths of the total) are of one of a handful of other categories, like:

• CLI testing
• Property testing
• Fuzzing
• Container integration testing
• Etc

Anyway. I don't expect even splits to happen often. They might happen from time to time, but the tendency that I've seen is for the distribution of a category into sub-categories to be rather non-uniform.

[mhatzl]

Maybe what we want is tags

I was also thinking about this while creating the tools list,
but imo this makes it harder to visualize the tools in a way that makes it easy for people to see at a glance what the state of Rust tooling is.
With categories, you can nicely group tools by default.
With tags, you'd have one big table/list and users would first need to set filters to see which tools belong together, or we'd show a tool in multiple tables/lists per tag.

Looking at the current categories, maybe it's better if we adapt them to be related to common activities. Tools should then be put into the category/activity that the tool will most likely be used in.

• testing ... for tools that are primarily used while testing. Merging test-runner and test-generation. mutest-rs would imo also fall into this category, because it helps to improve your testsuite, and improving your testsuite is a testing activity. Without tests, mutest-rs would have nothing to check against.
• formal-verification ... It may be for testing, but can also be part of implementation. e.g. specifying contracts in your source code
• static-analysis ... formal-verification is part of this, so we could combine those two, but since we already have ~3 tools dedicated to formal-verification and I expect this to grow, having them separate is ok
• profiling ... rename of profiler, because it better fits the activity intent
• debugging ... same as profiling
• code-coverage ... although mostly used while testing, you could measure coverage while executing your regular application or benchmarks. The name is not that fitting for an activity, but I cannot think of a better name right now
• requirements-traceability ... is it's own activity, because you may trace requirements to source code, tests, or other artifacts
• package-manager ... not an activity, but e.g. supply chain management also doesn't feel right. cargo is also special, because it manages dependencies, builds, tests, publishes your Crates... for cargo, the tags approach would be good
• compiler ... building might be a better category name, but imo people looking for safety critical tools will expect a compiler category to be there, since a compiler is that important

safety critical standards are mostly activity based, so having tools grouped this way should make it easier for people to find the tools they are looking for.

[mhatzl]

I don't know if that's a good criterion for picking categories. Maybe it is, but I'm not sure if doing that is going to bite us in the future.

we can always change/adapt the list format. I even think that is is highly likely we will adapt it over the next years if more tools are added. But I don't see a problem with that.

I'm going to go out on a limb and say that ~evenly distributing the tools is almost never going to be possible.

I also don't worry about perfect distribution. My approach would be that once a category has too many tools, we look at all the tools and see if we can find two (or more if needed) categories that somewhat nicely split the tools up. Similar to formal-verification and static-analysis the original category can remain as is and we just pick out tools that can be grouped to a sub-category. But imo this is a case-by-case thing that is not needed to be handled in general.

[felix91gr]

Looking at the current categories, maybe it's better if we adapt them to be related to common activities. Tools should then be put into the category/activity that the tool will most likely be used in.

Oh, I like how this sounds. We should document this somewhere, if we can (along with the meaning we have in mind for the categories, I think). Currently I don't see the Category bit explained in the tool submission form. A link to the docs that explain the categories and our use of them for this should suffice :)

My approach would be that once a category has too many tools, we look at all the tools and see if we can find two (or more if needed) categories that somewhat nicely split the tools up. Similar to formal-verification and static-analysis the original category can remain as is and we just pick out tools that can be grouped to a sub-category. But imo this is a case-by-case thing that is not needed to be handled in general.

That seems very reasonable :)

[mhatzl]

We should document this somewhere, if we can (along with the meaning we have in mind for the categories, I think). Currently I don't see the Category bit explained in the tool submission form. A link to the docs that explain the categories and our use of them for this should suffice :)

yep agree that we need to document the meaning of the categories.
not sure where though. Maybe as paragraph at the beginning of a category section on the new tools-list page? This way it is visible for people searching for tools, and we can link to the sections from the submission form.

[zalanlevai]

I also like @mhatzl's ideas on the tooling list front. Categories makes things easier to parse a very long list of tools, especially if they are based on the primary "activity" they relate to, and subcategories can break things up for big top-level categories, such as testing. So, +1 from me.

RE: @felix91gr

Of course it could always be put under a generic testing category as well.

Indeed. Though we can do much better than that, and I'd say that mutation testing is a useful enough tool to warrant a better-suiting category :)

I agree that putting mutation testing under just a generic testing category, with no other distinction within that, would do it a bit of a disservice. I would personally expect most things under a testing category (if there were no other subcategories within it) to relate to test authoring, specifically things like quickcheck for property testing and criterion for benchmarks. Of course this is not an issue with subcategories, where the many different testing related activities, whether that's authoring or quality measurement or running or something else, can be separated.

Though test strengthening, hardening and improvement are all goals of someone using MT, they are not exactly what MT does, right? @zalanlevai. Maybe test-quality-measurement or something along those lines could describe it even better?

That is true! I suppose it depends on whether you want the categories to be strictly outcome-oriented, or whether you allow them to be more goal-oriented. I think either is a fair stance. The test-quality-measurement category is a good choice for the former. We also use the phrase "test adequacy" for this idea, at least in mutation testing research, but I think "test quality" is more accessible to a wider audience.

The tricky things with this, at least for mutest-rs, is that we provide tools not just strictly for measurement, but also use the data, both dynamic (MT) and static analysis (reachability trace analysis), to effectively provide a testing improvement "workbench" of sorts in the form of an interactive inspector: see the screenshots below. In this way, mutest-rs might be going further than most existing mutation testing tools.

Caption: mutest-rs provides a web-based inspector showing the generated mutations, and whether they were detected by the test suite.

Caption: For any given mutation, mutest-rs can highlight the relevant test cases, whether they individually detected the mutation, and the traces along which the tests could reach the mutated function.

Caption: Mutations can have interesting outcomes, and may be exercised by fewer or more test cases than others.

Caption: The inspector in mutest-rs can display the exact snippets of code relevant between a test and a mutation.

Caption: Of course, mutest-rs can provide an overview of testing quality from the perspective of the tests.

Caption: mutest-rs can also break things down for individual tests and the mutations relevant to them.