Compiler built-in support for testing

[DyXel]

TLDR: I saw the "write 1 line in any C++ file and start seeing benefit" sticky note in Herb's presentation, and thought: "What if that 1 line is a test function?" And so I quickly hacked something together to experiment. After playing around with this for a bit, I can't stop thinking how this would make perfect sense to be directly supported by the compiler as opposed to being a separate library, let me share my thoughts with you all; My hope is to start up the discussion about better testing, and if we agree on something, to start implementing a testing framework for cppfront.

Why?

Why not?

Other languages, such as Python, D and Rust (to name a few) have direct support for testing in their standard library or baked-in as language constructs, I don't see anybody complaining about this, other than wanting even better support than what there is currently. C++ itself of course has many, many testing libraries as well, each one with different trade-offs and hacks to provide a nicer UX. Testing and measuring is something we already teach everyone to do (cf. Testing - often to an extreme extent - is essential.); We have the opportunity to provide first-class support for it, with nicer diagnostics, simpler teachability, better composition with other language features, and obeying the zero-overhead principle (if you don't write any tests, you pay exactly 0 for it, and if you do write any test, you couldn't write better testing support by hand).

Adoptability

I think its very seducing to think that the first line of code you could write in Cpp2 for your existing Cpp1 project is test code, making your existing work a little safer and more reliable, while warming up for when you get to refactor bigger parts of it in Cpp2, this would also overall help adopters to learn the Cpp2 syntax and its capabilities. Also, having a fully unit tested codebase (talking about cppfront itself here) inspires confidence to adopters who wish to use the software; They know that care is being taken to not break stuff, and that the same test framework that the compiler developers are using is the same one they can use, all that power in the palm of their hand for them to just tap into.

Composition with other language features

Recently I learned about The Lakos Rule and while I agree that its good practice for existing C++ code, I disagree that Contracts+Exceptions and Testing should conflict with each other, or that users "should be careful about testing noexcept with contracts as it might cause surprises". I think writing your tests along contracts and exceptions should just work out the box in Cpp2, and to that end, the compiler should have specific support for it. More on this later.

Self-diagnosis

Cppfront itself needs to be unit tested, the project should be kept simple with no external dependencies and approachable by users who would like to hack on it. Having a testing framework would solve these two problems, and would also make refactoring easier in the long run.

What should this unit test framework support?

The minimum bar is (IMO, feel free to add more):

• Auto-registering of test functions and auto-running them at startup (what I partially implemented on my experiment branch, see above)
• Assert statements (compose together with and reuse Contracts support rather than inventing additional constructs)
• Simple diagnostics for failed cases (file, line number)
• Ability to opt-out off generating tests (release builds)

After these, better support for other things should be added, but I think an implementation that satisfy those points can already prove useful in cppfront and smaller projects. Other important features I would like to see implemented are:

Native support for testing private parts of a type

I have seen enough stackoverflow questions asking how to test private members, and the hacks people come up with are both ingenious and scary (such as defining a macro to override the private keyword).

Folks oftentimes disagree and say that private parts should not be accessible except by the class, I say a test function should be able appear anywhere that a normal function can, including inside a type definition's private part, and that this is compatible with the other's party argument, if the test function itself is within the type, it can test things such as invariants holding after performing certain work and so on without breaking the world. Some people suggest that tests should be able to live close to the code they are testing, and I agree with it; We should make writing a test function as simple and general as writing a regular function.

Work together with other tooling, not against it

This is mostly specific to sanitizers, a compiler-level testing framework should help you catch undefined behavior and other problematic code, current compilers provide hooks to customize this functionality, and we could string it together so that contracts, exceptions and sanitizers work all together and help the user more easily debug problems and test correctness.

Death tests & crash resilience

Today only GTest supports this (AFAIK), sometimes its necessary to test that a program terminates, if, say, a contract is violated. It is also a necessity that if a running test crashes, it is handled gracefully and doesn't bring the whole session down. This is not trivial to implement, but I think is a necessary part of a mature testing framework nonetheless.

Expression decomposition

This is the next thing I wanted to experiment on when I get the chance. I initially got the idea from pytest (An excellent Python test framework library which I use daily at work, I can't overstate how helpful this feature is); Its implementation consists of modifying the AST near assert statements and injecting additional code to provide very helpful error reporting information to end-users. Catch2 has a decomposition implementation as well, though it is very limited, it consists of using operator overloading and operator precedence inside a macro in order to decompose the rhs of an assert expression vs the lhs.

This is not just about UX! The reason I wanted to experiment on this next is that I believe this is the solution to the problems we currently face when detecting issues within a contract. If we are able to wrap the expression up, and evaluate it step by step while bookkeeping what we have currently processed, then we can provide the user with detailed information and say what went wrong, even when UB is detected by a sanitizer.

This is also another reason why I believe testing should be directly supported by the compiler, as opposed to giving users the opportunity to do it themselves; I think general support for mutating the AST will bring up way more issues than it will fix, and I remember Herb himself being against such thing, with reason. A compiler built-in metafunction can side-step the issue since it can be blessed by compiler implementors with access to internals.

Details of my current experiment

I decided to just run with a single metafunction, called @test. You spell it as: my_test is a @test function with body statements-block, here's how it looks like:

my_test: @test () = {
    assert(true);
}

The compiler then saves the name and later generates Cpp1 code that registers the function for running:

//=== Cpp2 test registry for this TU =====================================
namespace {
cpp2::testing::auto_reg t0(&my_test);
}

You can then just run the session whenever you see fit. At the moment I auto-generate the run statement if a main function was defined, which I think is a good default:

auto main(int const argc_, char** argv_) -> int{
    cpp2::testing::session::the().run(argc_, argv_);
    auto const args = cpp2::make_args(argc_, argv_);
    std::cout << "Hello World!\n";
    return 0;
}

To me this already looks promising, and we still have the tests' template arguments, function arguments, and return types as real estate for customization. I have not yet thought how these could be effectively used to write good, concise tests, feel free to discuss :^)

Not a type metafunction?

I think it could be a thing on top, GTest uses classes as "fixtures" in order to parametrize tests, other frameworks such as Catch2 don't recommend them and instead suggest using "generators". Myself, I don't think we should be constrained to stuffing test functions inside a type, or a namespace or whatever. I think we should be able to write a single function and see positive results, I hope I have made a good case for adding proper support for metafunctions for a function with this example 😉

Conclusion

I believe we can and should have such a nice thing be directly supported within the compiler, and be just as ergonomic or even better than what there is currently out there in different libraries or other programming languages. I look forward to hearing all your opinions, and it goes without saying that if there's good incentive to implement something like this, I would be more than happy to contribute to make it happen!

[JohelEGP]

Ability to opt-out off generating tests (release builds)

Sounds like this has ties to #942, a more general feature.

Death tests & crash resilience

Today only GTest supports this (AFAIK), sometimes its necessary to test that a program terminates, if, say, a contract is violated.

μt does, too, with fatal, throws, and aborts.

parametrize tests

μt just uses the C++ language.
I suppose a @test in a type should be a non-template friend defined out-of-line.
Then you can more easily test templated types.

[DyXel]

μt just uses the C++ language.
I suppose a @test in a type should be a non-template friend defined out-of-line.
Then you can more easily test templated types.

Yeah, I can see this being a nice direction to take: Write as much test code as possible in idiomatic C++, sprinkle some metafunctions on top to extend to types and other advanced kind of tests!