-
-
Notifications
You must be signed in to change notification settings - Fork 11
Description
Unit tests are faster to run (we just run cecilifier) than output based ones (in which we compile and run the cecilified code) but they come with some disadvantages, not guaranteeing correctness being the main one: they just ensure that the generated code matches some patterns (and I've found more cases than I'd like to admit in which these tests were checking for incorrect patterns).
By the other hand, output based tests ensures correctness by:
- Comparing the output of the execution of the assembly produced by the cecilified code against the expected output
- Running il verification on the produced assembly
Our goal is to find a middle ground in which the development workflow does not get too much impacted due to a large number of slow tests and at the same time having the confidence that the tests are accurate.
To that end we propose:
- Introduce another kind of test equivalent to the merge of Unit + Output based tests
- Run only the
unittest part of these tests in the developer workflow - In CI run both parts of these tests
- Convert all
output based teststo this new model - Consider converting
unit teststo this new model when one need to be maintained.
In the poc implementation in branch test-env one can control whether a piece of test code will be executed or not either by setting CECILIFIER_TESTS_VERIFY_OUTPUT environment variable or by adding AlwaysRunOutputBasedTestsAttribute to a Test method or to a TestFixture.
We can use that to have a couple of smoke tests attributed with AlwaysRunOutputBasedTestsAttribute (maybe having at least one test method per C# syntax / Cecilifier feature) and setting the environment variable in CI (maybe on specific branches?) enabling quick development cycle and correctness at the cost of small chance of only detecting some breakages when code is pushed to repository.