Open
Description
E.g. on cf4695a, test with
-p '/Adversarial chains win if checked with relaxed parameters/'
--quickcheck-replay="(SMGen 11912988973458833518 16836906162971932891,5)"
(depends on the speed of your machine).
In contrast to other flaky test, this test is inherently flaky by construction:
- For Genesis, we have a generator for leader schedules satisfying certain constraints implied by Praos/Genesis, for example "there are k blocks within a stability window".
- The test that is failing above is about making sure that we at least sometimes generate test cases that only narrowly meet the constraints, to ensure good coverage. For example, this means that we sometimes generate test cases where have exactly k blocks in a stability window.
- We do so by eg telling the generator to generate a schedule satisfying "k blocks within a stability window", but testing for "k+1 blocks within a stability window" (ie testing for a mutation of the constraints). We then repeat this until we find an example where this test fails.
- Usually, this happens quite quickly. However, in theory, it can take almost arbitrarily long (run time should roughly follow a geometric distribution), so we have a time limit of 20 seconds, as this seems like a better behavior than CI taking a very long time. This time limit was exceeded in the failure above.
- I just ran that test locally with the time limit removed. It finished after 48 seconds, so the property is still true. We could decide to bump the timeout to eg 60 seconds to make the test less flaky if we see this repeatedly in the future.