Benchmarking

[astutecat]

I was speaking to the developer of benchee (https://github.com/PragTob/benchee) about combining benchee and propcheck, wondering if it was possible to somehow hook benchee into a property-based test case so I can get tasty tasty statistics.

Given how property-based testing works it would generate some nice information about whether or not the performance of the thing under test scale in a linear fashion or not without having to write and maintain two sets of generators.

Any ideas?

[evnu]

Interesting idea! An implementation has to ensure reproducibility, though, which can be complicated (see proper-testing/proper#165, where setting the seed is discussed). Additionally, the problem size must be considered. I guess that proper use of sized/2 could be used for this.

[astutecat]

Good point - I didn't consider the reproducibility of the benchmarks, which as you say makes this definitely more non-trivial.

[alfert]

Benchmarking via properties is a mis-use of PropEr (and other PBT tools) due to its random nature. In principal, you can use deterministic (via a common seed) data generators for load creation, but I would suggest to use DataStream instead of PropCheck since it is more tailored to be used outside of a property based testing approach.