Can we run the benchmarks multiple times and report mean and sd?

[SteveBronder]

Description

I noticed that when running the gold models like SIR multiple times it can can anywhere from 100 seconds to 150 seconds on my machine with the develop branch. That's a pretty big variance! I think for the performance benchmarks we report back via Jenkins it would be a good idea to run the models 15-20 times and then report the mean and variance of the performance benchmark for each model.

@serban-nicusor-toptal can you point me to the code that the Jenkins uses to set off the performance tests? I can also modify the compare-hash.py file to report the mean and sd. If I remember right you had something like this in the works?

Current Version:

v2.20.0

[bob-carpenter]

While you're at it, would you mind adding something to the output to signal whether larger or smaller numbers are better. Thanks.

[serban-nicusor-toptal]

@bob-carpenter There's a PR for that: stan-dev/performance-tests-cmdstan#24

[serban-nicusor-toptal]

@SteveBronder
From math perspective triggering a performance run goes in this order.

1. Math triggers Stan build: https://github.com/stan-dev/math/blob/develop/Jenkinsfile#L271
2. Stan triggers Cmdstan build: https://github.com/stan-dev/stan/blob/develop/Jenkinsfile#L194
3. Cmdstan triggers Performance tests: https://github.com/stan-dev/cmdstan/blob/develop/Jenkinsfile#L140

Note the pr variables passed around.

Tests are running here in the Jenkinsfile:

1. Compare against target branch
2. Known-Good
3. Shotgun

If I can help you with anything let me know

[bob-carpenter]

Thanks so much. I can never keep my signs or numerators/denominators straight in anything.