Plugins for the most common Continuous Integration Systems

[adamsitnik]

I would like to start a discussion, it is not a complete vision yet.

Problem: in terms of Continuous Integration the software performance is usually ignored, or in best case it is not measured properly. Most developers do the profiling and optimizations when the clients start to complain. The performance is not considered and treated as a feature.

As a user I would like to know when code changes are decreasing performance.

The basic idea is to have the results of the benchmarks compared with previous results. If the difference is big enough, then the CI should go red as it would with a broken test.

[smudge202]

I don't think there's a good way of doing this due to persistence (or lack thereof). I've never come across a CI pipeline that needs (or even has access to) previous builds.

The only way I can imagine it working is if benchmarks are committed to source control. CI would fail the build if the difference between the committed benchmark and the CI benchmark was outside of a given tolerance, requiring a user to run the benchmarks and commit the updated results if they're happy with them. However, such a process would imply that the developer run the benchmarks locally which the CI would then compare to running them on a server, and there are very few if any situations where a developers' local machine is similar to a CI build agent, making the benchmarks either unlikely to match, or require such a huge tolerance of change that it be pointless.

I'd love to have such a feature, I just can't imagine how it could work. How do you envision it, @adamsitnik ?

[mattwarren]

The basic idea is to have the results of the benchmarks compared with previous results. If the difference is big enough, then the CI should go red as it would with a broken test.

Yeah the persistence side of things is tricky and I'm not sure how or if BenchmarkDotNet can handle that. The best thing I can think of is a plugin that knows where to look for previous results and can compare the current run against those.

But it would probably be the responsibility of the CI infrastructure to ensure that previous Benchmark results were stored somewhere after BenchmarkDotNet had finished running.

Just for reference, at the moment this is the type of things that BenchmarkDotNet can help you with, inside a regular unit test (I know these aren't strictly unit tests, more like integration tests, but it's very easy to re-use the existing tooling (XUnit in this case)):

1. Check that Benchmark A runs faster than Benchamark B
2. Ensure that a Benchmark runs within a certain time tolerance

So it's fairly primitive, but it's something at least.

[smudge202]

Perhaps two or three extensibility points could help achieve this, @mattwarren. For example, let's say BDN allows consumers to write their own benchmark result importers as you've suggested. As a consumer, I would want to utilise BDN API to specify that the imported Benchmark should be compared to the current benchmark to determine pass/fail of test.

Similarly, I'd want to be able to override the comparison process so that any default tolerance mechanisms can be superseded.

We can already override the logger mechanisms as part of export, but (correct me if I'm wrong), what we don't have is a Benchmark DTO that we could simply import/export. Instead, in order to export and persist benchmarks, I think we'd have to extrapolate (deserialize) the content of the log in order to build up such a DTO. Essentially, it would be better (hopefully not confusing?) if both log and dto were available for export, and the ability to import said DTO's for comparison existed?

[adamsitnik]

I have just found a dotTrace plugin for Team City that does something very similar to what I wanted to get: link

"It’s also possible to compare measured values against values from previous successful builds."

I know that the persistence side would be tricky. Do you think that custom plugins don't have access to CI's database or can not create it's own during installation process?

I have some initial idea for a simple proof of concept for Team City.

• TC can run .NET executable with given parameters (Runner type: .NET Process Runner).
• It is possible to configure TC build to publish artifacts (it could be any files that our process have created, in our case benchmark results) & get address of the latest succesfull build artifacts.

We could create new BDN runner that would extend the existing in following way:

• Download the files from given http address (Team City artifacts are just hosted by the web server) passed to our new runner in parameters.
• Parse the files (as in given unit test examples) to get the exact results
• Compare old results with new results
• Fail the process if the difference is bigger than given treshold (could be also passed in parameters)

[TimirP]

I'm looking for something so that I can run BenchmarkDotNet as Azue WebJob or though Azure Build Pipeline. I would like to benchmark performance of my application running on Azure and Accessing azure resources. I can run it from my development machine.

[oformaniuk]

I was looking for Github Actions support and found a PR to Continuous Benchmark action: benchmark-action/github-action-benchmark/pull/34. Would leave it here for further tracking.

[tomatopunk]

Hello @adamsitnik

The tail in 2020,Is there any good product or solution recommended by the community

As described in your post,,I would like to know when code changes are decreasing performance.

As convenient as a unit test pipeline