Benchmark References?

[m-reza-rahman]

Is there a benchmark I can credibly reference that shows performance differences between Quarkus, Helidon, Micronaut, Spring Boot and Open Liberty? I do see a few community ones. Are those good enough? Is there one folks here would recommend in particular?

For example the Open Liberty folks have a few benchmarks that I would not hesitate to share as I know they were done with reasonable professionalism and are likely maintained: https://developer.ibm.com/articles/6-reasons-why-open-liberty-is-an-ideal-choice-for-developing-and-deploying-microservices/.

[maxandersen]

Look at techempower. It has its flaws but at least has somewhat consistent comparison.

But we always recommend to do your own comparisons.

[m-reza-rahman]

Is this the correct link: https://www.techempower.com/benchmarks/?

While I certainly understand benchmarks are inevitably tied to the particulars of an application and methodology, it would be very valuable to have something easy to understand that can be referenced quickly (particularly for decision makers in the cloud native Java space). I'll share the techempower for now. Hopefully it is sufficient for folks that would ask about benchmarks.

As food for thought, this seems pretty good except for the fact that it lacks polish, so probably not that credible to the audience I have in mind: https://www.baeldung.com/spring-boot-vs-quarkus.

[maxandersen]

That specific article from Baeldung is NOT properly done. The tool used (jmeter) is notoriously known for not measuring latency properly thus please don't take the results in that article as proof for anything.

Matt Raible also published an article where we also pointed out broken assumptions in his testing. He published a corrected one. Still waiting for Baeldung to do the same but they claim the article is not meant as a full performance test.

If you do want to know our views on performance testing see https://quarkus.io/guides/performance-measure

https://quarkus.io/blog/runtime-performance/

https://youtu.be/xdG8b9iDYbE

[m-reza-rahman]

Thanks for the details. I’ll take a look.

[holly-cummins]

We're developing a new comparison application which is designed to be simple enough to run locally, but also to be run in a lab with proper tooling and hardware. It's still very much a work in progress, but you can see what we've got now at https://github.com/quarkusio/spring-quarkus-perf-comparison. The results from the performance lab are auto-published to https://github.com/quarkusio/benchmarks?tab=readme-ov-file#spring-quarkus-perf-comparison.

We're planning to use this as our new performance reference, following a period of community and expert review.