Create hyperfoil yaml rather than running wrk directly

[edeandrea]

This also produces the nice html reports, which are also downloaded by the run-benchmarks.sh script :

Fixes #42

[edeandrea]

@holly-cummins / @franz1981 Let me know what you think of this.

I wasn't sure if I should modify the stress.sh script at all for this. The wrk output on the terminal is much better than run@hyperfoil.

[franz1981]

I am all in to do this tbh as it allow more fine grain analysis, especially if we add more endpoints.

More; I would create two hf workload:

• like this one you created
• using ramupUp and fixed rate (with no variance)

This would help to:

1. Observe the behavior of the "always" one in term of rampup time and Mac throughput
2. Run the second one parameterizing it with what is observed in the first one

This would allow comparing Quarkus and Spring with the same sustainable throughput (to the slower one - which tend to be Spring so far) and appropriate rampup to not overload it.

But what we can compare?
Since the throughput is the control variable, is fair to compare all the other metrics collected e.g. CPU used, allocation rate, distribution of latencies.
In this way, none can object that the two frameworks are stressed by the same load.

One note about it: clearly such a test configuration depends by which resource is mostly utilized till saturation (or near to): if we measure with N TPS that framework S runs w 80% of available CPU time, but in a subsequent test, we double the CPU cores, the test is not granted anymore to be CPU bound - and we need to verify it.

In relation to the throughput output used - I can check with my team mates what we can do to help/improve w it but right now is already possible to parse the json and computing it ourself.

[holly-cummins]

@holly-cummins / @franz1981 Let me know what you think of this.

I wasn't sure if I should modify the stress.sh script at all for this. The wrk output on the terminal is much better than run@hyperfoil.

I think the two goals of the stress.sh script are:

• pleasant terminal output
• human-readable script with low cognitive load (ie as few unfamiliar dependencies as possible)

If those were in opposition, it would be tricky, but if both of those are better with wrk, I suggest we stick with wrk. We could add some comments in the script explaining the design choice.

[franz1981]

Re which one is better @holly-cummins we (perf team) could improve the run cmd output with ease or just NOT reuse the same script as the purpose of the two tests is not exactly the same (we trade the cognitive load for accuracy and correctness)

[edeandrea]

@franz1981 the run@hyperfoil script doesn't even calculate throughput. I have to manually compute it from the results in all.json.

It would be nice if run@hyperfoil could just do that.

[franz1981]

Iirc there is a way to print the result in csv of the avg throughput per each phase

[edeandrea]

FYI I opened https://github.com/Hyperfoil/jbang-catalog/issues/14 to help with the output of run@hyperfoil. I think for now we should leave stress.sh alone.

[franz1981]

Yep @edeandrea you found the output I was mentioning. Consider that hyperfoil is using a saved csv formatted file for each run with it, so it is surely there.
Or..I could we could just add it to the json and leave the output silent too.

[edeandrea]

Or..I could we could just add it to the json and leave the output silent too.

But for stress.sh we want the output front and center.