AirspeedVelocity.jl for benchmark in CI

[Moelf]

https://discourse.julialang.org/t/easy-github-benchmarking-with-new-airspeedvelocity-jl/129327

I noticed we have a separate repo in https://github.com/graeme-a-stewart/JetReconstructionBenchmarks.jl/ , I guess we can checkout that repo during CI

@graeme-a-stewart do you think this is a good idea?

[m-fila]

I don't know, those benchmarks are quite heavy and I feel were written to make more comprehensive comparisons with different variants and fastjet.
Perhaps something like simplified instrumented-reco would be more appropriate for CI to early detect regressions. And easier to put in BenchmarkTools layout

[graeme-a-stewart]

This looks very interesting - we should investigate. JetReconstructionBenchmarks.jl is way too much. The value here is to quickly identify regressions in the main use cases that we have. If you look at examples/benchmark.sh those are the baseline measurements to take to show that none of the algorithms regressed.

[m-fila]

I took a look and compared Airspeed velocity and a few alternatives but forgot to post my results. IIRC with noise on GitHub runner it was impossible to see a difference (or sometimes it even reported the faster code to be 5% slower) between find_min using LoopVecotrization and #151 for the benchmarks as in examples/benchmark.sh but perhaps setting a much bigger sample could help?? I'll revisit this later

[m-fila]

I checked:

• AirspeedVelocity - Bechmark in CI with AirspeedVelocity m-fila/JetReconstruction.jl#4 - I think it's quite nice although young. The action runs code from PR and upstream and make a comparison. The results can be posted either as job summary or comment, but each julia version creates a separate comment so it's quite spammy and having different architectures overwrites the comment - job summary doesn't have these problems but is less visible.
• BenchmarkCI.jl - Bechmark in CI with BenchmarkCI m-fila/JetReconstruction.jl#5 - this seems to be abandoned, also works only on x86_64 for whatever reason. The reports produced are posted as PR comments and are very verbose
• github-action-benchmark - Benchmark in CI with action-benchmark m-fila/JetReconstruction.jl#6 - This is by far the most mature one out of the comparison and also not Julia-specific. The difference here is that the results for commits are stored in the repository and the action compares benchmark with the last one stored from repository - I don't like this conceptually because then we are comparing results from different workers. But it's very customizable and allows to print reports either as comments or in action summary or even set a threshold and post comment or fail job if the performance drops below the threshold

At least AirspeedVelocity is expecting the benchmarks to be run to be already present in the master so I did this in my fork. The bechmarks were cases from examples/benchmark.sh adapted to use BenchmarkTools.jl - I'll still have to check if changing the number of samples/tuning allows to see a few percent difference in the performance.