Transition benchmarks to https://docs.rs/bencher/latest/bencher/

[alexpyattaev]

Problem

Our benchmarking setup relies on nightly compiler which is not what we use for actual builds of agave. Using the same compiler (stable) for releases and benches would be preferable

Proposed Solution

• Transition tests to https://docs.rs/bencher/latest/bencher/

[alessandrod]

criterion is garbage for anything that isn't purely cpu bound fyi

[alexpyattaev]

What would you approve of? Divan?

[puhtaytow]

Divan?

for consistency reasons used should be one but how many of crates here could benefit from tighter i/o measurements?

--edit--
after quick google it seem like divan won't produce reports and comparisons

[alexpyattaev]

Yes divan is intended to be quite simple and barebones. This, arguably, could be a good thing.

After some discussion with @alessandrod on slack, some ideas were:

• If something does IO it probably should not be using criterion, might be best for it to be an example that happens to note time taken to go over some operations rather than a bench.
• If something is purely CPU bound it might make sense to transition to criterion so we can run the benches on stable and get nice reports.

[steviez]

If something is purely CPU bound it might make sense to transition to criterion so we can run the benches on stable and get nice reports.

Any more details you can elaborate on with "might" here ? I see @puhtaytow has opened a bunch of PR's, but I think it would be good to get agreement on path forward before anyone opens any more PR's

[alexpyattaev]

I think the final word should be with @alessandrod as he is the suubject matter expert.

[puhtaytow]

I did my best to pick crates that are mostly or entirely cpu bound, but 💯 would be better to have a list before moving forward.