Making perf_bench useful in CI

[jepler]

I have been investigating how to make the perf bench useful in CI.

My understanding is that times in github hosted CI are too variable to be useful.

One possibility is to use valgrind's "callgrind" tool to count instructions executed.

I have prototyped this. It gives output like the following:

$ tools/ci.sh cachegrind_report 
Reference:  WIP [cb749fba64]

Comparison: WIP [e3c1fd117a]

bm_chaos               :   375781417 ->   375780854  -563 -0.000%
bm_fannkuch            :   588512242 ->   588512206   -36 -0.000%
bm_fft                 :   436038154 ->   436038100   -54 -0.000%
bm_float               :   262278666 ->   262278612   -54 -0.000%
bm_hexiom              :   188126370 ->   188126352   -18 -0.000%
bm_nqueens             :   183939540 ->   183939485   -55 -0.000%
bm_pidigits            :   213666469 ->   213666433   -36 -0.000%
bm_wordcount           :  1084635964 ->  1084635856  -108 -0.000%
core_import_mpy_multi  :    60675776 ->    60675182  -594 -0.001%
core_import_mpy_single :     4749648 ->     4749630   -18 -0.000%
core_locals            :  1520561967 ->  1520561931   -36 -0.000%
core_qstr              :    21683826 ->    21683809   -17 -0.000%
core_str               :   208841678 ->   208846798 +5120 +0.002%
core_yield_from        :    89220275 ->    89220239   -36 -0.000%
misc_aes               :   285696790 ->   285696754   -36 -0.000%
misc_mandel            :   192274623 ->   192274568   -55 -0.000%
misc_pystone           :   169430902 ->   169430866   -36 -0.000%
misc_raytrace          :   170968294 ->   170968239   -55 -0.000%
viper_call0            :    81063901 ->    81063865   -36 -0.000%
viper_call1a           :    82887493 ->    82887457   -36 -0.000%
viper_call1b           :    94014082 ->    94014046   -36 -0.000%
viper_call1c           :    93710120 ->    93710084   -36 -0.000%
viper_call2a           :    84104539 ->    84104503   -36 -0.000%
viper_call2b           :   102447683 ->   102447647   -36 -0.000%
Max delta 5120

The code is the same, so the differences (ranging from -594 to +5120) are run-to-run differences observed on my machine. Because it's simply valgrind running against the unix standard build, there is no difference due to CPU performance; however, the number reported is just instruction cycles, which is different than actual performance.

My thought is that this might be useful, and would find gross performance regressions, but it would probably not be a substitute for double checking with a real perf benchmark run on hardware.

Is this worth pursuing to PR'able status?

[jepler]

.. and should something similar be done for internal_bench?