Description
Opening this after offline discussion of this PR with @jrevels: JuliaCI/BenchmarkTools.jl#129
I'm copy-pasting some of that discussion here:
Here's the context for this:
I'm trying to define a BenchmarkGroup for https://juliaci.github.io/PkgBenchmark.jl. But in my existing benchmark file, before using PkgBenchmark, I'm subtracting the results of two @benchmark runs (one where I do my operation, and one where I don't) in order to get the exact time for just my operation. Is there any way I can reproduce that type of logic with the @benchmarkable structure I need for the BenchmarkGroup?
Here's my existing logic that I'm trying to convert to be a BenchmarkGroup:
https://github.com/JuliaMath/FixedPointDecimals.jl/blob/3e7da851ea9caa0e267c21e0bb067ae32ee9ad77/bench/decimal-representation-comparisons.jl#L96-L99
What we want is to be able to subtract/diff the results of two benchmark runs as part of generating the final BenchmarkResult from PkgBenchmark.jl. In that PR, JuliaCI/BenchmarkTools.jl#129, I suggested a feature to allow adding this subtraction step to the "computation graph" built for the BenchmarkGroup:
bbase = @benchmarkable $fbase()
bbench = @benchmarkable $fbench()
SUITE["bench"] = bbench - bbase
But after talking to @jrevels, we think that this kind of "post-processing" doesn't really belong in the BenchmarkGroup structure. Instead, it's something that you should do after you have your results, no different than a judge step, or taking a ratio would be.
However, currently PkgBenchmark doesn't support any kind of post-processing. It simply runs the BenchmarkGroup and returns the results. Could we consider accepting an additional argument that allows performing arbitrary post-processing on the results?
I'm imagining maybe taking a callback that accepts a BenchmarkResults and returns a new BenchmarkResults. Or maybe takes a BenchmarkGroup (after it's been run) and returns a new BenchmarkGroup. This callback param could be added to benchmarkpkg and judge.
What do you think? :) If you think this makes sense, i'm happy to create and send the PR!! :)