Hey! First of all, thanks for providing this nice package, it makes comparing of several commits really easy :)
I've tried to judge benchmarks for several julia versions. Unfortunately, this does not work due to
|
juliacmd = benchmarkconfig.juliacmd |
|
juliacmd = `$(Base.julia_cmd(juliacmd[1])) $color $(juliacmd[2:end])` |
which takes the sysimage from the parent process.
julia> Base.julia_cmd(`julia-1.5`)
`julia-1.5 -Cnative -J/Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib -g1`
I'd argue, that a user supplied command should be taken as is without changes which depend on the parent command whatsoever.
Maybe it would be better to default the juliacmd in BenchmarkConfig to Base.julia_cmd() and just do
juliacmd = benchmarkconfig.juliacmd
in the actual benchmark function?
Hey! First of all, thanks for providing this nice package, it makes comparing of several commits really easy :)
I've tried to judge benchmarks for several julia versions. Unfortunately, this does not work due to
PkgBenchmark.jl/src/runbenchmark.jl
Lines 249 to 250 in baa8bee
which takes the sysimage from the parent process.
I'd argue, that a user supplied command should be taken as is without changes which depend on the parent command whatsoever.
Maybe it would be better to default the
juliacmdin BenchmarkConfig toBase.julia_cmd()and just doin the actual benchmark function?