Description
This is my first post here, so let me start by saying a big thank you to all the asv developers for all their work, time and effort!
I'd like to use asv for one of my projects, but I'm a little unclear on the best practice of storing and publishing benchmark results. Regarding the repo structure, I'd like to follow the example of numpy:
- Keep the benchmarking code in a benchmarks/ subfolder in the main repo.
- Store the results in a separate repository (in numpy's case: https://github.com/pv/numpy-bench).
However, I'm not quite clear on the recommended way of running the benchmarks and committing the output of each run to the separate results repo. Especially also in the context of running benchmarks as part of CI (or at least regularly) and committing/publishing results automatically.
From what I can see, currently the only option to specify where results are stored is via the results_dir setting in asv.conf.json, but this value is interpreted relative to the asv.conf.json file. Does this mean the benchmarks should always be run in the main repo, and then the results need to be copied manually to the results repo? This seems ok, albeit a little clunky, so I'm wondering if there is a more elegant way.
It would be useful to have a --results-dir command line flag, or a ASV_RESULTS_DIR environment variable to specify the location where the output should be stored. For example:
# Option 1
asv run --results-dir=./results
# Option 2
export ASV_RESULTS_DIR=./results
asv run
Does either (or both) seem like a good idea? If so, I'm happy to open a pull request to add this to asv. Or is there a better/different way?
It would also be good to add a section to the docs on recommended workflows for storing and publishing results, and how to hook it up with CI. I'm happy to take a first stab at this if someone can provide me some more information on how the various projects (numpy/scipy, pandas, astropy) do this.
Many thanks for any suggestions!