Skip to content

Recommended workflow for storing results in separate repo? #774

Open
@maxalbert

Description

@maxalbert

This is my first post here, so let me start by saying a big thank you to all the asv developers for all their work, time and effort!

I'd like to use asv for one of my projects, but I'm a little unclear on the best practice of storing and publishing benchmark results. Regarding the repo structure, I'd like to follow the example of numpy:

However, I'm not quite clear on the recommended way of running the benchmarks and committing the output of each run to the separate results repo. Especially also in the context of running benchmarks as part of CI (or at least regularly) and committing/publishing results automatically.

From what I can see, currently the only option to specify where results are stored is via the results_dir setting in asv.conf.json, but this value is interpreted relative to the asv.conf.json file. Does this mean the benchmarks should always be run in the main repo, and then the results need to be copied manually to the results repo? This seems ok, albeit a little clunky, so I'm wondering if there is a more elegant way.

It would be useful to have a --results-dir command line flag, or a ASV_RESULTS_DIR environment variable to specify the location where the output should be stored. For example:

# Option 1
asv run --results-dir=./results

# Option 2
export ASV_RESULTS_DIR=./results
asv run

Does either (or both) seem like a good idea? If so, I'm happy to open a pull request to add this to asv. Or is there a better/different way?

It would also be good to add a section to the docs on recommended workflows for storing and publishing results, and how to hook it up with CI. I'm happy to take a first stab at this if someone can provide me some more information on how the various projects (numpy/scipy, pandas, astropy) do this.

Many thanks for any suggestions!

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionTriaged as a question, not requiring action after answering

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions