Existing sample/recipe for integrating asv with github

[yarikoptic]

Backstory aside - is anything aware of an existing use of asv integrated with GitHub?

What I envision is having a container image with eg buildbot and asv installed, to be deployed on a dedicated box, configured to monitor/report back on PRs to GitHub about impact on benchmarks.

May be there is a service which already does something like that?

I will look into setting up such one, but wanted to make sure I am not reinventing the wheel or making it square.

[saraedum]

@yarikoptic did you manage to build such a service?

[yarikoptic]

Nope, but my colleagues set it up to just run asv to get an estimate for performance impact comparing against master in PRs. See e.g. our datalad github actions workflow: https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml and the actual helper https://github.com/datalad/datalad/blob/master/tools/ci/benchmark-travis-pr.sh . Eventually we just might look into establishing self-hosted runner for actions: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners so we get more stable estimates

[yarikoptic]

note - that we nohow report back overall effect, this action is merely for manual investigation if there is a suspicion that PR might affect performance. Would be nice to report back to action state if e.g. any benchmark became significantly slower. we haven't looked into that yet

[saulshanabrook]

In case it helps anyone, I just set up asv to run a job on each PR which compares it against the branch it is being merged into. It doesn't save the results anywhere, but anyone can look at the logs there to see if affects the performance.

My github action looked like this:

  benchmark:
    name: benchmark
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v2
        with:
          python-version: "3.10"
      - uses: actions/checkout@v3
      - run: git fetch origin $GITHUB_BASE_REF:base $GITHUB_REF:pr
      - run: pip install asv virtualenv
      - run: asv machine --yes
      - run: asv continuous base pr --strict -e

[yarikoptic]

in https://github.com/datalad/datalad/blob/master/.github/workflows/benchmarks.yml we also fail the workflow if any benchmarks gets significantly worse:

    - name: Fail if any benchmarks have slowed down too much
      run: |
        ! asv compare --factor 1.2 --split refs/bm/merge-target refs/bm/pr | grep -q "got worse"

[JuroOravec]

Hi, I managed to integrate asv with django-components in django-components/django-components#999.

Our integration covers these use cases:

• Performance report on pull requests.
• Benchmarking the package across releases.
• Displaying performance results on a website.

Since I have this info so fresh in my mind, and since it was so tricky to get it working (just the Github workflow integration took me a day), I'm up for writing this down as a tutorial, so that others don't have to reinvent the wheel.

I'm thinking there could be a "Guides" section in the docs, and the use cases above could be pages in the "guides" section.

@HaoZeke @mattip What do you think?

---

Related to this, I made some changes as I was trying to get things working:

I had to monkeypatch from outside these bugs:

• BUG: "OSError: [Errno 7] Argument list too long" asv_runner#45
• fix: timeraw with params doesn't use the returned script string asv_runner#44

These are things that were not clear or missing from the docs, so I updated them and opened PRs for them:

• docs: document pitfalls of changing benchmark_name benchmark attribute #1470
• docs: document the version field in asv.conf.json.rst #1467

I'd be great if we could get these bug fixes and docs PRs resolved as soon as possible, so that the guides articles wouldn't have to include the workarounds. Because the workarounds don't look pretty (see example)

---

P.S.: For completeness, here are other suggestions I made that would make the interaction with asv smoother and more flexible:

• Feature request: Allow to pass env vars to timeraw #1471
• Feature request: @benchmark decorator #1469
• Feature request: Allow to override the files generated with asv publish #1473

[HaoZeke]

Hi, I managed to integrate asv with django-components in django-components/django-components#999.

Our integration covers these use cases:

• Performance report on pull requests.
• Benchmarking the package across releases.
• Displaying performance results on a website.

Since I have this info so fresh in my mind, and since it was so tricky to get it working (just the Github workflow integration took me a day), I'm up for writing this down as a tutorial, so that others don't have to reinvent the wheel.

I'm thinking there could be a "Guides" section in the docs, and the use cases above could be pages in the "guides" section.

@HaoZeke @mattip What do you think?

Related to this, I made some changes as I was trying to get things working:

I had to monkeypatch from outside these bugs:

• BUG: "OSError: [Errno 7] Argument list too long" asv_runner#45
• fix: timeraw with params doesn't use the returned script string asv_runner#44

These are things that were not clear or missing from the docs, so I updated them and opened PRs for them:

• docs: document pitfalls of changing benchmark_name benchmark attribute #1470
• docs: document the version field in asv.conf.json.rst #1467

I'd be great if we could get these bug fixes and docs PRs resolved as soon as possible, so that the guides articles wouldn't have to include the workarounds. Because the workarounds don't look pretty (see example)

P.S.: For completeness, here are other suggestions I made that would make the interaction with asv smoother and more flexible:

• Feature request: Allow to pass env vars to timeraw #1471
• Feature request: @benchmark decorator #1469
• Feature request: Allow to override the files generated with asv publish #1473

A guides section would be fantastic. I've been meaning to bring the ASV documentation into the modern style (e.g. NumPy, SciPy, anything following diataxis basically) of having user stories, API details and tutorials, but haven't gotten the time to focus on it yet. This would make a great guide, and I look forward to reviewing it as soon as it is ready