Runtime cleanups split out from the type hinting work #89
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Benchmarks | |
| on: | |
| pull_request: | |
| branches: | |
| - master | |
| # Cancel in-flight runs when newer commits are pushed to the same PR. | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| benchmark: | |
| name: Benchmarks | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v7 | |
| - name: Install uv and Python 3.14 | |
| uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 | |
| with: | |
| python-version: "3.14" | |
| enable-cache: true | |
| cache-dependency-glob: "pyproject.toml" | |
| - name: Install dependencies | |
| run: uv sync | |
| # On PRs: run benchmarks alternating between master code and PR | |
| # code (3 runs each), then check that the PR is not *consistently* | |
| # slower than master. Single runs are far too noisy to gate on: | |
| # process-level effects (hash randomization, memory layout, runner | |
| # load) shift individual results by 20-40% on identical code. | |
| # Interleaving the runs and comparing the fastest PR run against | |
| # the slowest master run makes the guard robust against that noise | |
| # (see bench/compare_runs.py). | |
| - name: Run benchmarks | |
| env: | |
| # Fixed hash seed: hash randomization moves dict/set benchmark | |
| # timings between processes, which would make the master and | |
| # PR runs incomparable. | |
| PYTHONHASHSEED: "0" | |
| run: | | |
| git fetch origin master | |
| for i in 1 2 3; do | |
| # Run benchmarks with master code as baseline. Tolerate | |
| # failures: the PR's benchmarks may exercise APIs that don't | |
| # exist on master yet. | |
| git checkout origin/master -- observ/ | |
| uv run --no-sync pytest bench \ | |
| --benchmark-only \ | |
| --benchmark-save=master$i \ | |
| --benchmark-sort=mean || true | |
| # Run benchmarks on PR code | |
| git checkout HEAD -- observ/ | |
| uv run --no-sync pytest bench \ | |
| --benchmark-only \ | |
| --benchmark-save=branch$i \ | |
| --benchmark-sort=mean | |
| done | |
| - name: Compare benchmarks | |
| run: | | |
| uv run --no-sync python bench/compare_runs.py \ | |
| --baseline '.benchmarks/*/*_master?.json' \ | |
| --branch '.benchmarks/*/*_branch?.json' \ | |
| --threshold 0.10 |