Use a set difference to release stale deps in cleanup_deps #101
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Benchmarks | |
| on: | |
| pull_request: | |
| branches: | |
| - master | |
| # Cancel in-flight runs when newer commits are pushed to the same PR. | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| benchmark: | |
| name: Benchmarks | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v7 | |
| - name: Install uv and Python 3.14 | |
| uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 | |
| with: | |
| python-version: "3.14" | |
| enable-cache: true | |
| cache-dependency-glob: "pyproject.toml" | |
| - name: Install dependencies | |
| run: uv sync | |
| # On PRs: run benchmarks alternating between master code and PR | |
| # code (3 runs each), then check that the PR is not *consistently* | |
| # slower than master. Single runs are far too noisy to gate on: | |
| # process-level effects (hash randomization, memory layout, runner | |
| # load) shift individual results by 20-40% on identical code. | |
| # Interleaving the runs and comparing the fastest PR run against | |
| # the slowest master run makes the guard robust against that noise | |
| # (see bench/compare_runs.py). | |
| - name: Run benchmarks | |
| env: | |
| # Fixed hash seed: hash randomization moves dict/set benchmark | |
| # timings between processes, which would make the master and | |
| # PR runs incomparable. | |
| PYTHONHASHSEED: "0" | |
| run: | | |
| git fetch origin master | |
| for i in 1 2 3; do | |
| # Run benchmarks with master code as baseline. Tolerate | |
| # failures: the PR's benchmarks may exercise APIs that don't | |
| # exist on master yet. | |
| git checkout origin/master -- observ/ | |
| uv run --no-sync pytest bench \ | |
| --benchmark-only \ | |
| --benchmark-save=master$i \ | |
| --benchmark-sort=mean || true | |
| # Run benchmarks on PR code | |
| git checkout HEAD -- observ/ | |
| uv run --no-sync pytest bench \ | |
| --benchmark-only \ | |
| --benchmark-save=branch$i \ | |
| --benchmark-sort=mean | |
| done | |
| # The 25% threshold makes this guard a tripwire for gross | |
| # accidental regressions (an accidental copy in a hot path, an | |
| # algorithmic slip), not a precision instrument. Even with the | |
| # interleaved fastest-vs-slowest gate above, whole runs of | |
| # identical code on hosted runners have been observed to differ | |
| # by 10-30% (the same commit went fail/fail/pass across three | |
| # attempts at a 10% threshold, flagging code paths its diff never | |
| # touched). Gating below that noise floor just breeds re-run | |
| # rituals. Regressions too small to trip this guard should be | |
| # measured deliberately instead: repeated local runs of the bench | |
| # suite on an idle machine. | |
| - name: Compare benchmarks | |
| run: | | |
| uv run --no-sync python bench/compare_runs.py \ | |
| --baseline '.benchmarks/*/*_master?.json' \ | |
| --branch '.benchmarks/*/*_branch?.json' \ | |
| --threshold 0.25 |