Release 0.10.0 #25

Workflow file for this run

.github/workflows/benchmark.yml at 67309b0

	name: Benchmarks

	on:
	pull_request:
	branches:
	- master

	# Cancel in-flight runs when newer commits are pushed to the same PR.
	concurrency:
	group: ${{ github.workflow }}-${{ github.ref }}
	cancel-in-progress: true

	jobs:
	benchmark:
	name: Benchmarks
	runs-on: ubuntu-latest
	timeout-minutes: 20
	steps:
	- uses: actions/checkout@v7

	- name: Install uv
	uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0

	- name: Set up Python
	uses: actions/setup-python@v6
	with:
	python-version-file: "pyproject.toml"

	- name: Install dependencies
	run: uv sync

	# On PRs: run benchmarks alternating between master code and PR
	# code (3 runs each), then check that the PR is not consistently
	# slower than master. Single runs are far too noisy to gate on:
	# process-level effects (hash randomization, memory layout, runner
	# load) shift individual results by 20-40% on identical code.
	# Interleaving the runs and comparing the fastest PR run against
	# the slowest master run makes the guard robust against that noise
	# (see bench/compare_runs.py).
	- name: Run benchmarks
	env:
	# Fixed hash seed: hash randomization moves dict/set benchmark
	# timings between processes, which would make the master and
	# PR runs incomparable.
	PYTHONHASHSEED: "0"
	run: \|
	git fetch origin master

	for i in 1 2 3; do
	# Run benchmarks with master code as baseline. Tolerate
	# failures: the PR's benchmarks may exercise APIs that don't
	# exist on master yet.
	git checkout origin/master -- collagraph/
	uv run --no-sync pytest bench \
	--benchmark-only \
	--benchmark-save=master$i \
	--benchmark-sort=mean \|\| true

	# Run benchmarks on PR code
	git checkout HEAD -- collagraph/
	uv run --no-sync pytest bench \
	--benchmark-only \
	--benchmark-save=branch$i \
	--benchmark-sort=mean
	done

	# The 25% threshold makes this guard a tripwire for gross
	# accidental regressions (an accidental copy in a hot path, an
	# algorithmic slip), not a precision instrument. Even with the
	# interleaved fastest-vs-slowest gate above, whole runs of
	# identical code on hosted runners have been observed to differ
	# by 10-30% (the same commit went fail/fail/pass across three
	# attempts at a 10% threshold, flagging code paths its diff never
	# touched). Gating below that noise floor just breeds re-run
	# rituals. Regressions too small to trip this guard should be
	# measured deliberately instead: repeated local runs of the bench
	# suite on an idle machine.
	- name: Compare benchmarks
	run: \|
	uv run --no-sync python bench/compare_runs.py \
	--baseline '.benchmarks//_master?.json' \
	--branch '.benchmarks//_branch?.json' \
	--threshold 0.25

	- name: Upload benchmarks
	if: always()
	uses: actions/upload-artifact@v7
	with:
	name: Benchmarks
	path: .benchmarks/
	include-hidden-files: true

Navigation Menu

Uh oh!

Release 0.10.0 #25

Workflow file

Release 0.10.0 #25

Uh oh!

Workflow file for this run