Skip to content

ci: Add PR benchmark comparison workflow - #38

Merged
mrecachinas merged 4 commits into
masterfrom
add-benchmark-workflow
Feb 1, 2026
Merged

ci: Add PR benchmark comparison workflow#38
mrecachinas merged 4 commits into
masterfrom
add-benchmark-workflow

Conversation

@mrecachinas

Copy link
Copy Markdown
Owner

Summary

This PR adds a new GitHub Actions workflow that automatically compares benchmark performance between PR branches and the base branch.

Features

  • Parallel benchmarking: Runs benchmarks on both base and PR branches simultaneously
  • Detailed comparison: Calculates percentage change for each benchmark
  • PR comments: Posts/updates a formatted table showing:
    • Benchmark name
    • Base branch time (median)
    • PR branch time (median)
    • Delta (% faster/slower)
    • Status indicator
  • Status indicators:
    • ✅ Faster (>5% improvement)
    • ➖ Within noise (±5%)
    • ⚠️ Slower (5-20% regression)
    • ❌ Severe regression (>20% slower)
  • Automatic failure: Workflow fails if any benchmark has >20% regression
  • Comment updates: Uses peter-evans/find-comment and create-or-update-comment to update existing comments instead of creating new ones

Technical Details

  • Uses ubuntu-latest runner with Python 3.11
  • Benchmarks run with --benchmark-disable-gc for consistent results
  • Uses median time (more stable than mean) for comparisons
  • Inline Python script for comparison logic (no external dependencies)

@github-actions

github-actions Bot commented Feb 1, 2026

Copy link
Copy Markdown

📊 Benchmark Comparison Results

Status Benchmark Base PR Delta
bytes_arr_within[ 1024 elems,s=32,end] 0.0223ms 0.0196ms 12.4% faster
bytes_arr_within[ 1024 elems,s=32,mid] 0.0112ms 0.0099ms 11.6% faster
➖ 22 benchmarks within noise (click to expand)
Benchmark Base PR Delta
bytes_arr_within[ 512 elems,s=16,at 0] 0.0001ms 0.0001ms 1.7% faster
bytes_arr_within[ 512 elems,s=16,end] 0.0018ms 0.0018ms 0.7% faster
bytes_arr_within[ 512 elems,s=16,mid] 0.0010ms 0.0010ms 0.3% faster
bytes_arr_within[ 1024 elems,s=32,at 0] 0.0001ms 0.0001ms 1.5% faster
bytes_arr_within[16384 elems,s=64,at 0] 0.0001ms 0.0001ms 2.2% faster
bytes_arr_within[16384 elems,s=64,end] 0.0926ms 0.0854ms 7.7% faster
bytes_arr_within[16384 elems,s=64,mid] 0.0460ms 0.0428ms 7.0% faster
hex_within_dist 0.0002ms 0.0002ms 13.6% slower
hd_bytes[1000-diff] 0.0002ms 0.0002ms 0.1% slower
hd_bytes[1000-same] 0.0002ms 0.0002ms 0.3% slower
hd_bytes[1024-diff] 0.0002ms 0.0001ms 2.7% faster
hd_bytes[1024-same] 0.0001ms 0.0001ms 2.1% faster
hd_bytes[3-diff] 0.0001ms 0.0001ms 2.6% faster
hd_bytes[3-same] 0.0001ms 0.0001ms 2.9% faster
hd_bytes[64-diff] 0.0001ms 0.0001ms 0.6% faster
hd_string[1000-diff] 0.0003ms 0.0003ms 0.6% slower
hd_string[1000-same] 0.0003ms 0.0003ms 0.8% slower
hd_string[1024-diff] 0.0003ms 0.0003ms 2.5% slower
hd_string[1024-same] 0.0003ms 0.0003ms 0.9% slower
hd_string[3-diff] 0.0002ms 0.0001ms 41.4% faster
hd_string[3-same] 0.0001ms 0.0001ms 0.6% faster
hd_string[64-diff] 0.0001ms 0.0001ms 5.2% slower

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

- Shorten benchmark names for readability (hd_ prefix, etc.)
- Use coefficient of variation to detect noise vs real regressions
- Raise regression threshold to 30% AND must exceed 2x noise level
- Remove truncation that was hiding important test parameters
- Hide 'within noise' benchmarks in collapsible details section
- Show regressions first, then improvements
- Cleaner summary when all benchmarks are within noise
For benchmarks < 1µs, ignore percentage changes since the absolute
difference is too small to matter. This prevents false positives
from noise on very fast operations.
@mrecachinas
mrecachinas merged commit c27e688 into master Feb 1, 2026
3 of 8 checks passed
@mrecachinas
mrecachinas deleted the add-benchmark-workflow branch February 1, 2026 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant