Skip to content

[ci] Benchmark tracking #3698

Merged
patrick-ogrady merged 15 commits into
mainfrom
cl/bench-tracker
May 4, 2026
Merged

[ci] Benchmark tracking #3698
patrick-ogrady merged 15 commits into
mainfrom
cl/bench-tracker

Conversation

@clabby
Copy link
Copy Markdown
Collaborator

@clabby clabby commented Apr 30, 2026

Overview

Adds a new workflow for running a configurable list of benchmarks on PRs and comparing them with results from main in order for us to more easily catch performance regressions in important areas.

@clabby clabby self-assigned this Apr 30, 2026
@clabby clabby added benchmark Adding benchmarks for performance testing performance labels Apr 30, 2026
@clabby clabby added this to Tracker Apr 30, 2026
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 30, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
commonware-mcp ab6116b May 04 2026, 06:43 PM

@clabby clabby moved this to In Progress in Tracker Apr 30, 2026
@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Apr 30, 2026

Deploying monorepo with  Cloudflare Pages  Cloudflare Pages

Latest commit: ab6116b
Status: ✅  Deploy successful!
Preview URL: https://a9053fbe.monorepo-eu0.pages.dev
Branch Preview URL: https://cl-bench-tracker.monorepo-eu0.pages.dev

View logs

@clabby clabby force-pushed the cl/bench-tracker branch from 6e914f3 to 52f555f Compare April 30, 2026 15:20
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 30, 2026

Benchmark results

Tip

PASSED: No benchmark exceeded the regression threshold.

Warning

1 benchmark(s) had no uploaded main-branch baseline.

Benchmark comparison table
Benchmark Baseline (main) Current Delta Threshold Status
qmdb::merkleize/variant=any::unordered::fixed::mmr keys=10000 ch=false sync=false - 1.533 ms - 10.00% ⚠️ WARN missing baseline

@clabby clabby force-pushed the cl/bench-tracker branch 3 times, most recently from b7ccf34 to 0c043f3 Compare April 30, 2026 16:29
Comment thread .github/scripts/benchmark-tracking.py Outdated
@patrick-ogrady
Copy link
Copy Markdown
Contributor

As discussed, we should probably resolve this:

- [P2] Gate PR comments to write-capable runs — /Users/patrickogrady/code/monorepo-pr-3698/.github/workflows/benchmark-tracking.yml:101-101
    For pull_request runs from forks or Dependabot, GitHub downgrades GITHUB_TOKEN to read-only even though the workflow requests issues: write, so this comment step will 403 after a successful benchmark run and mark the check
    failed even when there is no benchmark regression. Gate this step to same-repository PRs or make comment posting non-blocking so external PRs can still use the step summary and regression failure logic.

@clabby clabby force-pushed the cl/bench-tracker branch from 3b7f527 to 8b8bc58 Compare May 2, 2026 20:30
@clabby clabby marked this pull request as ready for review May 4, 2026 15:04
@clabby clabby moved this from In Progress to Ready for Review in Tracker May 4, 2026
Comment thread .github/scripts/benchmark-tracking.py Outdated
Comment thread .github/scripts/benchmark-tracking.py
@clabby clabby force-pushed the cl/bench-tracker branch 2 times, most recently from 713e9bf to 82fe7ea Compare May 4, 2026 18:36
@clabby clabby force-pushed the cl/bench-tracker branch from 82fe7ea to f9e35f3 Compare May 4, 2026 18:38
@clabby clabby force-pushed the cl/bench-tracker branch from f9e35f3 to 48655ee Compare May 4, 2026 18:38
@patrick-ogrady patrick-ogrady merged commit 1c5e468 into main May 4, 2026
15 checks passed
@patrick-ogrady patrick-ogrady deleted the cl/bench-tracker branch May 4, 2026 18:44
@github-project-automation github-project-automation Bot moved this from Ready for Review to Done in Tracker May 4, 2026
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ab6116b. Configure here.


concurrency:
group: check-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unconditional cancel-in-progress breaks main baseline uploads

Medium Severity

cancel-in-progress: true is unconditional, unlike every other workflow in this repo which uses cancel-in-progress: ${{ github.event_name == 'pull_request' }}. This means rapid pushes to main will cancel in-progress benchmark runs before they can upload their baseline artifact, potentially leaving PRs with a stale or missing baseline. For a 180-minute-timeout benchmark job, this window is significant.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit ab6116b. Configure here.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.77%. Comparing base (52f31c8) to head (ab6116b).
⚠️ Report is 3 commits behind head on main.

@@           Coverage Diff           @@
##             main    #3698   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files         459      458    -1     
  Lines      178700   178691    -9     
  Branches     4158     4158           
=======================================
- Hits       171143   171139    -4     
+ Misses       6232     6226    -6     
- Partials     1325     1326    +1     

see 10 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52f31c8...ab6116b. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmark Adding benchmarks for performance testing performance

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants