Skip to content

Commit 1bb6c6d

Browse files
zeitlingerzeitlinger
authored
ci: compare PR benchmarks with base (#2296)
## Summary Make PR benchmark comments answer whether the PR is faster or slower than its base commit, while keeping wall-clock time close to a single benchmark run. ## Changes - Run base and head benchmarks in parallel matrix jobs. - Add a lightweight summary job that downloads both raw JMH artifacts and generates the PR comment. - Add a `Comparison with base` table to the generated `README.md` / PR comment. - Mark each benchmark as `faster`, `slower`, or `within noise` using JMH confidence intervals when available. - Capture and show runner hardware for both base and head jobs, including a note that parallel jobs may run on different hardware. - Rename the existing per-run relative column to `Within run` so it is clear that it is not the base comparison. - Store `baseline-results.json` in the uploaded benchmark artifact for inspection. ## Validation - Ran `mise run lint:fix`. - Ran `python3 -m py_compile .mise/tasks/generate_benchmark_summary.py`. - Ran `git diff --check`. - Replayed summary generation with the latest #2252 benchmark artifact and the published benchmark baseline; the output now shows `HistogramBenchmark.prometheusClassic` as `-78.6%` / `slower`. - Verified no-baseline summary generation still omits the comparison section. ## Tradeoff Parallel base/head jobs are faster wall-clock than measuring both commits sequentially on one runner, but runner hardware can differ. The comment now prints both runner descriptions and calls out that caveat.
1 parent 4a6958a commit 1bb6c6d

2 files changed

Lines changed: 399 additions & 27 deletions

File tree

‎.github/workflows/pr-benchmarks.yml‎

Lines changed: 74 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,29 @@ jobs:
1919
benchmark:
2020
if: github.event.label.name == 'benchmark'
2121
runs-on: ubuntu-24.04
22+
strategy:
23+
fail-fast: false
24+
matrix:
25+
include:
26+
- name: base
27+
repository: ${{ github.repository }}
28+
ref: ${{ github.event.pull_request.base.sha }}
29+
sha: ${{ github.event.pull_request.base.sha }}
30+
- name: head
31+
repository: ${{ github.event.pull_request.head.repo.full_name }}
32+
ref: ${{ github.event.pull_request.head.sha }}
33+
sha: ${{ github.event.pull_request.head.sha }}
2234
permissions:
2335
contents: read # checkout only
36+
env:
37+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
38+
PR_NUMBER: ${{ github.event.pull_request.number }}
2439
steps:
25-
- name: Checkout PR head
40+
- name: Checkout ${{ matrix.name }}
2641
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
2742
with:
28-
repository: ${{ github.event.pull_request.head.repo.full_name }}
29-
ref: ${{ github.event.pull_request.head.sha }}
43+
repository: ${{ matrix.repository }}
44+
ref: ${{ matrix.ref }}
3045
persist-credentials: false
3146
fetch-depth: 0
3247

@@ -40,22 +55,71 @@ jobs:
4055
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
4156
with:
4257
path: ~/.m2/repository
43-
key: ${{ runner.os }}-pr-bench-maven-${{ github.event.pull_request.head.sha }}
58+
key: ${{ runner.os }}-pr-bench-maven-${{ matrix.name }}-${{ matrix.sha }}
4459
restore-keys: |
4560
${{ runner.os }}-pr-bench-maven-
4661
${{ runner.os }}-maven-
4762
48-
- name: Run JMH benchmarks
63+
- name: Run ${{ matrix.name }} JMH benchmarks
4964
run: mise run benchmark:ci-json
5065

66+
- name: Capture ${{ matrix.name }} runner info
67+
run: |
68+
python3 ./.mise/tasks/generate_benchmark_summary.py \
69+
--write-system-info runner-info.json
70+
71+
- name: Upload ${{ matrix.name }} raw benchmark results
72+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
73+
with:
74+
name: pr-benchmark-raw-${{ matrix.name }}-${{ env.PR_NUMBER }}-${{ env.HEAD_SHA }}
75+
path: |
76+
benchmark-results.json
77+
runner-info.json
78+
retention-days: 5
79+
80+
summarize:
81+
if: github.event.label.name == 'benchmark'
82+
runs-on: ubuntu-24.04
83+
needs: benchmark
84+
permissions:
85+
contents: read # checkout only
86+
env:
87+
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
88+
PR_NUMBER: ${{ github.event.pull_request.number }}
89+
steps:
90+
- name: Checkout PR head
91+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
92+
with:
93+
repository: ${{ github.event.pull_request.head.repo.full_name }}
94+
ref: ${{ github.event.pull_request.head.sha }}
95+
persist-credentials: false
96+
fetch-depth: 0
97+
98+
- name: Download base benchmark results
99+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
100+
with:
101+
name: pr-benchmark-raw-base-${{ env.PR_NUMBER }}-${{ env.HEAD_SHA }}
102+
path: /tmp/base-benchmark-results
103+
104+
- name: Download head benchmark results
105+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
106+
with:
107+
name: pr-benchmark-raw-head-${{ env.PR_NUMBER }}-${{ env.HEAD_SHA }}
108+
path: /tmp/head-benchmark-results
109+
51110
- name: Generate benchmark summary
52111
run: |
53-
mise run benchmark:generate-summary \
54-
--input benchmark-results.json \
112+
python3 ./.mise/tasks/generate_benchmark_summary.py \
113+
--input /tmp/head-benchmark-results/benchmark-results.json \
114+
--baseline /tmp/base-benchmark-results/benchmark-results.json \
115+
--system-info /tmp/head-benchmark-results/runner-info.json \
116+
--baseline-system-info /tmp/base-benchmark-results/runner-info.json \
55117
--output-dir benchmark-results \
56-
--commit-sha "${{ github.event.pull_request.head.sha }}"
118+
--commit-sha "${{ github.event.pull_request.head.sha }}" \
119+
--baseline-sha "${{ github.event.pull_request.base.sha }}"
57120
env:
58121
GITHUB_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name }}
122+
GITHUB_BASE_REPOSITORY: ${{ github.repository }}
59123

60124
- name: Prepare PR comment summary
61125
run: |
@@ -64,13 +128,13 @@ jobs:
64128
- name: Upload PR benchmark results
65129
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
66130
with:
67-
name: pr-benchmark-results-pr-${{ github.event.pull_request.number }}-${{ github.event.pull_request.head.sha }}
131+
name: pr-benchmark-results-pr-${{ env.PR_NUMBER }}-${{ env.HEAD_SHA }}
68132
path: benchmark-results
69133
retention-days: 5
70134

71135
- name: Upload PR benchmark comment
72136
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
73137
with:
74-
name: pr-benchmark-comment-pr-${{ github.event.pull_request.number }}-${{ github.event.pull_request.head.sha }}
138+
name: pr-benchmark-comment-pr-${{ env.PR_NUMBER }}-${{ env.HEAD_SHA }}
75139
path: pr-benchmark-comment.md
76140
retention-days: 5

0 commit comments

Comments
 (0)