Skip to content

Commit d84c042

Browse files
yulnryulnr
authored
benches: add iai-callgrind PR gate, retire criterion CI (dial9-rs#360)
* benches: add iai-callgrind PR gate, retire criterion CI Replace noisy wall-clock benches on shared ubuntu-latest with a deterministic instruction-count gate. iai-callgrind runs each bench once under callgrind, so same code + same input gives byte-identical counts; a 1% threshold catches real regressions without false fires. * gate iai bench files behind --cfg iai_enabled * allow workflow_dispatch to override Bencher branch * trim bencher comment + add skip-bench bypass * drop criterion variants of iai-covered benches * drop --ci-only-thresholds from iai_main * cache valgrind apt install * faster iai jobs - paths-ignore on pull_request: skip whole workflow on docs / viewer / examples-only - swap `cargo install` for `cargo binstall` on iai-callgrind-runner - concurrency: cancel-in-progress so new PR pushes kill stale runs * amortize tracing_layer_iai * Rename bench jobs * cap threshold sample size at 1 for iai gate Prevents the default percentage testing of comparing against an average over a window of historical main runs. For iai, the more specific the better, and averages after baselines increases would be too noisy. * cap iai gate baseline to latest main run * bump the bencher action SHA to v0.6.4 * fix tresholds-reset typo * trigger CI * amortize tracing_layer_iai * remove --tresholds-reset * drop Harness in teardown to keep iai measurement deterministic * pin iai_micro checkout to PR head SHA * split bench workflow + extract iai-bench composite * include --treshold args for all measures * split into benchmarks.yml + benchmarks-pr.yml * cleanup
1 parent 32aa04c commit d84c042

13 files changed

Lines changed: 822 additions & 451 deletions

File tree

‎.github/actions/iai-bench/action.yml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: 'iai-callgrind bench'
2+
description: >-
3+
Sets up rust toolchain, build cache, valgrind, iai-callgrind-runner, and the
4+
Bencher CLI, then runs the iai bench suite via `bencher run`. Caller does
5+
`actions/checkout` so the ref can be event-specific.
6+
7+
inputs:
8+
iai-callgrind-version:
9+
description: Version of iai-callgrind-runner to install
10+
required: false
11+
default: '0.16.1'
12+
bencher-token:
13+
description: Bencher API token
14+
required: true
15+
bencher-branch:
16+
description: Bencher branch name to post results under
17+
required: true
18+
bencher-extra-args:
19+
description: Extra flags appended to each `bencher run` invocation
20+
required: false
21+
default: ''
22+
23+
runs:
24+
using: composite
25+
steps:
26+
- uses: dtolnay/rust-toolchain@stable
27+
- uses: Swatinem/rust-cache@v2
28+
29+
- name: Install valgrind (cached)
30+
uses: awalsh128/cache-apt-pkgs-action@latest
31+
with:
32+
packages: valgrind
33+
version: '1.0'
34+
35+
- name: Cache iai-callgrind-runner
36+
id: runner_cache
37+
uses: actions/cache@v4
38+
with:
39+
path: ~/.cargo/bin/iai-callgrind-runner
40+
key: iai-runner-${{ inputs.iai-callgrind-version }}
41+
42+
- name: Install cargo-binstall
43+
if: steps.runner_cache.outputs.cache-hit != 'true'
44+
uses: cargo-bins/cargo-binstall@v1.19.1
45+
46+
- name: Install iai-callgrind-runner (prebuilt via binstall)
47+
if: steps.runner_cache.outputs.cache-hit != 'true'
48+
shell: bash
49+
run: cargo binstall iai-callgrind-runner --version ${{ inputs.iai-callgrind-version }} --no-confirm --locked
50+
51+
- uses: bencherdev/bencher@fa25a72c516046b4c0e31659bef70c5266b0f94d # v0.6.4
52+
53+
- name: Run iai benches → Bencher
54+
shell: bash
55+
run: |
56+
set -euo pipefail
57+
bencher run \
58+
--token '${{ inputs.bencher-token }}' \
59+
--branch '${{ inputs.bencher-branch }}' \
60+
--testbed ubuntu-latest \
61+
--adapter rust_iai_callgrind \
62+
${{ inputs.bencher-extra-args }} \
63+
"cargo bench --workspace \
64+
--features dial9-tokio-telemetry/tracing-layer \
65+
--bench writer_encode_iai \
66+
--bench writer_write_encoded_iai \
67+
--bench threadlocal_encode_iai \
68+
--bench tracing_layer_iai \
69+
--bench codec_iai"

‎.github/workflows/benchmarks-pr.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# iai-callgrind PR regression gate. See benchmarks.yml for prereqs.
2+
#
3+
# Compares against the persistent thresholds maintained on the main
4+
# branch (set by benchmarks.yml). Skipped on PRs labeled `skip-bench`.
5+
# Same-repo PRs only (fork PRs have no access to BENCHER_API_TOKEN).
6+
7+
name: Benchmarks (PR)
8+
9+
on:
10+
pull_request:
11+
paths-ignore:
12+
- '**.md'
13+
- 'docs/**'
14+
- 'examples/**'
15+
- 'dial9-viewer/**'
16+
- '.github/ISSUE_TEMPLATE/**'
17+
18+
permissions:
19+
checks: write
20+
pull-requests: write
21+
22+
concurrency:
23+
group: ${{ github.workflow }}-${{ github.head_ref }}
24+
cancel-in-progress: true
25+
26+
jobs:
27+
pr_gate:
28+
if: github.event.pull_request.head.repo.full_name == github.repository && !contains(github.event.pull_request.labels.*.name, 'skip-bench')
29+
name: PR gate
30+
runs-on: ubuntu-latest
31+
timeout-minutes: 30
32+
env:
33+
RUST_BACKTRACE: 1
34+
BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT }}
35+
RUSTFLAGS: '--cfg tokio_unstable -C force-frame-pointers=yes --cfg iai_enabled'
36+
steps:
37+
- uses: actions/checkout@v4
38+
with:
39+
ref: ${{ github.event.pull_request.head.sha }}
40+
- uses: ./.github/actions/iai-bench
41+
with:
42+
bencher-token: ${{ secrets.BENCHER_API_TOKEN }}
43+
bencher-branch: ${{ github.head_ref }}
44+
bencher-extra-args: >-
45+
--start-point main
46+
--start-point-reset
47+
--ci-only-thresholds
48+
--error-on-alert
49+
--github-actions ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/benchmarks.yml‎

Lines changed: 54 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -1,136 +1,80 @@
1-
# Continuous benchmarking with Bencher (https://bencher.dev)
1+
# Continuous benchmarking with Bencher (https://bencher.dev).
22
#
33
# PREREQUISITES:
44
# 1. Create a project at https://bencher.dev and note its slug.
55
# 2. Add the following in GitHub → Settings → Secrets and variables → Actions:
66
# - Secret BENCHER_API_TOKEN: API token from bencher.dev
77
# - Variable BENCHER_PROJECT: your project slug
8-
# 3. GITHUB_TOKEN is provided automatically — no setup needed.
98
#
10-
# overhead_bench uses a custom harness (not Criterion). Its --bmf flag
11-
# outputs Bencher Metric Format directly.
9+
# This workflow posts numbers and maintains the persistent regression
10+
# threshold on the target branch. The PR gate (benchmarks-pr.yml) reads
11+
# main's threshold via --start-point-reset.
1212

1313
name: Benchmarks
1414

1515
on:
1616
push:
17-
branches:
18-
- main
19-
pull_request:
17+
branches: [main]
18+
paths-ignore:
19+
- '**.md'
20+
- 'docs/**'
21+
- 'examples/**'
22+
- 'dial9-viewer/**'
23+
- '.github/ISSUE_TEMPLATE/**'
2024
workflow_dispatch:
25+
inputs:
26+
bencher_branch:
27+
description: 'Override Bencher branch name (e.g. "main" to seed baseline from a feature branch). Leave empty to use the running branch.'
28+
required: false
29+
default: ''
2130

2231
permissions:
2332
checks: write
2433
pull-requests: write
2534

26-
jobs:
27-
# Runs on every push to main to build the statistical baseline.
28-
benchmark_main:
29-
name: Benchmark — ${{ matrix.bench.name }} (main baseline)
30-
if: ${{ false && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') }} # skip job
31-
runs-on: ubuntu-latest
32-
timeout-minutes: 30
33-
strategy:
34-
fail-fast: false
35-
matrix:
36-
bench:
37-
- name: writer_encode
38-
adapter: rust_criterion
39-
command: cargo bench --package dial9-tokio-telemetry --bench writer_encode
40-
- name: codec
41-
adapter: rust_criterion
42-
command: cargo bench --package dial9-trace-format --bench codec
43-
- name: overhead_bench
44-
adapter: json
45-
command: cargo bench --bench overhead_bench -- --bmf 10
46-
- name: overhead_bench_ctimer
47-
adapter: json
48-
command: DIAL9_FORCE_CTIMER=1 cargo bench --bench overhead_bench -- --bmf 10
49-
- name: e2e_workload
50-
adapter: json
51-
command: cargo bench --bench e2e_workload -- --bmf 10
52-
env:
53-
RUST_BACKTRACE: 1
54-
BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT }}
55-
steps:
56-
- uses: actions/checkout@v4
57-
- uses: dtolnay/rust-toolchain@stable
58-
- uses: Swatinem/rust-cache@v2
35+
concurrency:
36+
group: ${{ github.workflow }}-${{ github.ref }}
5937

60-
- name: Enable perf_event_open and kallsyms
61-
run: |
62-
sudo sysctl kernel.perf_event_paranoid=1
63-
sudo sysctl kernel.kptr_restrict=0
64-
65-
- uses: bencherdev/bencher@0f8f620172ccd6225d40a7590598eb7b41718af8 # v0.6.2
66-
67-
- name: Run benchmark
68-
run: |
69-
bencher run \
70-
--token '${{ secrets.BENCHER_API_TOKEN }}' \
71-
--branch '${{ github.ref_name }}' \
72-
--testbed ubuntu-latest \
73-
--adapter '${{ matrix.bench.adapter }}' \
74-
"${{ matrix.bench.command }}"
75-
76-
# Runs on same-repo PRs. Fork PRs are skipped — they have no access to
77-
# BENCHER_API_TOKEN, so the job would fail rather than silently skip.
78-
benchmark_pr:
79-
name: Benchmark — ${{ matrix.bench.name }} (PR regression check)
80-
if: ${{ false && (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository) }} # skip job
38+
jobs:
39+
baseline:
40+
name: baseline
8141
runs-on: ubuntu-latest
8242
timeout-minutes: 30
83-
strategy:
84-
fail-fast: false
85-
matrix:
86-
bench:
87-
- name: writer_encode
88-
adapter: rust_criterion
89-
command: cargo bench --package dial9-tokio-telemetry --bench writer_encode
90-
- name: codec
91-
adapter: rust_criterion
92-
command: cargo bench --package dial9-trace-format --bench codec
93-
- name: overhead_bench
94-
adapter: json
95-
command: cargo bench --bench overhead_bench -- --bmf 10
96-
- name: overhead_bench_ctimer
97-
adapter: json
98-
command: DIAL9_FORCE_CTIMER=1 cargo bench --bench overhead_bench -- --bmf 10
99-
- name: e2e_workload
100-
adapter: json
101-
command: cargo bench --bench e2e_workload -- --bmf 10
10243
env:
10344
RUST_BACKTRACE: 1
10445
BENCHER_PROJECT: ${{ vars.BENCHER_PROJECT }}
46+
# Mirror .cargo/config.toml [build] rustflags + add iai_enabled cfg
47+
# so bench files compile their real iai entry point (not the no-op
48+
# stub `cargo test --all-targets` uses).
49+
RUSTFLAGS: '--cfg tokio_unstable -C force-frame-pointers=yes --cfg iai_enabled'
10550
steps:
10651
- uses: actions/checkout@v4
107-
- uses: dtolnay/rust-toolchain@stable
108-
- uses: Swatinem/rust-cache@v2
109-
110-
- name: Enable perf_event_open and kallsyms
111-
run: |
112-
sudo sysctl kernel.perf_event_paranoid=1
113-
sudo sysctl kernel.kptr_restrict=0
114-
115-
- uses: bencherdev/bencher@0f8f620172ccd6225d40a7590598eb7b41718af8 # v0.6.2
116-
117-
- name: Run benchmark
118-
run: |
119-
bencher run \
120-
--token '${{ secrets.BENCHER_API_TOKEN }}' \
121-
--branch '${{ github.head_ref }}' \
122-
--start-point main \
123-
--start-point-reset \
124-
--testbed ubuntu-latest \
125-
--adapter '${{ matrix.bench.adapter }}' \
126-
--threshold-measure latency \
127-
--threshold-test percentage \
128-
--threshold-lower-boundary _ \
129-
--threshold-upper-boundary 0.25 \
130-
--threshold-measure throughput \
131-
--threshold-test percentage \
132-
--threshold-lower-boundary 0.25 \
133-
--threshold-upper-boundary _ \
134-
--error-on-alert \
135-
--github-actions '${{ secrets.GITHUB_TOKEN }}' \
136-
"${{ matrix.bench.command }}"
52+
- uses: ./.github/actions/iai-bench
53+
with:
54+
bencher-token: ${{ secrets.BENCHER_API_TOKEN }}
55+
bencher-branch: ${{ github.event.inputs.bencher_branch != '' && github.event.inputs.bencher_branch || github.ref_name }}
56+
bencher-extra-args: >-
57+
--threshold-measure instructions
58+
--threshold-test percentage
59+
--threshold-max-sample-size 1
60+
--threshold-upper-boundary 0.01
61+
--threshold-measure l1-hits
62+
--threshold-test percentage
63+
--threshold-max-sample-size 1
64+
--threshold-upper-boundary 0.01
65+
--threshold-measure ram-hits
66+
--threshold-test percentage
67+
--threshold-max-sample-size 1
68+
--threshold-upper-boundary 0.01
69+
--threshold-measure total-read-write
70+
--threshold-test percentage
71+
--threshold-max-sample-size 1
72+
--threshold-upper-boundary 0.01
73+
--threshold-measure estimated-cycles
74+
--threshold-test percentage
75+
--threshold-max-sample-size 1
76+
--threshold-upper-boundary 0.01
77+
--threshold-measure ll-hits
78+
--threshold-test percentage
79+
--threshold-max-sample-size 1
80+
--threshold-upper-boundary 0.05

0 commit comments

Comments
 (0)