Skip to content

Commit 94aa839

Browse files
CameronBrooks11CameronBrooks11
committed
feat(bench): add policy-based benchmark evaluation and CI evidence workflow
1 parent 9418a8f commit 94aa839

12 files changed

Lines changed: 827 additions & 22 deletions

File tree

‎.github/workflows/benchmark-evidence.yml‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
name: Benchmark Evidence
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
policy_profile:
7+
description: Policy profile for evaluation
8+
required: false
9+
default: ci-hosted
10+
type: choice
11+
options:
12+
- ci-hosted
13+
- ci-dedicated
14+
include_optional:
15+
description: Include JSON/YAML loader benchmarks
16+
required: false
17+
default: false
18+
type: boolean
19+
baseline_path:
20+
description: Optional baseline JSON path (repo-relative)
21+
required: false
22+
default: ""
23+
type: string
24+
schedule:
25+
- cron: "0 7 * * 1"
26+
27+
env:
28+
VCPKG_COMMIT: "66c0373dc7fca549e5803087b9487edfe3aca0a1"
29+
30+
jobs:
31+
windows-benchmark:
32+
name: Windows benchmark evidence
33+
runs-on: windows-2022
34+
timeout-minutes: 45
35+
36+
steps:
37+
- name: Checkout
38+
uses: actions/checkout@v4
39+
40+
- name: Setup vcpkg
41+
uses: lukka/run-vcpkg@v11
42+
with:
43+
vcpkgGitCommitId: ${{ env.VCPKG_COMMIT }}
44+
doNotCache: false
45+
46+
- name: Setup Python
47+
uses: actions/setup-python@v5
48+
with:
49+
python-version: "3.12"
50+
51+
- name: Resolve benchmark options
52+
id: opts
53+
shell: pwsh
54+
run: |
55+
$profile = "ci-hosted"
56+
$includeOptional = "false"
57+
$baseline = ""
58+
59+
if ("${{ github.event_name }}" -eq "workflow_dispatch") {
60+
if ("${{ github.event.inputs.policy_profile }}" -ne "") {
61+
$profile = "${{ github.event.inputs.policy_profile }}"
62+
}
63+
if ("${{ github.event.inputs.include_optional }}" -eq "true") {
64+
$includeOptional = "true"
65+
}
66+
if ("${{ github.event.inputs.baseline_path }}" -ne "") {
67+
$baseline = "${{ github.event.inputs.baseline_path }}"
68+
}
69+
}
70+
71+
$benchDir = "artifacts/benchmarks/${{ github.run_id }}_${{ github.run_attempt }}_ci-windows-release"
72+
73+
"profile=$profile" >> $env:GITHUB_OUTPUT
74+
"include_optional=$includeOptional" >> $env:GITHUB_OUTPUT
75+
"baseline=$baseline" >> $env:GITHUB_OUTPUT
76+
"bench_dir=$benchDir" >> $env:GITHUB_OUTPUT
77+
78+
- name: Run benchmark suite + policy evaluation
79+
shell: pwsh
80+
run: |
81+
$args = @(
82+
"-Preset", "ci-windows-release",
83+
"-Config", "Release",
84+
"-OutputDir", "${{ steps.opts.outputs.bench_dir }}",
85+
"-PolicyProfile", "${{ steps.opts.outputs.profile }}"
86+
)
87+
88+
if ("${{ steps.opts.outputs.include_optional }}" -eq "true") {
89+
$args += "-IncludeOptional"
90+
}
91+
92+
if ("${{ steps.opts.outputs.baseline }}" -ne "") {
93+
$args += @("-Baseline", "${{ steps.opts.outputs.baseline }}")
94+
}
95+
96+
.\scripts\bench.ps1 @args
97+
98+
- name: Upload benchmark artifacts
99+
uses: actions/upload-artifact@v4
100+
with:
101+
name: benchmark-evidence-windows-${{ github.run_id }}-${{ github.run_attempt }}
102+
path: ${{ steps.opts.outputs.bench_dir }}
103+
if-no-files-found: error

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3636
- server protected-write rejection
3737
- server `--dt` timestep behavior
3838
- SignalStore unit-declaration behavior
39+
- Benchmark evidence tooling for Phase 2:
40+
- policy profiles (`local`, `ci-hosted`, `ci-dedicated`)
41+
- benchmark evaluation script (`scripts/evaluate_benchmarks.py`)
42+
- benchmark evidence CI workflow with artifact upload (`.github/workflows/benchmark-evidence.yml`)
3943

4044
### Fixed
4145

‎benchmarks/policy/README.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Benchmark Policy
2+
3+
This directory defines benchmark evaluation policy profiles and baseline references.
4+
5+
## Files
6+
7+
1. `bench_policy.json`: policy profiles used by `scripts/evaluate_benchmarks.py`.
8+
2. `baseline_template.json`: template for stable-hardware latency baseline.
9+
10+
## Profile Intent
11+
12+
1. `local`: developer machine feedback, no latency hard fail.
13+
2. `ci-hosted`: GitHub runner monitoring, warning-oriented due host variance.
14+
3. `ci-dedicated`: strict gating profile for stable hardware and publication/release evidence.
15+
16+
## Baseline Workflow
17+
18+
1. Run benchmark suite on stable hardware.
19+
2. Copy measured latency metrics into a baseline file derived from `baseline_template.json`.
20+
3. Commit baseline file and reference it with `--baseline` in dedicated-gate runs.

‎benchmarks/policy/baseline_template.json‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{
2+
"metadata": {
3+
"description": "Replace values with a validated baseline run from stable hardware.",
4+
"source_artifact": "artifacts/benchmarks/<timestamp>_<preset>/benchmark_results.json"
5+
},
6+
"metrics": {
7+
"benchmark_tick.simple_avg_tick_us": 0.0,
8+
"benchmark_tick.complex_avg_tick_us": 0.0
9+
}
10+
}

‎benchmarks/policy/bench_policy.json‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
{
2+
"version": 1,
3+
"profiles": {
4+
"local": {
5+
"description": "Developer workstation profile. Collect evidence and surface regressions without noisy latency hard-fails.",
6+
"checks": {
7+
"tick_alloc_per_tick": {
8+
"enabled": true,
9+
"severity": "warning",
10+
"max_value": 0.0,
11+
"metric_keys": [
12+
"benchmark_tick.simple_alloc_per_tick",
13+
"benchmark_tick.complex_alloc_per_tick"
14+
]
15+
},
16+
"latency_regression": {
17+
"enabled": false,
18+
"warn_pct": 20.0,
19+
"fail_pct": 50.0,
20+
"require_baseline": false,
21+
"enforce_fail_threshold": false,
22+
"missing_metric_severity": "warning",
23+
"missing_baseline_metric_severity": "warning",
24+
"metric_keys": [
25+
"benchmark_tick.simple_avg_tick_us",
26+
"benchmark_tick.complex_avg_tick_us"
27+
]
28+
}
29+
}
30+
},
31+
"ci-hosted": {
32+
"description": "GitHub-hosted runner profile. Allocation regressions and latency drift are tracked as warnings due host variance.",
33+
"checks": {
34+
"tick_alloc_per_tick": {
35+
"enabled": true,
36+
"severity": "warning",
37+
"max_value": 0.0,
38+
"metric_keys": [
39+
"benchmark_tick.simple_alloc_per_tick",
40+
"benchmark_tick.complex_alloc_per_tick"
41+
]
42+
},
43+
"latency_regression": {
44+
"enabled": true,
45+
"warn_pct": 20.0,
46+
"fail_pct": 50.0,
47+
"require_baseline": false,
48+
"enforce_fail_threshold": false,
49+
"missing_metric_severity": "warning",
50+
"missing_baseline_metric_severity": "warning",
51+
"metric_keys": [
52+
"benchmark_tick.simple_avg_tick_us",
53+
"benchmark_tick.complex_avg_tick_us"
54+
]
55+
}
56+
}
57+
},
58+
"ci-dedicated": {
59+
"description": "Dedicated stable hardware profile for release/publication evidence gating.",
60+
"checks": {
61+
"tick_alloc_per_tick": {
62+
"enabled": true,
63+
"severity": "error",
64+
"max_value": 0.0,
65+
"metric_keys": [
66+
"benchmark_tick.simple_alloc_per_tick",
67+
"benchmark_tick.complex_alloc_per_tick"
68+
]
69+
},
70+
"latency_regression": {
71+
"enabled": true,
72+
"warn_pct": 10.0,
73+
"fail_pct": 20.0,
74+
"require_baseline": true,
75+
"enforce_fail_threshold": true,
76+
"missing_metric_severity": "error",
77+
"missing_baseline_metric_severity": "error",
78+
"metric_keys": [
79+
"benchmark_tick.simple_avg_tick_us",
80+
"benchmark_tick.complex_avg_tick_us"
81+
]
82+
}
83+
}
84+
}
85+
}
86+
}

‎docs/performance-benchmarks.md‎

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,17 @@ The wrappers call `scripts/run_benchmarks.py`, which:
5353
1. Configures/builds benchmark targets (unless `--no-build` is set).
5454
2. Runs each benchmark executable.
5555
3. Captures stdout/stderr logs per target.
56-
4. Emits a machine-readable manifest with environment and git metadata.
56+
4. Emits a machine-readable manifest with environment, git metadata, and parsed benchmark metrics.
5757

5858
By default, status failures are reported but do not fail the run; execution failures still fail.
5959
Use `--fail-on-status` when running gate-enforced benchmark checks.
6060

61+
Wrappers then run `scripts/evaluate_benchmarks.py` with a policy profile:
62+
63+
1. `local`: informational for workstation variability.
64+
2. `ci-hosted`: warning-oriented checks for shared CI runners.
65+
3. `ci-dedicated`: strict gate profile for stable hardware evidence.
66+
6167
## Artifact Contract
6268

6369
Artifacts are stored under:
@@ -67,10 +73,11 @@ Artifacts are stored under:
6773
Required files:
6874

6975
1. `benchmark_results.json`
70-
2. `<target>.stdout.log`
71-
3. `<target>.stderr.log`
72-
4. `configure.stdout.log`, `configure.stderr.log` (when build enabled)
73-
5. `build.stdout.log`, `build.stderr.log` (when build enabled)
76+
2. `benchmark_evaluation.json`
77+
3. `<target>.stdout.log`
78+
4. `<target>.stderr.log`
79+
5. `configure.stdout.log`, `configure.stderr.log` (when build enabled)
80+
6. `build.stdout.log`, `build.stderr.log` (when build enabled)
7481

7582
`benchmark_results.json` contains:
7683

@@ -80,6 +87,14 @@ Required files:
8087
4. Git commit hash and dirty-worktree flag
8188
5. Per-benchmark executable path, command, exit code, duration, parsed PASS/FAIL status lines
8289

90+
Tick benchmark output additionally tracks measured heap allocations during the timed loop (`Allocations`, `Alloc/tick`) so Phase 2 zero-allocation evidence is captured per scenario.
91+
92+
`benchmark_evaluation.json` contains:
93+
94+
1. selected policy profile
95+
2. issue summary (errors/warnings)
96+
3. per-check findings with metric keys and threshold context
97+
8398
## Evidence Rules
8499

85100
For any published performance claim, attach:
@@ -94,16 +109,16 @@ Claims without linked artifacts are treated as unsupported.
94109

95110
## CI Guidance
96111

97-
Benchmarks are intentionally not part of default CI pass/fail lanes due to runtime and host variance.
112+
Benchmarks are intentionally separated from the default CI correctness lanes.
98113

99114
Recommended:
100115

101-
1. Run benchmark workflow on demand (`workflow_dispatch`) or scheduled lane.
116+
1. Run benchmark evidence workflow on demand (`workflow_dispatch`) or scheduled lane.
102117
2. Store artifacts as CI build artifacts.
103-
3. Compare against previous baseline with explicit tolerance policy.
118+
3. Apply `ci-hosted` profile on hosted runners and reserve strict gating for dedicated runners.
104119

105120
## Next Phase 2 Steps
106121

107-
1. Add allocation-count instrumentation in tick path.
108-
2. Define numeric regression thresholds per workload class.
109-
3. Introduce automated baseline comparison tooling.
122+
1. Define numeric regression thresholds per workload class.
123+
2. Introduce automated baseline comparison tooling.
124+
3. Add a dedicated perf CI lane with artifact upload and threshold evaluation.

‎scripts/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,10 +43,15 @@ Options:
4343
- `--include-optional` / `-IncludeOptional`: include JSON/YAML loader benchmarks.
4444
- `--no-build` / `-NoBuild`: run existing binaries without configure/build.
4545
- `--fail-on-status` / `-FailOnStatus`: return non-zero when benchmark reports `Status: FAIL`.
46+
- `--policy-profile` / `-PolicyProfile`: benchmark policy profile (`local`, `ci-hosted`, `ci-dedicated`).
47+
- `--policy-file` / `-PolicyFile`: policy JSON path (default: `benchmarks/policy/bench_policy.json`).
48+
- `--baseline` / `-Baseline`: optional baseline JSON for latency regression checks.
49+
- `--no-evaluate` / `-NoEvaluate`: skip policy evaluation stage.
4650

4751
Artifacts are written under `artifacts/benchmarks/<timestamp>_<preset>/` with:
4852

4953
- `benchmark_results.json` (manifest + metadata + per-benchmark status)
54+
- `benchmark_evaluation.json` (policy evaluation summary)
5055
- `*.stdout.log` and `*.stderr.log`
5156
- configure/build logs when build is enabled
5257

0 commit comments

Comments
 (0)