Skip to content

Commit 0d543bd

Browse files
marius-bughiuclaude
andcommitted
fix(dashboard): render the EnumMap and EnumSet cards, and gate blank cards in CI
The dashboard required an `(ItemCount: N)` suffix on every BenchmarkDotNet result name. `EnumMapBenchmark` and `EnumSetBenchmark` deliberately declare no `[Params]` sweep — their workload is the bounded enum universe — so their measurements were published to gh-pages and then dropped by `parseName`, and both cards had rendered empty since they shipped. The `ItemCount` group is now optional in both `index.html` and `detail.html`. An unparameterized class parses with `itemCount === NO_SWEEP` and charts as a single bucket: the grid card omits the "@ N items" detail, the detail page drops its item-count toggle, and such benchmarks are excluded from the headline speedup stats rather than bucketed at zero items. The suffix is still matched strictly as `ItemCount`, so a params property named anything else is rejected rather than charted under an "items" label it does not mean. That failure mode was silent, and it had bitten twice: `DisjointSet` also blanked for five runs when its params property was briefly named `ElementCount`. So `scripts/check_dashboard_coverage.js` now fails CI on it. It lifts the `COLLECTIONS` tables and both name parsers straight out of the dashboard HTML rather than reimplementing them, so it validates the parser that actually ships. Structural checks (the two `COLLECTIONS` arrays agree; every charted collection has a `{Key}Benchmark` in `CoreBenchmarks`) run on every PR in `ci.yml`, since `benchmarks.yml` only fires on `src/**` and would never see a dashboard-only change. The full check runs against the merged report in the aggregate job, last, so a wiring mistake reddens the job without costing us the measurement. Closes #301 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent b961162 commit 0d543bd

8 files changed

Lines changed: 316 additions & 28 deletions

File tree

.github/workflows/benchmarks.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ jobs:
155155
exit 0
156156
fi
157157
echo "have_head=true" >> "$GITHUB_OUTPUT"
158+
echo "head_shards=${#head_files[@]}" >> "$GITHUB_OUTPUT"
158159
159160
# Stitch every shard's joined report back into one: keep shard 0's metadata,
160161
# concatenate all shards' Benchmarks arrays.
@@ -418,3 +419,11 @@ jobs:
418419
git -c user.name="github-actions" -c user.email="github-actions@github.com" \
419420
commit -m "Sync custom dashboard from ${GITHUB_SHA:0:7}"
420421
git push origin gh-pages
422+
423+
# Deliberately the LAST step: a dashboard-wiring mistake must not cost us the
424+
# measurement, so the comment and the gh-pages publish above happen first and this
425+
# only reddens the job. Skipped unless every shard reported, since a partial merge
426+
# is legitimately missing whole benchmark classes (the job runs with if: always()).
427+
- name: Check dashboard coverage of the merged report
428+
if: steps.merge.outputs.have_head == 'true' && steps.merge.outputs.head_shards == env.SHARD_TOTAL
429+
run: node scripts/check_dashboard_coverage.js /tmp/pr-report-full.json

.github/workflows/ci.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,21 @@ jobs:
5757
if-no-files-found: warn
5858
retention-days: 7
5959

60+
# The benchmark workflow only fires on `src/**`, so a dashboard-only change never
61+
# reaches the report-backed run of this same script in benchmarks.yml. These are the
62+
# structural checks, which need no measurements and cost seconds.
63+
dashboard-coverage:
64+
name: dashboard-coverage
65+
runs-on: ubuntu-latest
66+
67+
steps:
68+
- uses: actions/checkout@v4
69+
with:
70+
filter: tree:0
71+
72+
- name: Check dashboard wiring
73+
run: node scripts/check_dashboard_coverage.js
74+
6075
aot-publish:
6176
name: aot-publish (linux-x64, ${{ matrix.tfm }})
6277
runs-on: ubuntu-latest

CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ All notable changes to Celerity are documented here. This project follows [Keep
1717

1818
### Fixed
1919

20+
- **The `EnumMap` and `EnumSet` cards on the benchmark dashboard rendered empty.** The page required an `(ItemCount: N)` suffix on every result name, and those two benchmarks deliberately declare no item-count sweep — the enum universe is bounded — so their measurements were published to `gh-pages` and then discarded at render time. Both cards now chart their real numbers, on the grid and on their detail pages, and unparameterized benchmarks are excluded from the headline speedup stats rather than bucketed at zero items. Closes [#301](https://github.com/marius-bughiu/Celerity/issues/301).
21+
- A blank dashboard card is now a red CI check instead of a silent gap. `scripts/check_dashboard_coverage.js` asserts that the two `COLLECTIONS` arrays agree, that every charted collection is registered in the CI benchmark suite, and — against the merged benchmark report — that every published result name parses and every card resolves to both a BCL and a Celerity measurement. It runs on every PR in `ci.yml` and, in full, in the aggregate job of `benchmarks.yml`. Closes [#301](https://github.com/marius-bughiu/Celerity/issues/301).
2022
- **The coverage gate measured only one of the six shipped packages.** Coverlet's assembly filter is exact-match, so `Celerity.Hashing`, `Celerity.Primitives`, and the three showcase packages had been outside the gate since the 2.0.0 package split — any of them could have dropped to 0% with CI green. All six are now measured, the gaps that exposed are backfilled to **100% line and branch** coverage, and the floor is raised from 95%/90% to match. Closes [#314](https://github.com/marius-bughiu/Celerity/issues/314).
2123

2224
## [2.4.0] - 2026-07-26

CONTRIBUTING.md

Lines changed: 23 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,15 +76,35 @@ dotnet run -c Release -- --filter '*' # run everything with the default (slow, h
7676

7777
### CI
7878

79-
The `benchmarks` job in [`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs the full suite on `ubuntu-latest` after `build-and-test` succeeds. It uses a faster `CiConfig` (3 warmup × 5 measurement iterations) so the whole suite completes in ~5 min.
79+
[`.github/workflows/benchmarks.yml`](.github/workflows/benchmarks.yml) runs the CI-tracked core suite (the `CoreBenchmarks` array in `Program.cs`) at full BenchmarkDotNet accuracy, sharded across a parallel matrix. On a PR each shard measures its slice of both the PR head and the `main` tip back-to-back on the same runner, so hardware variance cancels; an aggregate job stitches the shard reports back together.
8080

8181
Results are parsed by [`benchmark-action/github-action-benchmark`](https://github.com/benchmark-action/github-action-benchmark) and:
8282

83-
- **On a PR**: a comment is posted with the comparison vs the last `main` baseline. If any benchmark regresses by more than **200%** (i.e. is 2× slower or worse), the job fails red. The threshold is deliberately loose because GitHub-hosted runners are noisy — we'll tighten it once we have history to calibrate against.
84-
- **On a push to `main`**: the new measurement is appended to the `gh-pages`-stored history powering the dashboard at `https://marius-bughiu.github.io/Celerity/dev/bench/` (enable Pages on the `gh-pages` branch once the first run creates it).
83+
- **On a PR**: a comment is posted with the same-runner A/B comparison vs `main`. Rows that move by more than ±10% *and* beyond the combined standard deviation of both measurements are flagged; the flags are advisory, so a noisy row does not fail the job.
84+
- **On a push to `main`**: the new measurement is appended to the `gh-pages`-stored history powering the dashboard at <https://marius-bughiu.github.io/Celerity/dev/bench/>.
8585

8686
If a change is motivated by performance, include before/after numbers from a local Release run in the PR description — the CI job is a guardrail, not a precision instrument. Numbers without `-c Release` are not useful — BenchmarkDotNet refuses to run in Debug.
8787

88+
### The dashboard
89+
90+
The dashboard reads BenchmarkDotNet result *names*, so a benchmark's naming is part of its contract with the site. A name that the page's parser does not recognise is dropped silently — the data publishes to `gh-pages` correctly and the card just renders blank, with nothing red anywhere.
91+
92+
Two rules keep a benchmark chartable:
93+
94+
- Methods are named `{TypeName}_{Op}``EnumSet_Contains`, `Dictionary_Lookup`. The type name decides whether the row is the Celerity arm or the BCL baseline (`BCL_TYPES` in the dashboard source), and `{Op}` is what the card is titled.
95+
- A `[Params]` sweep property must be called **`ItemCount`**. A class may declare no sweep at all — `EnumMap` / `EnumSet` are bounded by the enum universe, so a synthetic item count would chart a dimension that does not exist — in which case the dashboard renders a single bucket. Any *other* property name is rejected rather than charted under an "items" label it does not mean.
96+
97+
Adding a collection to the site means updating three lists by hand, since the published data alone does not tell the page what to draw: the ship card in `web/index.html`, and the `COLLECTIONS` array in **both** `web/dev/bench/index.html` and `web/dev/bench/detail.html` (`items: [NO_SWEEP]` for an unparameterized class).
98+
99+
[`scripts/check_dashboard_coverage.js`](scripts/check_dashboard_coverage.js) enforces all of this so the failure mode is a red check rather than a blank card. It lifts the `COLLECTIONS` tables and the name parsers out of the dashboard HTML rather than reimplementing them, so it validates the code that actually ships. Run it any time you touch a benchmark or the dashboard:
100+
101+
```bash
102+
node scripts/check_dashboard_coverage.js # structural checks
103+
node scripts/check_dashboard_coverage.js path/to/joined-report-full.json # + verify the data
104+
```
105+
106+
The structural half — the two `COLLECTIONS` arrays agree, and every charted collection has a `{Key}Benchmark` registered in `CoreBenchmarks` — runs on every PR in `ci.yml`. The full check runs in the aggregate job of `benchmarks.yml`, against the merged report, and additionally asserts that every published name parses and that every card resolves to both a BCL and a Celerity measurement.
107+
88108
## Versioning
89109

90110
Celerity uses [MinVer](https://github.com/adamralph/minver) to derive NuGet package versions exclusively from **git tags**. There is no `<Version>` or `<PackageVersion>` property in any `.csproj` file — the single source of truth is the `v`-prefixed annotated tag on the commit that represents a release.

ROADMAP.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,11 +225,12 @@ The 2026-Q3 review surveyed the shipped surface against the current .NET 8/9/10
225225

226226
**A fourth core package: `Celerity.Sorting`.** `Array.Sort` / `MemoryExtensions.Sort` are scalar comparison introsort with no radix, counting, or selection path for primitive keys — and the BCL structurally cannot close it, because `Array.Sort` is contractually in-place while radix needs `O(n)` scratch. That is precisely the flexibility-for-speed trade this project's Vision licenses, against a named BCL counterpart. Would layer on `Celerity.Primitives`, mirroring how `Hashing` and `Collections` layer today. Status: `planned` — see the package-scoping caveat below.
227227

228-
**Build- and release-pipeline integrity.** Three guards the repo advertises but does not have.
228+
**Build- and release-pipeline integrity.** Guards the repo advertises but does not have.
229229

230230
- The coverage gate measures one of the six shipping assemblies. `src/coverage.runsettings` filters to `[Celerity]*` with the comment "Measure only the shipping library assembly" — written when there was one. `Celerity.Hashing`, `Celerity.Primitives` and the three showcase packages are unmeasured, while `CONTRIBUTING.md` and `CLAUDE.md` describe the 95%/90% gate as library-wide. Status: `planned`.
231231
- Nothing can fail after the NuGet push. `release.yml` pushes six packages irreversibly, *then* extracts the release notes and creates the GitHub Release — so an over-long release body (a failure this repo has actually hit) leaves a half-published release. The notes check should be hoisted ahead of the push. Status: `planned`.
232232
- No API-compatibility gate. Six packages publish on a tag with no `ApiCompat` / `PackageValidation` / public-API-baseline check anywhere in the repo — in a project that already needed a hand-written `TypeForwarders.cs` to survive one assembly split. Status: `planned`.
233+
- No guard on the benchmark dashboard. The site parses BenchmarkDotNet result *names*, so a benchmark it cannot parse is dropped at render time — the data publishes correctly and the card just goes blank, with no CI signal. `EnumMap` and `EnumSet` had rendered empty since they shipped (they declare no `[Params]` sweep, by design), and `DisjointSet` blanked for five runs when its params property was briefly named `ElementCount`. Status: `done` — the parser now treats the `ItemCount` suffix as optional and renders an unparameterized class as a single bucket, excluded from the headline stats; `scripts/check_dashboard_coverage.js` fails CI on an unparseable name, a card with no measurements behind it, a collection missing from either `COLLECTIONS` array, or one not registered in the CI benchmark suite. It lifts those tables and the parsers out of the dashboard HTML rather than reimplementing them, so the check cannot drift from the page it guards. Tracked in [#301](https://github.com/marius-bughiu/Celerity/issues/301).
233234

234235
Two areas were judged real but deliberately deferred rather than rostered: a `Celerity.Statistics` package (DDSketch / reservoir sampling / running moments — a coherent fourth axis, but two new packages in one cycle is too much at once), and a batch of fuzz-target and AOT-smoke-coverage gaps (real, but low expected defect yield; better folded into whichever collection PR lands next than pursued on their own).
235236

0 commit comments

Comments
 (0)