Skip to content

Commit 5e42a5d

Browse files
mishamyteclaude
andcommitted
bench: rename projects to role-based names (NuGet/Current)
After the baseline moved to 9.0.0, a project named Benchmarks.V8 containing the v9 package was a lie waiting to confuse someone. The names now state roles, not versions: - Benchmarks.V8 -> Benchmarks.NuGet (the published baseline package) - Benchmarks.V9 -> Benchmarks.Current (this source tree) - Benchmarks.YetAnother unchanged (third-party yardstick) Assembly names, build.fsx, benchmark.yml, comments and READMEs follow; the historical v8-vs-v9 result tables keep their version labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 358d868 commit 5e42a5d

12 files changed

Lines changed: 51 additions & 48 deletions

File tree

.github/workflows/benchmark.yml

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ jobs:
5050
runs-on: ubuntu-latest
5151
timeout-minutes: 90
5252
steps:
53-
# Full history + tags for MinVer (the V9 project builds ../src).
53+
# Full history + tags for MinVer (the Current project builds ../src).
5454
- uses: actions/checkout@v6
5555
with:
5656
fetch-depth: 0
@@ -73,7 +73,7 @@ jobs:
7373
- name: Resolve baseline version (latest on NuGet)
7474
id: baseline
7575
run: |
76-
# Keep in sync with the BaselineVersion default in the V8 benchmark fsproj.
76+
# Keep in sync with the BaselineVersion default in the NuGet benchmark fsproj.
7777
# The latest NuGet version is used only when it shares that pin's API major:
7878
# a different major means a different public API, and the baseline project's
7979
# Benchmarks.fs must be migrated before the pin can move.
@@ -93,14 +93,14 @@ jobs:
9393
- name: Run baseline (NuGet ${{ steps.baseline.outputs.version }})
9494
run: >
9595
dotnet run -c Release
96-
--project benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V8
96+
--project benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.NuGet
9797
-p:BaselineVersion=${{ steps.baseline.outputs.version }}
9898
-- --filter "$BENCH_FILTER"
9999
100100
- name: Run source
101101
run: >
102102
dotnet run -c Release
103-
--project benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V9
103+
--project benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.Current
104104
-- --filter "$BENCH_FILTER"
105105
106106
- name: Run YetAnother yardstick
@@ -115,8 +115,8 @@ jobs:
115115
BASELINE_VERSION: ${{ steps.baseline.outputs.version }}
116116
BASELINE_NOTE: ${{ steps.baseline.outputs.note }}
117117
run: |
118-
v8="benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V8/bin/Release/net8.0/BenchmarkDotNet.Artifacts"
119-
v9="benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V9/bin/Release/net8.0/BenchmarkDotNet.Artifacts"
118+
nuget="benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.NuGet/bin/Release/net8.0/BenchmarkDotNet.Artifacts"
119+
current="benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.Current/bin/Release/net8.0/BenchmarkDotNet.Artifacts"
120120
ya="benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.YetAnother/bin/Release/net8.0/BenchmarkDotNet.Artifacts"
121121
122122
{
@@ -127,12 +127,12 @@ jobs:
127127
echo '> `Allocated` is exact and deterministic — treat it as the regression signal.'
128128
echo '> `Mean` on shared CI runners is noisy; deltas within ±10% are not meaningful.'
129129
echo ''
130-
dotnet fsi benchmarks/compare-results.fsx "$v8" "$v9" "v$BASELINE_VERSION" 'source'
130+
dotnet fsi benchmarks/compare-results.fsx "$nuget" "$current" "v$BASELINE_VERSION" 'source'
131131
if [ -d "$ya" ]; then
132132
echo ''
133133
echo '### vs Serilog.Sinks.Loki.YetAnother (yardstick)'
134134
echo ''
135-
dotnet fsi benchmarks/compare-results.fsx "$ya" "$v9" 'YetAnother' 'source'
135+
dotnet fsi benchmarks/compare-results.fsx "$ya" "$current" 'YetAnother' 'source'
136136
fi
137137
} > benchmark-report.md
138138

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Local Loki + Grafana for manual testing: `docker compose up -d loki` (Loki on :3
6767

6868
- **Unit tests** (`tests/...UnitTests`, xUnit + **Unquote** `test <@ ... @>` assertions, all 3 TFMs): wire-format tests assert on exact serialized JSON captured via fake handlers/loopback; `AppSettingsBindingTests` builds a logger from JSON config to pin the appsettings contract; `ExtensionDefaultsTests` guards default-value drift.
6969
- **Integration tests** (`tests/...IntegrationTests`, net10.0 only): Testcontainers spins up a real Loki and queries pushed entries back. Fail fast without Docker — no skip logic. Not part of the FAKE `Default` chain.
70-
- **Benchmarks** (`benchmarks/`): three BenchmarkDotNet executables — V9 = this source (ProjectReference), V8 = the published NuGet baseline (pin via `BaselineVersion`/`VersionOverride`, default 9.0.0 — the "V8" name is historical), and Serilog.Sinks.Loki.YetAnother (yardstick) — sharing sources from `benchmarks/Shared/`. Deliberately **not in the .slnx** (each pins a conflicting Serilog/sink closure with the same assembly name). `Allocated` is the deterministic CI regression signal; `Mean` is noisy. `compare-results.fsx` diffs two result dirs (used by `.github/workflows/benchmark.yml` for PR comparisons against the latest NuGet version).
70+
- **Benchmarks** (`benchmarks/`): three BenchmarkDotNet executables — `Current` = this source (ProjectReference), `NuGet` = the published baseline (pin via `BaselineVersion`/`VersionOverride`, default 9.0.0; CI moves it to the latest same-major release), and `YetAnother` (third-party yardstick) — sharing sources from `benchmarks/Shared/`. Deliberately **not in the .slnx** (each pins a conflicting Serilog/sink closure with the same assembly name). `Allocated` is the deterministic CI regression signal; `Mean` is noisy. `compare-results.fsx` diffs two result dirs (used by `.github/workflows/benchmark.yml` for PR comparisons against the latest NuGet version).
7171

7272
## Conventions and gotchas
7373

benchmarks/README.md

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
1-
# Benchmarks — V8 vs V9 vs YetAnother
1+
# Benchmarks — NuGet baseline vs current source vs YetAnother
22

33
BenchmarkDotNet suites comparing three Grafana Loki sinks:
44

55
| Project | Package | Notes |
66
|---|---|---|
7-
| `…Benchmarks.V8` | `Serilog.Sinks.Grafana.Loki` (NuGet, `BaselineVersion`, default **9.0.0**) | the published **baseline**; CI moves it to the latest same-major release on NuGet |
8-
| `…Benchmarks.V9` | this repo (`../src`, `ProjectReference`) | the current **source**; streams JSON **bytes** through a pooled `Utf8JsonWriter` |
7+
| `…Benchmarks.NuGet` | `Serilog.Sinks.Grafana.Loki` (NuGet, `BaselineVersion`, default **9.0.0**) | the published **baseline**; CI moves it to the latest same-major release on NuGet |
8+
| `…Benchmarks.Current` | this repo (`../src`, `ProjectReference`) | the current **source**; streams JSON **bytes** through a pooled `Utf8JsonWriter` |
99
| `…Benchmarks.YetAnother` | `Serilog.Sinks.Loki.YetAnother` **4.0.5** (NuGet) | third-party yardstick; Serilog 4.x, also a streaming "low-allocation" design |
1010

11-
> Until v9.0.0 shipped, the baseline project compiled against the v8 API (hence its name) and
12-
> pinned `8.3.2`the result tables below are that release-time v8 → v9 comparison, kept as
13-
> the historical record.
11+
> Until v9.0.0 shipped, the baseline project compiled against the v8 API and pinned `8.3.2`
12+
> the result tables below are that release-time v8 → v9 comparison, kept as the historical
13+
> record.
1414
1515
## Why three projects
1616

17-
V8 and V9 ship the **same** assembly name (`Serilog.Sinks.Grafana.Loki.dll`) and cannot
17+
The NuGet and Current projects ship the **same** assembly name (`Serilog.Sinks.Grafana.Loki.dll`) and cannot
1818
coexist in one process. Each contender therefore lives in its own executable;
1919
BenchmarkDotNet runs each in isolation and the reports are merged below. The shared
2020
workload, config and entry point live in `Shared/` and are linked into all three
@@ -25,7 +25,7 @@ projects, so every side measures byte-for-byte identical inputs.
2525
- **Public API only, all sides.** Nothing reaches into internals. Each sink is driven
2626
through its public configuration extension (`WriteTo.GrafanaLoki` / `WriteTo.Loki`).
2727
- **Identical in-process transport.** Each sink exposes an HTTP injection point
28-
(V9 `httpMessageHandler`, V8 `ILokiHttpClient`, YetAnother `httpClient`). A fake drains
28+
(NuGet/Current `httpMessageHandler`, YetAnother `httpClient`). A fake drains
2929
the request body — forcing the real serialization to run — and returns `204` like Loki
3030
on success. No sockets, no network variance. Set `LOKI_BENCH_TARGET` to hit a real Loki
3131
instead (see below).
@@ -42,16 +42,17 @@ projects, so every side measures byte-for-byte identical inputs.
4242
1. **`SinkBenchmarks`** — the full public pipeline (configure → write _N_ events → dispose
4343
→ serialize + POST). The real production path, run for all three sinks.
4444
`EventCount` ∈ {1000, 10000}, `Payload` ∈ {Simple, Exception}.
45-
2. **`FormatterBenchmarks`** — per-event `LokiJsonTextFormatter.Format`, **V8 and V9 only**
46-
(YetAnother has no public per-event formatter). V9's sink doesn't use this public path
47-
in production (it writes bytes directly), so this group is a conservative view of V9.
45+
2. **`FormatterBenchmarks`** — per-event `LokiJsonTextFormatter.Format`, **NuGet and Current
46+
only** (YetAnother has no public per-event formatter). The sink doesn't use this public
47+
path in production (it writes bytes directly), so this group is a conservative view.
4848

4949
## Results
5050

5151
> AMD Ryzen 9 3950X, .NET 8.0 host, BenchmarkDotNet 0.15.8, fake in-process transport.
5252
> Lower is better; **bold** marks the best in each row.
53-
> V8 and YetAnother are fixed NuGet packages, so their numbers are stable references; V9 is
54-
> the current source build. Regenerate the whole table with `dotnet fsi build.fsx -- --target Benchmark`.
53+
> V8 8.3.2 and YetAnother 4.0.5 are fixed NuGet packages, so their numbers are stable
54+
> references; the V9 column was the source build at the v9.0.0 release. (Re-running
55+
> `dotnet fsi build.fsx -- --target Benchmark` today compares the current baseline instead.)
5556
5657
### End-to-end sink — Allocated (MB/op)
5758

@@ -143,7 +144,7 @@ Optional environment variables:
143144
To run a single suite directly:
144145

145146
```bash
146-
dotnet run -c Release --project benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V9
147+
dotnet run -c Release --project benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.Current
147148
```
148149

149150
Reports land in each project's `bin/Release/net8.0/BenchmarkDotNet.Artifacts/results/`

benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V9/Benchmarks.fs renamed to benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.Current/Benchmarks.fs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ open Serilog.Events
2121
open Serilog.Sinks.Grafana.Loki
2222
open Benchmarks.Shared
2323

24-
/// In-process transport for the V9 sink: drains the request body — which forces the
24+
/// In-process transport for the current source sink: drains the request body — which forces the
2525
/// streaming serialization in LokiPushContent to run — then returns 204, exactly like
2626
/// Loki's success response. No sockets, fully deterministic. Injected via the sink's
2727
/// `httpMessageHandler` option, so the sink owns the HttpClient and runs its real path.
@@ -37,8 +37,8 @@ type private Fake204Handler() =
3737
}
3838

3939
// ── Group 1: per-event body formatter (public ITextFormatter surface) ─────────────
40-
// NOTE: the V9 sink does NOT use this public Format path in production — it calls the
41-
// internal byte-oriented FormatToBuffer. This group therefore measures V9's *public*
40+
// NOTE: the sink does NOT use this public Format path in production — it calls the
41+
// internal byte-oriented FormatToBuffer. This group therefore measures the *public*
4242
// formatter (which round-trips through a string to honour the TextWriter contract),
4343
// a deliberately conservative view. The real production path is in Group 2.
4444

benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V9/Serilog.Sinks.Grafana.Loki.Benchmarks.V9.fsproj renamed to benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.Current/Serilog.Sinks.Grafana.Loki.Benchmarks.Current.fsproj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
<PropertyGroup>
44
<OutputType>Exe</OutputType>
55
<TargetFramework>net8.0</TargetFramework>
6-
<AssemblyName>Serilog.Sinks.Grafana.Loki.Benchmarks.V9</AssemblyName>
6+
<AssemblyName>Serilog.Sinks.Grafana.Loki.Benchmarks.Current</AssemblyName>
77
<RootNamespace>Benchmarks</RootNamespace>
88
<!-- BenchmarkDotNet refuses to run a non-optimized build. -->
99
<Optimize>true</Optimize>
@@ -19,7 +19,7 @@
1919

2020
<ItemGroup>
2121
<PackageReference Include="BenchmarkDotNet"/>
22-
<!-- V9 under test: the local source project (multi-targeted; net8.0 asset resolved). -->
22+
<!-- Current = the sink under test: the local source project (multi-targeted; net8.0 asset resolved). -->
2323
<ProjectReference Include="..\..\src\Serilog.Sinks.Grafana.Loki\Serilog.Sinks.Grafana.Loki.fsproj"/>
2424
</ItemGroup>
2525

benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V8/Benchmarks.fs renamed to benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.NuGet/Benchmarks.fs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ open Serilog.Events
2121
open Serilog.Sinks.Grafana.Loki
2222
open Benchmarks.Shared
2323

24-
/// In-process transport for the baseline sink (a published v9.x package): drains the
24+
/// In-process transport for the baseline sink (the published NuGet package): drains the
2525
/// request body — which forces the streaming serialization in LokiPushContent to run —
2626
/// then returns 204, exactly like Loki's success response. No sockets, fully
2727
/// deterministic. Injected via the sink's `httpMessageHandler` option, so the sink owns
@@ -38,8 +38,8 @@ type private Fake204Handler() =
3838
}
3939

4040
// ── Group 1: per-event body formatter (public ITextFormatter surface) ─────────────
41-
// Measures the baseline package's public formatter. Like the source project, this is
42-
// a conservative view: the v9 sink uses the internal byte-oriented path in production,
41+
// Measures the baseline package's public formatter. Like the Current project, this is
42+
// a conservative view: the sink uses the internal byte-oriented path in production,
4343
// which is measured end to end in Group 2.
4444

4545
[<Config(typeof<Config.MicroConfig>)>]

benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.V8/Serilog.Sinks.Grafana.Loki.Benchmarks.V8.fsproj renamed to benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.NuGet/Serilog.Sinks.Grafana.Loki.Benchmarks.NuGet.fsproj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33
<PropertyGroup>
44
<OutputType>Exe</OutputType>
55
<TargetFramework>net8.0</TargetFramework>
6-
<AssemblyName>Serilog.Sinks.Grafana.Loki.Benchmarks.V8</AssemblyName>
6+
<AssemblyName>Serilog.Sinks.Grafana.Loki.Benchmarks.NuGet</AssemblyName>
77
<RootNamespace>Benchmarks</RootNamespace>
88
<!-- BenchmarkDotNet refuses to run a non-optimized build. -->
99
<Optimize>true</Optimize>
10-
<!-- Unlike the V9 project, this one has no F# project reference to drag FSharp.Core
10+
<!-- Unlike the Current project, this one has no F# project reference to drag FSharp.Core
1111
in as a copy-local package, so the SDK's implicit (non-copy-local) reference
1212
leaves FSharp.Core.dll out of the output and the F# exe fails to start. Reference
1313
it explicitly (the version comes from central package management). -->

benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.YetAnother/Benchmarks.fs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ open Benchmarks.Shared
2424
/// In-process transport for Serilog.Sinks.Loki.YetAnother: drains the request body
2525
/// (forcing its streaming serialization to run) and returns 204 like Loki on success.
2626
/// Injected by wrapping it in the HttpClient the sink accepts — the same deterministic,
27-
/// socketless transport used for the V8 and V9 benchmarks.
27+
/// socketless transport used for the NuGet and Current benchmarks.
2828
type private Fake204Handler() =
2929
inherit HttpMessageHandler()
3030

@@ -39,7 +39,7 @@ type private Fake204Handler() =
3939
// ── End-to-end sink push (real production serialization + batching) ────────────────
4040
// YetAnother exposes no public per-event formatter (its message writer is internal),
4141
// so only the end-to-end group has a fair equivalent here. Same workload, same fake
42-
// transport, same batching settings as the V8/V9 SinkBenchmarks.
42+
// transport, same batching settings as the NuGet/Current SinkBenchmarks.
4343

4444
[<Config(typeof<Config.SinkConfig>)>]
4545
type SinkBenchmarks() =

benchmarks/Serilog.Sinks.Grafana.Loki.Benchmarks.YetAnother/Serilog.Sinks.Grafana.Loki.Benchmarks.YetAnother.fsproj

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
<Optimize>true</Optimize>
1010
<MSBuildWarningsAsMessages>MSB3277</MSBuildWarningsAsMessages>
1111
<!-- No F# project reference to drag FSharp.Core in as a copy-local package, so
12-
reference it explicitly (see the V8 project for the full rationale). -->
12+
reference it explicitly (see the NuGet project for the full rationale). -->
1313
<DisableImplicitFSharpCoreReference>true</DisableImplicitFSharpCoreReference>
1414
</PropertyGroup>
1515

benchmarks/Shared/BenchConfig.fs

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,10 @@ open BenchmarkDotNet.Loggers
2727
/// consumed by the CI comparison step), and the allocation diagnoser (whose numbers
2828
/// are exact regardless of the short timing job).
2929
///
30-
/// The artifacts path is pinned to this assembly's output directory. Both benchmark
30+
/// The artifacts path is pinned to this assembly's output directory. The benchmark
3131
/// projects use the same `Benchmarks` namespace, so their report files share names;
32-
/// keeping each project's output under its own bin folder stops V8 overwriting V9.
32+
/// keeping each project's output under its own bin folder stops one project
33+
/// overwriting another.
3334
let private applyCommon (config: ManualConfig) =
3435
config.ArtifactsPath <- Path.Combine(AppContext.BaseDirectory, "BenchmarkDotNet.Artifacts")
3536

0 commit comments

Comments
 (0)