Skip to content

Ratio value is not 1.00 for baseline #2648

Open
@jjanuszkiewicz

Description

@jjanuszkiewicz

For starters, please excuse me if this is not a bug, but my misunderstanding of what the Ratio value represents.

I've created 2 benchmarks, one of them is marked as baseline. I'm getting results such as:

| Method        | Mean       | Error     | StdDev    | Median   | Min      | Max        | Ratio | RatioSD | Baseline | Allocated | Alloc Ratio |
|-------------- |-----------:|----------:|----------:|---------:|---------:|-----------:|------:|--------:|--------- |----------:|------------:|
| SomeMethod_V1 | 1,161.1 ms | 319.86 ms | 943.10 ms | 739.0 ms | 524.7 ms | 5,106.1 ms |  1.40 |    1.35 | Yes      |  12.68 MB |        1.00 |
| SomeMethod_V2 |   498.8 ms |  18.31 ms |  53.99 ms | 469.1 ms | 442.1 ms |   684.2 ms |  0.60 |    0.25 | No       |  10.04 MB |        0.79 |

I expected Ratio to be 1.00 for SomeMethod_V1, which is the baseline (it is 1.00 for Alloc Ratio).

My benchmark code:

[MemoryDiagnoser]
[SimpleJob(RunStrategy.Monitoring, warmupCount: 1, iterationCount: 100)]
[MinColumn, MaxColumn, MeanColumn, MedianColumn, BaselineColumn]
public class SomeMethodBenchmarks
{
	[GlobalSetup]
	public async Task GlobalSetup() { /* ... */ }

	[GlobalCleanup]
	public void GlobalCleanup() { /* ... */ }

	[Benchmark(Baseline = true)]
	public async Task SomeMethod_V1() { /* ... */ }

	[Benchmark]
	public async Task SomeMethod_V2() { /* ... */ }
}

I'm using BenchmarkDotNet 0.14.0 on .NET 8.0.8.

Am I doing something wrong, or is this a bug?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions