Open
Description
When running this benchmark:
using System;
using System.Linq;
using BenchmarkDotNet.Attributes;
using Ardalis.Extensions.Enumerable;
using System.Collections.Generic;
namespace Ardalis.Extensions.Benchmarks.Enumerable;
[MemoryDiagnoser]
[ReturnValueValidator(failOnError: true)]
public class ProductBenchmarks
{
[Params(4, 5, 6, 7)]
public int N { get; set; }
private int[] _numbers;
[GlobalSetup]
public void GlobalSetup()
{
_numbers = System.Linq.Enumerable.Range(1, N).ToArray();
Console.WriteLine($"N = {N} and _numbers has a length of {_numbers.Length}");
}
[Benchmark(Baseline = true)]
public long ProductAggregate()
{
return _numbers.ProductAggregate();
}
[Benchmark]
public int ProductForEach()
{
return _numbers.ProductForEach();
}
[Benchmark]
public int Product()
{
return _numbers.Product();
}
}
static class ProductBenchmarksExtensions
{
public static int ProductAggregate(this IEnumerable<int> numbers)
{
return numbers.Aggregate(1, (acc, x) => acc * x);
}
public static int ProductForEach(this IEnumerable<int> numbers)
{
int product = 2; // <- Set to 2 so that we have different return values and make validation fail
foreach (var x in numbers)
{
product *= x;
}
return product;
}
}I get the following output:
// Validating benchmarks:
N = 4 and numbers has a length of 4
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=4]
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=5]
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=6]
Inconsistent benchmark return values in ProductBenchmarks: ProductAggregate: 24, ProductForEach: 48, Product: 24 [N=7]
The methods being benchmarked are returning the factorial of N. As you can see from the return values being identical, the GlobalSetup method that updates the List of integers to multiply doesn't seem to be ran before validating the return values for the subsequent Ns.
I can confirm that the GlobalSetup method runs before each benchmark for each N when there is not failing validation and we proceed with benchmarking.
Feel free to clone this repo, here is the exact commit to reproduce the above state: https://github.com/snowfrogdev/Ardalis.Extensions/tree/b17763f52b890e25098557095c1ec954168cafe4