Error "Benchmark was built without optimization enabled" even with DontFailOnError

[CaspecoMarcus]

I have a simple .NET Framework 4.7.1 project using BenchmarkDotNet 0.12.1.

I need to benchmark in debug mode and read that I needed to add the JitOptimizationsValidator.DontFailOnError validator to stop getting this error:

Assembly PerformanceBenchmarksTwo which defines benchmarks references non-optimized Api.Models
Assembly PerformanceBenchmarksTwo which defines benchmarks is non-optimized
Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.
If you want to debug the benchmarks, please see https://benchmarkdotnet.org/articles/guides/troubleshooting.html#debugging-benchmarks.

However, it doesn't seem to work. I can (for reasons described in #1441) not use DebugBuildConfig(), which does remove the error for me.
This is my code:

public class Program
{
    public class SimpleBenchmark
    {
        [Benchmark]
        public void MySimpleTest()
        {
            Console.WriteLine("Anything");
        }
    }

    private static void Main()
    {
        BenchmarkRunner.Run<SimpleBenchmark>(new CustomConfig());
    }

    private class CustomConfig : ManualConfig
    {
        public CustomConfig()
        {
            AddJob(new Job());
            AddValidator(JitOptimizationsValidator.DontFailOnError);
            AddLogger(ConsoleLogger.Default);
            AddColumnProvider(DefaultColumnProviders.Instance);
        }
    }
}

Thanks!

[adamsitnik]

Hi @Caspeco-MarcusOtterstrom

We really wanted to make sure that it's hard to achieve (to make sure that users are not ignoring this error when they run benchmarks in Debug mode by mistake) and you also need to use ConfigOptions.DisableOptimizationsValidator:

private static void Main()
{
    var config = var config = new ManualConfig()
        .WithOptions(ConfigOptions.DisableOptimizationsValidator)
        .AddValidator(JitOptimizationsValidator.DontFailOnError)
        .AddLogger(ConsoleLogger.Default)
        .AddColumnProvider(DefaultColumnProviders.Instance);

    BenchmarkRunner.Run<SimpleBenchmark>(config);
}

This should have been mentioned in the error message:

BenchmarkDotNet/src/BenchmarkDotNet/Validators/JitOptimizationsValidator.cs

Line 36 in 98d9f83

$"{Environment.NewLine}\tIf you don't, you can disable this policy by using 'config.With(ConfigOptions.DisableOptimizationsValidator)'."

Could you please provide a full error that you are getting? I want to make sure that this is printed.

[CaspecoMarcus]

Hello @adamsitnik
This is the full error message I am getting:

// Validating benchmarks:
Assembly PerformanceBenchmarks which defines benchmarks references non-optimized Api.Models
Assembly PerformanceBenchmarks which defines benchmarks references non-optimized Api.Data
Assembly PerformanceBenchmarks which defines benchmarks references non-optimized Api.Logic
Assembly PerformanceBenchmarks which defines benchmarks references non-optimized SecretsManager
Assembly PerformanceBenchmarks which defines benchmarks references non-optimized PayrollSystemEngine
Assembly PerformanceBenchmarks which defines benchmarks is non-optimized
Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.
If you want to debug the benchmarks, please see https://benchmarkdotnet.org/articles/guides/troubleshooting.html#debugging-benchmarks.
No exporters defined, results will not be persisted.
// ***** BenchmarkRunner: Start   *****

That link does not seem to provide any information about the ConfigOptions.
But what you mentioned does work, I added this line in the constructor of my custom config, which did remove the warnings.

Options |= ConfigOptions.DisableOptimizationsValidator;

Thank you for this awesome library!

[adamsitnik]

@Caspeco-MarcusOtterstrom thanks!

I've sent #1454 to make sure nobody is ever missing this information