Ability to recompile for each target runtime

[stephentoub]

Today in order to compare runtimes, I need to build against a single runtime (the lowest common denominator) with -f and then I can execute against multiple runtimes with --runtimes. That works in the majority use case, however sometimes you want to answer the question "how does this code compare on different runtimes?", and that includes compile-time impact due to new overloads being introduced, which necessitates building against the appropriate reference assemblies for each version.

[timcassell]

Possibly related? It would be nice to be able to recompile targets with different versions of a library/dll. Currently, I can use 2 different versions of the same library in separate projects, both projects referenced by the benchmark project and each called with a different benchmark method, and that compiles just fine, but fails at runtime with System.Reflection.TargetInvocationException.

I'm not sure how that could be achieved, considering I have to include the references in the csproj itself.

[timcassell]

This is already working for Core runtimes. I sent a PR to fix this for Framework in #2370. I'm also aware of classic Mono not being able to rebuild per target framework, but I'm not sure if we need to add that support. Do you need that, and are there any other runtimes that need this and are not already being rebuilt @stephentoub?

[stephentoub]

core and framework are the two I've personally cared about.

[stephentoub]

@timcassell, can you clarify what you mean about this already working for Core? I'm not seeing that. I see a project get build that targets the original .dll and whatever target it was built for.

[timcassell]

@stephentoub I added tests showing that we are recompiling. Can you share a repro of what you are seeing?

[stephentoub]

nuget.config:

<configuration>
  <packageSources>
    <add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />
  </packageSources>
</configuration>

Program.cs

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Environments;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using Microsoft.Extensions.Configuration;
using System;

var config = DefaultConfig.Instance
    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core70).WithId(".NET 7.0").WithNuGet("Microsoft.Extensions.Configuration", "7.0.0").AsBaseline())
    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core80).WithId(".NET 8.0").WithNuGet("Microsoft.Extensions.Configuration", "8.0.0-rc.1.23402.3"))
    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core80)
        .WithNuGet("Microsoft.Extensions.Configuration", "8.0.0-rc.1.23402.3")
        .WithNuGet("Microsoft.Extensions.Configuration.Binder", "8.0.0-rc.1.23402.3")
        .WithArguments(new Argument[] { new MsBuildArgument("/p:EnableConfigurationBindingGenerator=true") }));
BenchmarkSwitcher.FromAssembly(typeof(Tests).Assembly).Run(args, config);

[HideColumns("Error", "StdDev", "Median", "RatioSD", "NuGetReferences")]
[MemoryDiagnoser(false)]
public partial class Tests
{
    private MyConfigSection _data = new MyConfigSection();
    private IConfiguration _config;

    [GlobalSetup]
    public void Setup()
    {
        Environment.SetEnvironmentVariable("MyConfigSection__Message", "Hello World!");
        _config = new ConfigurationBuilder()
            .AddEnvironmentVariables()
            .Build();
    }

    [Benchmark]
    public void Load() => _config.Bind("MyConfigSection", _data);

    internal sealed class MyConfigSection
    {
        public string Message { get; set; }
    }
}

Benchmarks.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net8.0;net7.0</TargetFrameworks>
    <LangVersion>Preview</LangVersion>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <ServerGarbageCollection>true</ServerGarbageCollection>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="benchmarkdotnet" Version="0.13.6" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="8.0.0-rc.1.23402.3" Condition=" '$(TargetFramework)' == 'net8.0' " />
  </ItemGroup>

</Project>

Then run with:

dotnet run -c Release -f net7.0 --filter **

Method	Job	Runtime	Arguments	Mean	Ratio	Allocated	Alloc Ratio
Load	.NET 7.0	.NET 7.0	Default	2.828 us	1.00	1.3 KB	1.00
Load	.NET 8.0	.NET 8.0	Default	2.097 us	0.74	1.37 KB	1.05
Load	Job-UWBTBT	.NET 8.0	/p:EnableConfigurationBindingGenerator=true	2.045 us	0.72	1.37 KB	1.05

That last line should be closer to 60ns if it were actually being compiled as desired, and if you add <EnableConfigurationBindingGenerator>true</EnableConfigurationBindingGenerator> to the PropertyGroup in the csproj, you'll see it drops to that (but so does the other .NET 8 row).

[timcassell]

Oh I see, that's a separate issue. #1377

[stephentoub]

Oh I see, that's a separate issue. #1377

If I --keepFiles, I see the supplied argument being passed to the dotnet build command in the generated .bat file, but that's not the project that needs the argument.

[timcassell]

Do you know how to pass it to the proper project? I couldn't figure it out. When I tried it without BDN, it was passed to both projects, but with BDN it doesn't seem to work.

[stephentoub]

Nope 😦