How can I selectively combine the parameters to avoid full Cartesian product of all Params?

[xtofs]

I want to write a benchmark where I want to compare two approaches. Both have two input parameters but the second one is of a slightly different type for each method. So I have essentially an A and a B and a B' parameter

If I put both benchmark methods into the same benchmark class, I get the Cartesian product of all parameter combinations A x B x B' for each benchmark. (in the example below 2 * 3 * 3 * 3 ) But I really want A x B and A x B' (in the example below 2 * 3 * 3)

I can write two different benchmark classes and copy the first parameter set A or share it somehow through C#. But then I get two output tables.

To Illustrate (truly just illustration, these are obviously not the real benchmarks.)

    public class DemoBenchmark
    {
        [Params("A", "B", "C")]
        public string name;

        [Params(1, 2, 3)]
        public int count;

        [Params(5, 6, 7)]
        public long longCount;

        [Benchmark]
        public bool EvalMethod1() => name.Length == count;

        [Benchmark(Baseline = true)]
        public bool EvalMethod2() => name.Length == longCount;
    }

Is there a way to either

• annotate the Benchmark method to ignore one parameter and don't iterate over that parameter?
• or alternatively combine the results of two benchmark classes into one output table ?

[adamsitnik]

Hello @xtofs !

In such a case, you could use ArgumentsSource and define the combinations on your own.

Example:

[Benchmark]
[ArgumentsSource(nameof(FirstBenchmarkArguments))]
public void FirstBenchmark(string arg0, int arg1) { }

public IEnumerable<object[]> FirstBenchmarkArguments()
{
    foreach (string arg0 in new[] { "A", "B", "C" })
    {
        foreach (int arg1 in new int[] { 1, 2, 3 })
        {
            yield return new object[] { arg0, arg1 };
        }
    }
}

[Benchmark]
[ArgumentsSource(nameof(SecondBenchmarkArguments))]
public void SecondBenchmark(string arg0, long arg1) { }

public IEnumerable<object[]> SecondBenchmarkArguments()
{
    foreach (string arg0 in new[] { "A", "B", "C" })
    {
        foreach (long arg1 in new long[] { 5, 6, 7 })
        {
            yield return new object[] { arg0, arg1 };
        }
    }
}

However, the disadvantage of using Arguments vs Params is that the arguments are not available to setup and cleanup methods. To overcome this problem you can use a solution described in https://github.com/dotnet/performance/blob/master/docs/microbenchmark-design-guidelines.md#argumentssource

[adamsitnik]

annotate the Benchmark method to ignore one parameter and don't iterate over that parameter?

No

or alternatively combine the results of two benchmark classes into one output table ?

Yes. If you are using console line arguments, you need to pass --join. If you are using our config API then you should use following config:

DefaultConfig.Instance.With(ConfigOptions.JoinSummary)

I am going to close this issue. Please feel free to reopen it if something remains unanswered or the solution simply does not work for you.

[patricksadowski]

How can this be done for params that need setup and clean up? I tried a custom type overriding ToString(). This leads to less columns in the results and affects exporters.

[adamsitnik]

How can this be done for params that need setup and clean up?

You need to perform the setup in the argument type ctor and cleanup in IDisposable.Dispose. The latter was recently added (#1571) and has not been released to nuget.org yet, but we plan to do it next week. If you can't wait, you can use our preview packages as described here

public class Argument : IDisposable
{
    public Argument()
    {
        // setup
    }

    public void Dispose()
    {
        // cleanup
    }
    
    public override string ToString() => "this gets displayed in results";
}