Bad LOH allocations when deregistering from IOptionsMonitor

[Tragetaschen]

Description

We have an Orleans cluster and in the silo, each grain depends on global configuration. Since that can change during runtime, we are using IOptionsMonitor and when the grain is done, we dispose the handle we got from the OnChange call. That last dispose call is the lone source of some hefty LOH allocations. The grains come and go constantly and at a certain load on the silo (around 2500 grains), LOH allocations start to happen.

As far as I can tell, IOptionsMonitor.Dispose only deregisters from the internal multicast delegate and this deregistration is the main culprit: The multicast delegate allocates a new invocation list for each -=, so memory consumption becomes O(n) while it's amortized O(1) for += due to exponential growth.

This small benchmark shows the problem. The allocations grow with the number of already registered delegates.

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<Benchmark>();

[SimpleJob(iterationCount: 5)]
[MemoryDiagnoser]
public class Benchmark
{
    private Action? _methods;

    [Params(8, 10, 12, 14)]
    public int Power { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        _methods += Method;
        for (var i = 0; i < Power; ++i)
        {
            _methods += _methods;
        }
    }

    [Benchmark]
    public void AddRemove()
    {
        var m = new Action(Method);
        _methods += m;
        _methods -= m;
    }

    private void Method() { }
}

| Method    | Power | Mean       | Error      | StdDev     | Gen0     | Gen1     | Gen2     | Allocated |
|---------- |------ |-----------:|-----------:|-----------:|---------:|---------:|---------:|----------:|
| AddRemove | 8     |   1.512 us |  0.8605 us |  0.2235 us |   0.7629 |   0.0172 |        - |   6.23 KB |
| AddRemove | 10    |   5.376 us |  2.8169 us |  0.4359 us |   2.9602 |   0.1831 |        - |  24.23 KB |
| AddRemove | 12    |  18.840 us |  1.0445 us |  0.1616 us |  11.7493 |   1.4648 |        - |  96.23 KB |
| AddRemove | 14    | 135.817 us | 74.9703 us | 19.4695 us | 124.8779 | 124.8779 | 124.8779 | 384.28 KB |

[tarekgh]

@MihaZupan this issue is complaining about the perf of the delegate registration and deregistration. It is not related to the extension's options.

[MihaZupan]

A multicast delegate is immutable. Adding/removing items will be slow as the collection grows.
Maybe there's room for improvement, but I wouldn't expect things to fundamentally change complexity-wise.

The issue is complaining that registering and unregistering from IOptionsMonitor many times is extremely slow once you have lots of callbacks.
If this is a scenario that extension options wants to improve, it should likely be using a different mechanism, e.g. a Dictionary (hence why I switched the areas).

[tarekgh]

Maybe there's room for improvement, but I wouldn't expect things to fundamentally change complexity-wise.

From the benchmark numbers, it looks to me the allocated memory is exponentially growing as you increase the registration/deregistration. Sure, we can optimize the Options part, but this is not going to help in the general case for users using such multicast delegates.

[Tragetaschen]

The benchmark is increasing the number of registered baseline delegates exponentially (2^Power). The memory growth for the individual deregistration is only linear in the number of registered delegates.

[Tragetaschen]

Since the multicast delegate is immutable, there is an optimization along the += path where an existing invocation list is reused and shared between different immutable instances as long as there is room in the underlying array.

[tarekgh]

@Tragetaschen does your concern regarding IOptionsMonitor only?