Batch Removals with MergeManyChangeSets and Single Stream of Change Sets

[Bulgrak]

Hi!
I have been exploring the operators .MergeManyChangeSets() and .Group(), and I find them particularly useful for emitting a single stream of change sets when performing batch additions that involve multiple groupings.

However, I'm wondering why this is not the case when performing batch removals that involve multiple groupings?
Here is an example to demonstrate this:

using DynamicData;
using DynamicData.Binding;

public class Topic
{
    public int Id { get; init; }
    public required string Category { get; init; }
    public int Value { get; init; }
    public DateTime Timestamp { get; init; }

    public override string ToString() =>
        $"Id:{Id} Category:{Category} Value:{Value} Time:{Timestamp:HH:mm:ss}";
}


class Program
{
    static void Main()
    {
        var topicCache = new SourceCache<Topic, int>(e => e.Id);

        var latestTopicForEachCategoryObservable = topicCache.Connect()
            .Group(x => x.Category)
            .MergeManyChangeSets(x =>
            {
                return x.Cache.Connect()
                    .Top(SortExpressionComparer<Topic>.Descending(t => t.Timestamp), 1);
            });

        latestTopicForEachCategoryObservable.Subscribe(result =>
        {
            Console.WriteLine("Subscriber Updated:");
            Console.WriteLine(result);
            foreach (var topic in result.ToList())
            {
                Console.WriteLine(topic);
            }

            Console.WriteLine(new string('-', 50));
        });

        var now = DateTime.Parse("2025-01-01 10:00:00");
        topicCache.AddOrUpdate([
            new Topic { Id = 1, Category = "A", Timestamp = now.AddSeconds(-10), Value = 1 },
            new Topic { Id = 2, Category = "A", Timestamp = now.AddSeconds(-15), Value = 2 },
            new Topic { Id = 3, Category = "B", Timestamp = now.AddSeconds(-5), Value = 3 },
            new Topic { Id = 4, Category = "B", Timestamp = now.AddSeconds(-10), Value = 4 }
        ]);

        topicCache.Clear();
        
        Console.ReadKey();
    }
}

This produces the following output:

Subscriber Updated:
ChangeSet<Topic.Int32>. Count=2
Add, Key: 1, Current: Id:1 Category:A Value:1 Time:09.59.50, Previous: <None>
Add, Key: 3, Current: Id:3 Category:B Value:3 Time:09.59.55, Previous: <None>
--------------------------------------------------
Subscriber Updated:
ChangeSet<Topic.Int32>. Count=1
Remove, Key: 1, Current: Id:1 Category:A Value:1 Time:09.59.50, Previous: <None>
--------------------------------------------------
Subscriber Updated:
ChangeSet<Topic.Int32>. Count=1
Remove, Key: 3, Current: Id:3 Category:B Value:3 Time:09.59.55, Previous: <None>
--------------------------------------------------

My observations:

• When calling topicCache.AddOrUpdate(...), a single stream of change sets is emitted, which is what I expected.
• However, when calling topicCache.Clear(), multiple streams of change sets are emitted—one for each group—rather than a single stream of change sets.

Is this behavior by design, or is there a specific reason why .MergeManyChangeSets() handles batch removals differently than batch additions?
Any insights or guidance would be greatly appreciated. Thanks!

[dwcullop]

@Bulgrak That's a great question. The answer is ... It isn't MergeManyChangeSets that handles them differently. It is actually the Group operator that handles them differently and MergeManyChangeSets can only work with what it is given.

Why does Group handle them differently? I'm not 100% sure, but it is probably for historical reasons. That particular overload is using older code, and maybe when it was written, making sure that remove changes were batched wasn't a critical requirement.

There are other overloads that use a newer implementation and I went to great lengths to ensure that changes where batched together as much as possible.

If you group with this line instead:

.Group(Observable.Return((Topic x) => x.Category), Observable.Empty<Unit>())

You'll see that you get the behavior that you expect.

(That last Empty parameter should be an optional parameter, but there's a minor bug such that it will throw if you don't provide it. Fix is in #987 ...)

Perhaps we should consider updating the implementation to better match the other operator, but that risks potentially breaking people who are depending on the old behavior.

[JakenVeina]

I'm inclined to say that grouping or non-grouping of changesets falls in the category of implementation detail, except maybe in a few operators that are specifically ABOUT grouping things up. So yeah, I would consider that an improvement we can make in older operators.

[dwcullop]

It would be easy to do because we already have the dynamic Grouping code. We would just have to invoke it from the basic overloads instead of the more complicated ones. I'll have to look to see if there is unnecessary overhead, but I don't think it will be a problem.

I completely agree that an operator should only emit one changeset in response to a Clear whenever possible.

[JakenVeina]

I'll go a step further and say, ideally, ANY notification or action incoming within an operator should produce only one output changeset. The only scenarios where I can imagine this being not desirable are at the binding layer, where binding adapters can split things apart as they see fit.