Change Feed Processor multiple instances

[lei-ardanis]

Hi guys,
I am trying out v3 lib and notice some behaviors which I couldn't fully understand.

I have multiple instances with unique name for the same processor in a few separate host machines. I configured it something like this according the spec

.WithInstanceName(_**someuniquename**_)

All the processor seems to be starting fine with no issues, however it looks only one processer keeps consuming the feeds for the past few days, Is this normal ? I would expect multiple instance would pick up the feeds randomly ?

Thanks.

[ealsur]

This is covered in our docs:

• https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed-processor#deployment-unit
• https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed-processor#dynamic-scaling

Sounds like your lease container has 1 lease.

[lei-ardanis]

Thanks Ealsur, you are right, it looks there is only one instance registered. Multiple hosts are running the same processor and configs as below, each of the processer has a unique instance name as you can see

 _processor = cosmosClient.GetContainer("databaseName",
                    "mainContainer")
                .GetChangeFeedProcessorBuilder<TODO>(
                    "toDoConsumer", HandleChangesAsync)
                .WithInstanceName(Guid.NewGuid().ToString())
                .WithLeaseContainer(leaseContainer)
                .Build();

Only one instance is present in the lease container something like below.
"LeaseToken": "0",
"FeedRange": {
"Range": {
"min": "",
"max": "FF"
}
},
"Owner": "a3fe46a1-56b3-4b07-81ed-d3b1529f547s",

I thought each instance will register in the lease container automatically when starting up with their unique name ? Is there anything else we need to do ?

[ealsur]

That's not how it works, from the documentation, leases exist in the lease container and they are not a factor of how many instances you have running. The number of leases can grow as the collection grows. Each lease represents a "slice" of data in your monitored collection. As the collection grows, more leases will be dynamically added, independently of the number of processors.

The instances are running and they pick and acquire leases following an equal distribution algorithm and once they acquire a lease, they process the change feed for the slice of data it represents.

So if you have a small collection, 1 lease will be present, and if you have 5 instances, only 1 instance will acquire that lease (as per the documentation, a lease can only be owned by one instance at any given time, but an instance can own many leases). The rest of the instances will sit idle.

If the collection was bigger, you could have, for example, 10 leases, which would distribute as 2 leases per instance (assuming 5 instances), and each instance will process it's acquired leases.

As the collection grows, new leases get added, and running instances will automatically acquire them and apply the equal distribution algorithm.

The leases are a mechanism to distribute compute (increase the CPU pull), if the CPU of a single instance processing all leases is quite low, then adding more instances is not necessarily helping much.

[lei-ardanis]

Thank you so much for the clarification, that's clear