Silent write ingestion limits in Dataverse?

[tlienDE]

Hi

I wonder if Dataverse silenty throttles write requests if "too many" requests are received at once. I'm familiar with the official Dataverse throttling docs - this is not about exceeding 6k requests per 5 minute rolling window, processing time per 20 minute window or any of those documented limitations. The official docs even claim to support 52 concurrent requests per user which is far beyond what I'm able to achieve.

My setup

• All query expressions
• All async through the IOrganizationServiceAsync2 interface
• UpdateRequests performed on a new entity with only the updated properties in the attribute collection (one single property)
• Connection pool of 5 ServiceClient instances using their own respective credentials

So far we're following best practices, as far as I'm aware.

Spike and load tests

Important to note:

• All UpdateRequests are performed on different entities in the same table
• No SNAT Port exhaustion has been detected
• Unsure if table locks could be causing the requests to queue up
• Not a single 429 response was received

Now, we don't have a very high load on this specific API. Requests come in bursts as users query the API at predetermined times, therefore we need to be able to handle a very modest amount of concurrent requests with high reliability.

I've gone with 200 requests in my k6 tests.

Spike (200 incoming requests at once)

It seems like a write queue is quickly backing up in Dataverse, linearly increasing the execution time of the UpdateRequest.

First request takes ~1 sec:

Executed Command - Update : UpdateRequest To Dataverse via IOrganizationService : RequestID=8ec05716-6d91-4b6c-bf54-1d241fa0e8a0 : duration=00:00:01.0830160

Last request takes ~58 sec:

Executed Command - Update : UpdateRequest To Dataverse via IOrganizationService : RequestID=bf9ef2f8-910c-4133-a1a5-91c11f2d7468 : duration=00:00:58.3669693

This is obviously absolutely abysmal. Alas, we did just try to update 200 entities at once.

Load 1 (6 requests per second)

Execution time increases linearly just like in the spike test, however much slower.

First request takes 578 ms:

Executed Command - Update : UpdateRequest To Dataverse via IOrganizationService : RequestID=69148546-8080-4c75-90b0-733579111e6b : duration=00:00:00.5786045

Last request takes ~12 sec:

Executed Command - Update : UpdateRequest To Dataverse via IOrganizationService : RequestID=04ada8ff-ab37-454e-83b9-c72e2f8f831f : duration=00:00:12.0312131

Load 2 (3 requests per second)

Execution time remains constant at around ~500 ms.

Requests sent in sequence

If I just send one request at a time, execution time also remains constant.

What gives?

Based on the conducted tests I'm inclined to deduce that Dataverse has a silent cap on concurrent requests. I'm aware that measures can be taken to batch requests, but I just really want to know if anyone else has experienced the same thing that I'm describing here - and whether it's expected behavior.

Thanks.