WIP: Allow cosmos tests to run in parallel v2#37766
Draft
JoasE wants to merge 9 commits intodotnet:mainfrom
Draft
WIP: Allow cosmos tests to run in parallel v2#37766JoasE wants to merge 9 commits intodotnet:mainfrom
JoasE wants to merge 9 commits intodotnet:mainfrom
Conversation
Use guid for store name
…e collection calls and docs
…cosmos-test-parallelism
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With some spare time, I've been experimenting around with the cosmos functional tests to see if I can get them to run in parallel.
The biggest challenge appeared to be the cosmos db emulator, which can become unstable in certain scenario's.
After some testing I found that the create container implementation in the emulator has a race condition, and can crash the emulator when ran in parallel. Other CRUD requests for containers appear to have no direct effect on the running of the emulator, but parallel requests can cause the emulator to have to clear the data upon restart.
In addition, the docs state that the emulator could show performance degradation with more than 10 concurrent containers.
On top of that, my tests have shown the emulator will stop responding to create container requests at some point when creating many containers. On my local machine this happens consistently after the creation of the 25th container.
This means that when running the tests against the emulator we don't execute CRUD container requests in parallel and ideally no more than 10 containers exist at the same time, and never more than 25.
This could (currently) be fixed by just decreasing the parallelism using
CollectionBehavior(MaxParallelThreads = N), but this is risky as it will cause errors or slow tests when many tests are added that create a lot of containers and those happen to run in parallel. See: #37696Another option would be to manually control parallelism in CosmosTestStore based on the number of containers each test store is attempting to create. This is both faster and prevents possible errors, but there are a couple of downsides, being mainly:
This pr implements the latter.
Some other challenges were:
There was a custom import for a cosmos specific Northwind database, which didn't appear to be needed. The custom import was very slow compared to using savechange's new batching, so I removed it and had to slightly change 1 test.
This PR decreases total test run time:
locally:
from 26 to 12.5 minutes
pipeline: TODO
Run against real cosmos db: TODO