test: Improve resilience of MySQL, MSSQL, and NServiceBus integration tests#3656
Open
tippmar-nr wants to merge 2 commits into
Open
test: Improve resilience of MySQL, MSSQL, and NServiceBus integration tests#3656tippmar-nr wants to merge 2 commits into
tippmar-nr wants to merge 2 commits into
Conversation
… tests - MySQL: retry transient CreateProcedure failures and make DROP idempotent via a shared MySqlRetryHelper. - MsSqlTruncation: drop the non-deterministic BaselinePayloadBytes envelope check (truncation is already proven by the length/ellipsis assertions); keep the fixture mechanism for future use. - NsbSend: wait for the transaction_sample_data harvest and speed up the transaction-traces cycle so the trace-sample assertion is deterministic.
WaitForLogLines reads the agent log while the agent writes it. A momentary incompatible share mode on the writer side makes the reader's FileStream open throw a transient IOException, which the wait loop did not retry, aborting the exercise on its first iteration. Retry the open briefly in AgentLogFile.GetFileLines so a read that races a write no longer fails the test.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3656 +/- ##
=======================================
Coverage 81.99% 81.99%
=======================================
Files 511 511
Lines 34724 34724
Branches 4134 4134
=======================================
+ Hits 28471 28473 +2
+ Misses 5277 5276 -1
+ Partials 976 975 -1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens four integration tests against transient/timing-dependent failures observed in CI.
MySQL
Stored-procedure setup occasionally failed on transient errors and on re-runs where the procedure/table already existed. Added a shared
MySqlRetryHelperthat retries transientCreateProcedurefailures and makes drops idempotent; wired intoMySqlExerciserandMySqlConnectorExerciser.MSSQL truncation
MsSqlTruncationTestsopted into aBaselinePayloadBytessize-envelope check that summed all collector payloads across a non-deterministic number of harvest cycles, producing flaky +-5% failures. Truncation correctness is already proven deterministically by the existinglength == 4096andEndsWith("...")assertions, so the envelope check (and its companionWait 5000padding command) is removed. TheBaselinePayloadBytesmechanism remains in the shared fixture for possible future use.NServiceBus
NsbSendTestsasserts on a transaction trace (TryGetTransactionSample) but never waited for the trace harvest and left the transaction-traces cycle at the 60s default while only speeding up metrics. When the trace harvest had not yet fired,Assert.NotNull(transactionSample)failed. AddedConfigureFasterTransactionTracesHarvestCycle(10)and aWaitForLogLine(TransactionSampleLogLineRegex)so the read is deterministic.Agent-log read race (harness)
WaitForLogLinesreads the agent log while the agent writes it. A momentary incompatible share mode on the writer side made the reader'sFileStreamopen throw a transientIOException("being used by another process"), which the wait loop did not retry -- it aborted the exercise on its first iteration, so no data was harvested and the test's metric assertions all failed (seen onPostgresSqlExecuteScalarAsyncTestsCoreLatest).AgentLogFile.GetFileLinesnow retries the file open briefly (10 attempts, 100ms apart) so a read that races a write no longer fails the test. This fix applies to every test that reads the agent log.Testing
NsbSendTestsFW48passed 3/3 consecutive local runs after the NServiceBus fix.PostgresSqlExecuteScalarAsyncTestsCoreLatestpassed 2/2 local runs after the harness fix, confirming no regression to normal log reading. The share-mode race is not deterministically reproducible locally, so the harness retry was verified for non-regression rather than by forcing the original failure.