TestLogsAllCategoriesDefaultLevelFallback fails frequently

[jander-msft]

https://runfo.azurewebsites.net/search/tests/?q=started%3A~7%20definition%3Adiagnostics-public-ci%20name%3A%22testlogsallcategoriesdefaultlevelfallback%22

[jander-msft]

This test seems to only fail on Windows builds.

[jander-msft]

Similarly, the TestLogsUseAppFiltersAndFilterSpecs test fails frequently on Windows: https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E7+definition%3Adiagnostics-public-ci+name%3A%22TestLogsUseAppFiltersAndFilterSpecs%22

[tommcdon]

@josalem

[josalem]

Based on a cursory look at the test failures and code, it looks like this might be a timing issue with the pipeline tests. The failures are all "read past end of stream" (the client exits while we tried to read) or "operation cancelled" (we took too long to read and hit the timeout).

Both failures imply something wrong with the tracee process. The current test structure doesn't capture the sub-process' stderr and stdout, so we can't know what's going on there.

@jander-msft can you repro these locally? They all seem to be in the monitor.pipeline tests.

[jander-msft]

These have never reproduced locally for me.

In general, the tracee processes for many of these types of tests in the diagnostics are uncontrolled which exposes them to timing issues. I think it would be good to rewrite the tracee processes to have a better control flow in order to eliminate stability issues that might be causing these problems. In dotnet-monitor, we have a single test process with multiple scenarios with a single communication mechanism with the test host process that explicitly controls when the test app process should start and stop the intended scenario. Given some time, I can bring that back to diagnostics to potentially help alleviate these types of issues.

[mikem8361]

Not sure if related to this failure but TestLogsWildcardCategory is failing all the time now. We need to fix or disable this test right away.

[2:39 PM] Mike McLaughlin
✘ Microsoft.Diagnostics.Monitoring.EventPipe.UnitTests.EventLogsPipelineUnitTests.TestLogsWildcardCategory
System.Exception : Read past end of stream.  
   at FastSerialization.IOStreamStreamReader.Fill(Int32 minimum)
   at FastSerialization.MemoryStreamReader.ReadByte()
   at FastSerialization.Deserializer.ReadObject()
   at Microsoft.Diagnostics.Tracing.EventPipeEventSource.Process()
   at Microsoft.Diagnostics.Monitoring.EventPipe.DiagnosticsEventPipeProcessor.<>c__DisplayClass11_0.<<Process>b__1>d.MoveNext() in /_/src/Microsoft.Diagnostics.Monitoring.EventPipe/DiagnosticsEventPipeProcessor.cs:line 81
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Diagnostics.Monitoring.EventPipe.DiagnosticsEventPipeProcessor.<>c__DisplayClass11_0.<<Process>b__1>d.MoveNext() in /_/src/Microsoft.Diagnostics.Monitoring.EventPipe/DiagnosticsEventPipeProcessor.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Diagnostics.Monitoring.EventPipe.DiagnosticsEventPipeProcessor.Process(DiagnosticsClient client, TimeSpan duration, CancellationToken token) in /_/src/Microsoft.Diagnostics.Monitoring.EventPipe/DiagnosticsEventPipeProcessor.cs:line 47
   at Microsoft.Diagnostics.Monitoring.Pipeline.RunAsyncCore(CancellationToken token) in /_/src/Microsoft.Diagnostics.Monitoring/Pipeline.cs:line 99
   at Microsoft.Diagnostics.Monitoring.Pipeline.RunAsyncCore(CancellationToken token) in /_/src/Microsoft.Diagnostics.Monitoring/Pipeline.cs:line 109
   at Microsoft.Diagnostics.Monitoring.Pipeline.SafeExecuteTask(Func`1 acquireTask) in /_/src/Microsoft.Diagnostics.Monitoring/Pipeline.cs:line 223
   at Microsoft.Diagnostics.Monitoring.EventPipe.UnitTests.EventLogsPipelineUnitTests.GetLogsAsync(Action`1 settingsCallback) in /_/src/tests/Microsoft.Diagnostics.Monitoring.EventPipe/EventLogsPipelineUnitTests.cs:line 226
   at Microsoft.Diagnostics.Monitoring.EventPipe.UnitTests.EventLogsPipelineUnitTests.GetLogsAsync(Action`1 settingsCallback) in /_/src/tests/Microsoft.Diagnostics.Monitoring.EventPipe/EventLogsPipelineUnitTests.cs:line 227
   at Microsoft.Diagnostics.Monitoring.EventPipe.UnitTests.EventLogsPipelineUnitTests.TestLogsWildcardCategory() in /_/src/tests/Microsoft.Diagnostics.Monitoring.EventPipe/EventLogsPipelineUnitTests.cs:line 187
--- End of stack trace from previous location where exception was thrown ---

[mikem8361]

Disabled EventLogsPipelineUnitTests.TestLogsAllCategoriesAllLevels for Windows in PR #2569

[jander-msft]

Wanted to capture some of the failing builds before they disappear:

Azure DevOps Build: 1301195
Definition: diagnostics-public-ci
GitHub Repository: dotnet/diagnostics
Queue Time: 8/18/2021 5:18:12 AM
Target Branch: main
Pull Request: 2514
Attempts: 2
Result: Failed

Test Run	Test Name	Message
Tests Windows x64 Release	TestLogsAllCategoriesDefaultLevelFallback	Assert.Equal() Failure\r\n ↓ (pos 0)\r\nExpected: Some warning message with 6\r\nActual: Warning message.\r\n ↑ (pos 0)

Azure DevOps Build: 1316659
Definition: diagnostics-public-ci
GitHub Repository: dotnet/diagnostics
Queue Time: 8/24/2021 9:26:32 PM
Target Branch: main
Pull Request: 2508
Attempts: 2
Result: Succeeded

Test Run	Test Name	Message
Tests Windows x86 Release	TestLogsAllCategoriesDefaultLevelFallback	System.Exception : Read past end of stream.

Azure DevOps Build: 1319615
Definition: diagnostics-public-ci
GitHub Repository: dotnet/diagnostics
Queue Time: 8/25/2021 8:11:13 PM
Target Branch: main
Attempts: 1
Result: Failed

Test Run	Test Name	Message
Tests Windows x64 Release	TestLogsAllCategoriesDefaultLevelFallback	System.Exception : Read past end of stream.
Tests Windows x64 Debug	TestLogsUseAppFiltersAndFilterSpecs	Assert.Equal() Failure\r\n ↓ (pos 0)\r\nExpected: Another message\r\nActual: Warning message.\r\n ↑ (pos 0)