WriteAsync_CompletesWhenDataReceived fails intermittently

[tmeschter]

The WriteAsync_CompletesWhenDataReceived unit test is failing intermittently. We've see this in both PR and CI builds.

I'm going to disable this test, so this issue track investigating the failure, fixing, and re-enabling the test.

[tmeschter]

In my limited investigations the failures only occur when the isGeneric parameter is false. However, it is possible the failure is due to a timing issue in the test, and the fact that it fails for false and succeeds for true may be due to the scheduling of those two test invocations rather than indicating some deeper problem. Or it might be an actual product bug. 🤷🏻‍♂️

[drewnoakes]

@MiYanni found examples in CI of this failing with both true and false arguments.

[drewnoakes]

I've been playing with this for a bit.

If I run this locally I think I've just found a repro in the debugger.

The writeTask here has an exception:

JoinableTaskContextException: An attempt to switch to the main thread failed to reach the expected thread. Was the JoinableTaskContext initialized on the wrong thread or with a SynchronizationContext whose Post method does not execute its delegate on the main thread?

Abbreviated stack trace:

Microsoft.VisualStudio.Threading.JoinableTaskContextException: An attempt to switch to the main thread failed to reach the expected thread. Was the JoinableTaskContext initialized on the wrong thread or with a SynchronizationContext whose Post method does not execute its delegate on the main thread?
   at Microsoft.VisualStudio.Threading.JoinableTaskFactory.MainThreadAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d.MoveNext()
   at Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.<ExecuteAsync>d__3.MoveNext()
   at Microsoft.VisualStudio.ProjectSystem.OnceInitializedOnceDisposedUnderLockAsync.<ExecuteUnderLockCoreAsync>d__6`1.MoveNext() in D:\repos\project-system\src\Microsoft.VisualStudio.ProjectSystem.Managed\ProjectSystem\OnceInitializedOnceDisposedUnderLockAsync.cs:line 127
   at Microsoft.VisualStudio.ProjectSystem.LanguageServices.Workspace.<WriteAsync>d__45`1.MoveNext() in D:\repos\project-system\src\Microsoft.VisualStudio.ProjectSystem.Managed\ProjectSystem\LanguageServices\Workspace.cs:line 538<---

Full stack trace:

System.AggregateException: One or more errors occurred. ---> Microsoft.VisualStudio.Threading.JoinableTaskContextException: An attempt to switch to the main thread failed to reach the expected thread. Was the JoinableTaskContext initialized on the wrong thread or with a SynchronizationContext whose Post method does not execute its delegate on the main thread?
   at Microsoft.VisualStudio.Threading.JoinableTaskFactory.MainThreadAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.AwaitExtensions.ExecuteContinuationSynchronouslyAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.<ExecuteAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.VisualStudio.ProjectSystem.OnceInitializedOnceDisposedUnderLockAsync.<ExecuteUnderLockCoreAsync>d__6`1.MoveNext() in D:\repos\project-system\src\Microsoft.VisualStudio.ProjectSystem.Managed\ProjectSystem\OnceInitializedOnceDisposedUnderLockAsync.cs:line 127
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.VisualStudio.ProjectSystem.LanguageServices.Workspace.<WriteAsync>d__45`1.MoveNext() in D:\repos\project-system\src\Microsoft.VisualStudio.ProjectSystem.Managed\ProjectSystem\LanguageServices\Workspace.cs:line 538
   --- End of inner exception stack trace ---
---> (Inner Exception #0) Microsoft.VisualStudio.Threading.JoinableTaskContextException: An attempt to switch to the main thread failed to reach the expected thread. Was the JoinableTaskContext initialized on the wrong thread or with a SynchronizationContext whose Post method does not execute its delegate on the main thread?
   at Microsoft.VisualStudio.Threading.JoinableTaskFactory.MainThreadAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.AwaitExtensions.ExecuteContinuationSynchronouslyAwaiter.GetResult()
   at Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.<ExecuteAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.VisualStudio.ProjectSystem.OnceInitializedOnceDisposedUnderLockAsync.<ExecuteUnderLockCoreAsync>d__6`1.MoveNext() in D:\repos\project-system\src\Microsoft.VisualStudio.ProjectSystem.Managed\ProjectSystem\OnceInitializedOnceDisposedUnderLockAsync.cs:line 127
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.VisualStudio.ProjectSystem.LanguageServices.Workspace.<WriteAsync>d__45`1.MoveNext() in D:\repos\project-system\src\Microsoft.VisualStudio.ProjectSystem.Managed\ProjectSystem\LanguageServices\Workspace.cs:line 538<---

Setting the debugger to break on that exception type lead to:

 	Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.MainThreadAwaiter.GetResult()	Unknown
 	Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.ReentrantSemaphore.StackSemaphore.ExecuteAsync.AnonymousMethod__0()	Unknown
 	[Resuming Async Method]	
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	Unknown
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	Unknown
 	mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()	Unknown
 	mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()	Unknown
 	Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.SingleExecuteProtector.TryExecute()	Unknown
 	Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.SingleExecuteProtector..cctor.AnonymousMethod__20_0(object state)	Unknown
>	xunit.execution.desktop.dll!Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback callback, object state) Line 107	C#
 	xunit.execution.desktop.dll!Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc() Line 94	C#
 	xunit.execution.desktop.dll!Xunit.Sdk.XunitWorkerThread..ctor.AnonymousMethod__1_0(object s) Line 72	C#
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	Unknown
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)	Unknown
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	mscorlib.dll!System.Threading.ThreadHelper.ThreadStart(object obj)	Unknown

Note xUnit's custom scheduler Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext.

I suspect some conflict with between JTF and xUnit here.

Next step is to investigate JoinableTaskFactory.MainThreadAwaiter and Xunit.Sdk.MaxConcurrencySyncContext.