Nullable ref warnings surface only while documents are *closed*

[AArnott]

Version Used: VS 17.14 (36017.23.d17.14)

While I have VS configured with these C# settings:

I get nullable ref annotation related warnings for documents that are closed. When I double-click the errors, the document opens and the errors disappear. When I close the document, they re-appear.
I shouldn't be seeing these diagnostics.

@sharwell helped me understand that xunit has diagnostic suppression analyzers running, which is why I don't get these warnings in a real build or while the document is open. But when the document is closed, the compiler diagnostics for nullable references run while the xunit suppression analyzers do not run. As a result, I get a noisy error list with ghost errors (warnings as errors in my case).

Steps to Reproduce:

1. Configure VS as shown in the screenshot above.
2. Create a test project that uses xunit (a version that lacks the [NotNull] annotation on Assert.NotNull)
3. Write code like the following:

   object? o = null;
   Assert.NotNull(o);
   Console.WriteLine(o.ToString());

Expected Behavior:

No diagnostic about dereferencing a null value, whether the document is open or closed.

Actual Behavior:

The diagnostic appears when the document is closed, but disappears when it is open.