System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK(usePartialChain: True) Fails locally

[steveharter]

Description

On my local machine,
System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK(usePartialChain: True)
fails consistently even when run all by itself via
-method System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK

Reproduction Steps

Windows 11
20 Cores

build -s clr+libs+libs.tests -c Release
build -test -s libs.tests -c Release

Note that my machine is located in Fargo, ND at the Microsoft campus so if there are remote Azure machines, for example, that may be a factor.

Expected behavior

No failure

Actual behavior

c:\git\runtimeBaseline\artifacts\bin\System.Net.Quic.Functional.Tests\Release\net7.0-windows>"C:\git\runtimeBaseline\artifacts\bin\testhost\net7.0-windows-Release-x64\dotnet.exe" exec --runtimeconfig System.Net.Quic.Functional.Tests.runtimeconfig.json --depsfile System.Net.Quic.Functional.Tests.deps.json xunit.console.dll System.Net.Quic.Functional.Tests.dll -nologo -notrait category=OuterLoop -notrait category=failing
  Discovering: System.Net.Quic.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Quic.Functional.Tests (found 103 of 112 test cases)
  Starting:    System.Net.Quic.Functional.Tests (parallel test collections = on, max threads = 20)
    System.Net.Quic.Tests.MsQuicPlatformDetectionTests.UnsupportedPlatforms_ThrowsPlatformNotSupportedException [SKIP]
      Condition(s) not met: "IsQuicUnsupported"
    System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK(usePartialChain: True) [FAIL]
      System.Security.Authentication.AuthenticationException : Authentication failed because the remote party sent a TLS alert: 'UserCanceled'.
      Stack Trace:
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs(471,0): at System.Net.Quic.QuicConnection.HandleEventShutdownInitiatedByTransport(_SHUTDOWN_INITIATED_BY_TRANSPORT_e__Struct& data)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs(576,0): at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs(605,0): at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent)
        --- End of stack trace from previous location ---
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ValueTaskSource.cs(176,0): at System.Net.Quic.ValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs(313,0): at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs(76,0): at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs(76,0): at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\tests\FunctionalTests\QuicTestBase.cs(218,0): at System.Net.Quic.Tests.QuicTestBase.CreateConnectedQuicConnection(QuicClientConnectionOptions clientOptions, QuicListener listener)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\tests\FunctionalTests\QuicTestBase.cs(218,0): at System.Net.Quic.Tests.QuicTestBase.CreateConnectedQuicConnection(QuicClientConnectionOptions clientOptions, QuicListener listener)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\tests\FunctionalTests\MsQuicTests.cs(187,0): at System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK(Boolean usePartialChain)
        C:\git\runtimeBaseline\src\libraries\System.Net.Quic\tests\FunctionalTests\MsQuicTests.cs(189,0): at System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK(Boolean usePartialChain)
        --- End of stack trace from previous location ---
  Finished:    System.Net.Quic.Functional.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Quic.Functional.Tests  Total: 322, Errors: 0, Failed: 1, Skipped: 1, Time: 64.296s

Regression?

Not sure as I typically don't run all library tests; I don't remember seeing this a few weeks ago

Known Workarounds

No response

Configuration

No response

Other information

No response

[wfurt]

This test only use loopback @steveharter. Any suggestions how to debug this more @rzikm? I was planning to hook up the test event listener @ManickaP put in for other failure but this may be simply timing out....

[rzikm]

UserCanceled means we killed the connection before handshake finished, so around the time we are invoking the connectionOptions callback or certificate selection callback. A good place to start would be awaiting the QuicListener.AcceptConnection task or placing a breakpoint in QuicListener.PendingConnection.StartHandshake and see which exception pops up there.

runtime/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicListener.PendingConnection.cs

Lines 54 to 79 in 104fe14

	public async void StartHandshake(QuicConnection connection, SslClientHelloInfo clientHello, Func<QuicConnection, SslClientHelloInfo, CancellationToken, ValueTask<QuicServerConnectionOptions>> connectionOptionsCallback)
	{
	try
	{
	_cancellationTokenSource.CancelAfter(s_handshakeTimeout);
	QuicServerConnectionOptions options = await connectionOptionsCallback(connection, clientHello, _cancellationTokenSource.Token).ConfigureAwait(false);
	options.Validate(nameof(options)); // Validate and fill in defaults for the options.
	await connection.FinishHandshakeAsync(options, clientHello.ServerName, _cancellationTokenSource.Token).ConfigureAwait(false);
	_finishHandshakeTask.SetResult(connection);
	}
	catch (Exception ex)
	{
	// Handshake failed:
	// 1. Connection cannot be handed out since it's useless --> return null, listener will wait for another one.
	// 2. Shutdown the connection to send a transport error to the peer --> application error code doesn't matter here, use default.
	if (NetEventSource.Log.IsEnabled())
	{
	NetEventSource.Error(connection, $"{connection} Connection handshake failed: {ex}");
	}
	await connection.CloseAsync(default).ConfigureAwait(false);
	await connection.DisposeAsync().ConfigureAwait(false);
	_finishHandshakeTask.SetException(ex);
	}
	}

[ManickaP]

We have an internal timeout to finish the handshake: https://source.dot.net/#System.Net.Quic/System/Net/Quic/QuicListener.PendingConnection.cs,76d2d524386f0e43,references
Because at this point in time in the handshake process, msquic will not kill the connection on their own: microsoft/msquic#2705

10 seconds seemed like long enough time for the callback and the handshake. Note that if the test failed sooner than 10 seconds, it's something else.

Also note that no user cancellation tokens are used to kill this, they will only cancel pending AcceptConnection operation.

[ManickaP]

Also the exception from handshake will propagate to AcceptConnection on the listener, so you should see it there.

[karelz]

Triage: Looks environmental - CI and our machines work just fine. We should try to track it down on @steveharter's machine ...
Not critical for 7.0.

[steveharter]

Additional info:

Both the true and false argument for the test case input fails when debugging.

Run root exception causing the test to fail is System.Security.Cryptography.CryptographicException: 'Access is denied.' with this call stack

System.Security.Cryptography.dll!System.Security.Cryptography.X509Certificates.StorePal.Add(System.Security.Cryptography.X509Certificates.ICertificatePal certificate = {System.Security.Cryptography.X509Certificates.CertificatePal}) Line 55	C#	Symbols loaded.
 	System.Security.Cryptography.dll!System.Security.Cryptography.X509Certificates.X509Store.Add(System.Security.Cryptography.X509Certificates.X509Certificate2 certificate = {System.Security.Cryptography.X509Certificates.X509Certificate2}) Line 137	C#	Symbols loaded.
>	System.Net.Security.dll!System.Net.Security.SslStreamCertificateContext.SslStreamCertificateContext(System.Security.Cryptography.X509Certificates.X509Certificate2 target = {System.Security.Cryptography.X509Certificates.X509Certificate2}, System.Security.Cryptography.X509Certificates.X509Certificate2[] intermediates = {System.Security.Cryptography.X509Certificates.X509Certificate2[4]}, System.Net.Security.SslCertificateTrust trust = null) Line 77	C#	Symbols loaded.
 	System.Net.Security.dll!System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2 target = {System.Security.Cryptography.X509Certificates.X509Certificate2}, System.Security.Cryptography.X509Certificates.X509Certificate2Collection additionalCertificates = {System.Security.Cryptography.X509Certificates.X509Certificate2Collection}, bool offline = false, System.Net.Security.SslCertificateTrust trust = null, bool noOcspFetch = false) Line 107	C#	Symbols loaded.
 	System.Net.Security.dll!System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2 target = {System.Security.Cryptography.X509Certificates.X509Certificate2}, System.Security.Cryptography.X509Certificates.X509Certificate2Collection additionalCertificates = {System.Security.Cryptography.X509Certificates.X509Certificate2Collection}, bool offline = false, System.Net.Security.SslCertificateTrust trust = null) Line 23	C#	Symbols loaded.
 	System.Net.Quic.Functional.Tests.dll!System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK.AnonymousMethod__0(System.Net.Quic.QuicConnection _ = {System.Net.Quic.QuicConnection}, System.Net.Security.SslClientHelloInfo _ = {System.Net.Security.SslClientHelloInfo}, System.Threading.CancellationToken _ = IsCancellationRequested = false) Line 159	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.PendingConnection.StartHandshake(System.Net.Quic.QuicConnection connection = {System.Net.Quic.QuicConnection}, System.Net.Security.SslClientHelloInfo clientHello = {System.Net.Security.SslClientHelloInfo}, System.Func<System.Net.Quic.QuicConnection, System.Net.Security.SslClientHelloInfo, System.Threading.CancellationToken, System.Threading.Tasks.ValueTask<System.Net.Quic.QuicServerConnectionOptions>> connectionOptionsCallback = {Method = {System.Reflection.RuntimeMethodInfo}}) Line 59	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.HandleEventNewConnection(ref Microsoft.Quic.QUIC_LISTENER_EVENT._Anonymous_e__Union._NEW_CONNECTION_e__Struct data = {Microsoft.Quic.QUIC_LISTENER_EVENT._Anonymous_e__Union._NEW_CONNECTION_e__Struct}) Line 200	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.HandleListenerEvent(ref Microsoft.Quic.QUIC_LISTENER_EVENT listenerEvent = {Microsoft.Quic.QUIC_LISTENER_EVENT}) Line 212	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.NativeCallback(Microsoft.Quic.QUIC_HANDLE* listener = 0x00000241a6500750, void* context = 0x000002010bb41048, Microsoft.Quic.QUIC_LISTENER_EVENT* listenerEvent = 0x0000005c9d7feba0) Line 241	C#	Symbols loaded.

        public void Add(ICertificatePal certificate)
        {
            using (SafeCertContextHandle certContext = ((CertificatePal)certificate).GetCertContext())
            {
                if (!Interop.Crypt32.CertAddCertificateContextToStore(_certStore, certContext, Interop.Crypt32.CertStoreAddDisposition.CERT_STORE_ADD_REPLACE_EXISTING_INHERIT_PROPERTIES, IntPtr.Zero))
                    throw Marshal.GetLastPInvokeError().ToCryptographicException();
            }
        }

However, CryptographicException: 'Access is denied.' also occurs during test start-up in System.Net.Quic.Tests.CertificateSetup(), but the exception is eaten and doesn't cause the tests not to run:

>	System.Security.Cryptography.dll!System.Security.Cryptography.X509Certificates.StorePal.FromSystemStore(string storeName = "CA", System.Security.Cryptography.X509Certificates.StoreLocation storeLocation = LocalMachine, System.Security.Cryptography.X509Certificates.OpenFlags openFlags = ReadWrite) Line 191	C#	Symbols loaded.
 	System.Security.Cryptography.dll!System.Security.Cryptography.X509Certificates.X509Store.Open(System.Security.Cryptography.X509Certificates.OpenFlags flags = ReadWrite) Line 108	C#	Symbols loaded.
 	System.Net.Security.dll!System.Net.Security.SslStreamCertificateContext.SslStreamCertificateContext(System.Security.Cryptography.X509Certificates.X509Certificate2 target = {System.Security.Cryptography.X509Certificates.X509Certificate2}, System.Security.Cryptography.X509Certificates.X509Certificate2[] intermediates = {System.Security.Cryptography.X509Certificates.X509Certificate2[4]}, System.Net.Security.SslCertificateTrust trust = null) Line 51	C#	Symbols loaded.
 	System.Net.Security.dll!System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2 target = {System.Security.Cryptography.X509Certificates.X509Certificate2}, System.Security.Cryptography.X509Certificates.X509Certificate2Collection additionalCertificates = {System.Security.Cryptography.X509Certificates.X509Certificate2Collection}, bool offline = false, System.Net.Security.SslCertificateTrust trust = null, bool noOcspFetch = false) Line 107	C#	Symbols loaded.
 	System.Net.Security.dll!System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2 target = {System.Security.Cryptography.X509Certificates.X509Certificate2}, System.Security.Cryptography.X509Certificates.X509Certificate2Collection additionalCertificates = {System.Security.Cryptography.X509Certificates.X509Certificate2Collection}, bool offline = false, System.Net.Security.SslCertificateTrust trust = null) Line 23	C#	Symbols loaded.
 	System.Net.Quic.Functional.Tests.dll!System.Net.Quic.Tests.MsQuicTests.ConnectWithUntrustedCaWithCustomTrust_OK.AnonymousMethod__0(System.Net.Quic.QuicConnection _ = {System.Net.Quic.QuicConnection}, System.Net.Security.SslClientHelloInfo _ = {System.Net.Security.SslClientHelloInfo}, System.Threading.CancellationToken _ = IsCancellationRequested = false) Line 159	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.PendingConnection.StartHandshake(System.Net.Quic.QuicConnection connection = {System.Net.Quic.QuicConnection}, System.Net.Security.SslClientHelloInfo clientHello = {System.Net.Security.SslClientHelloInfo}, System.Func<System.Net.Quic.QuicConnection, System.Net.Security.SslClientHelloInfo, System.Threading.CancellationToken, System.Threading.Tasks.ValueTask<System.Net.Quic.QuicServerConnectionOptions>> connectionOptionsCallback = {Method = {System.Reflection.RuntimeMethodInfo}}) Line 59	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.HandleEventNewConnection(ref Microsoft.Quic.QUIC_LISTENER_EVENT._Anonymous_e__Union._NEW_CONNECTION_e__Struct data = {Microsoft.Quic.QUIC_LISTENER_EVENT._Anonymous_e__Union._NEW_CONNECTION_e__Struct}) Line 200	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.HandleListenerEvent(ref Microsoft.Quic.QUIC_LISTENER_EVENT listenerEvent = {Microsoft.Quic.QUIC_LISTENER_EVENT}) Line 212	C#	Symbols loaded.
 	System.Net.Quic.dll!System.Net.Quic.QuicListener.NativeCallback(Microsoft.Quic.QUIC_HANDLE* listener = 0x0000024793c70750, void* context = 0x00000206f9601050, Microsoft.Quic.QUIC_LISTENER_EVENT* listenerEvent = 0x000000f032b3eb30) Line 241	C#	Symbols loaded.

        internal static partial IStorePal FromSystemStore(string storeName, StoreLocation storeLocation, OpenFlags openFlags)
        {
            Interop.Crypt32.CertStoreFlags certStoreFlags = MapX509StoreFlags(storeLocation, openFlags);

            SafeCertStoreHandle certStore = Interop.crypt32.CertOpenStore(CertStoreProvider.CERT_STORE_PROV_SYSTEM_W, Interop.Crypt32.CertEncodingType.All, IntPtr.Zero, certStoreFlags, storeName);
            if (certStore.IsInvalid)
            {
                Exception e = Marshal.GetLastPInvokeError().ToCryptographicException();
                certStore.Dispose();
                throw e;
            }

[wfurt]

nice, thanks for the update @steveharter. It seems like we should deal with the error and skip the test when we cannot add needed certificates. Can you pass tests in System.Net.Security/tests/FunctionalTests ??? There is similar pattern and I'm wondering if it works there or not.

[steveharter]

Also note that the tests pass when running in administrator mode. They fail when running as current user, although I am an admin on the machine.

Can you pass tests in System.Net.Security/tests/FunctionalTest

In Sytem.Net.Security.Tests I do also get an auth failure that doesn't occur is running as admin:

    System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_UntrustedCaWithCustomTrust_OK(usePartialChain: True) [FAIL]
      System.Security.Cryptography.CryptographicException : Access is denied.
      Stack Trace:
        C:\git\runtimeBaseline\src\libraries\System.Security.Cryptography\src\System\Security\Cryptography\X509Certificates\StorePal.Windows.cs(55,0): at System.Security.Cryptography.X509Certificates.StorePal.Add(ICertificatePal certificate)
        C:\git\runtimeBaseline\src\libraries\System.Security.Cryptography\src\System\Security\Cryptography\X509Certificates\X509Store.cs(136,0): at System.Security.Cryptography.X509Certificates.X509Store.Add(X509Certificate2 certificate)
        C:\git\runtimeBaseline\src\libraries\System.Net.Security\src\System\Net\Security\SslStreamCertificateContext.Windows.cs(107,0): at System.Net.Security.SslStreamCertificateContext..ctor(X509Certificate2 target, X509Certificate2[] intermediates, SslCertificateTrust trust)
        C:\git\runtimeBaseline\src\libraries\System.Net.Security\src\System\Net\Security\SslStreamCertificateContext.cs(107,0): at System.Net.Security.SslStreamCertificateContext.Create(X509Certificate2 target, X509Certificate2Collection additionalCertificates, Boolean offline, SslCertificateTrust trust, Boolean noOcspFetch)
        C:\git\runtimeBaseline\src\libraries\System.Net.Security\src\System\Net\Security\SslStreamCertificateContext.cs(23,0): at System.Net.Security.SslStreamCertificateContext.Create(X509Certificate2 target, X509Certificate2Collection additionalCertificates, Boolean offline, SslCertificateTrust trust)
        C:\git\runtimeBaseline\src\libraries\System.Net.Security\tests\FunctionalTests\SslStreamNetworkStreamTest.cs(794,0): at System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_UntrustedCaWithCustomTrust_OK(Boolean usePartialChain)
        --- End of stack trace from previous location ---
    System.Net.Security.Tests.SslStreamNetworkStreamTest.SslStream_UnifiedHello_Ok [SKIP]
      Condition(s) not met: "SupportsTls10"
  Finished:    System.Net.Security.Tests
=== TEST EXECUTION SUMMARY ===
   System.Net.Security.Tests  Total: 5040, Errors: 0, Failed: 1, Skipped: 21, Time: 12.484s```

[ManickaP]

Triage: this is general error with certificates, happens with security tests as well. It is also a local problem. Solution might be to avoid calling tests that would fail with access denied (conditional test). Not critical for 8.0.