[9.0] Test assert failure in X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes

[carlossanlop]

Important: This was found in the 9.0 branch

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789147
Build error leg or test failing: System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(aiaResponseKind: Cert, mustIgnore: False)

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(aiaResponseKind: Cert, mustIgnore: False) [FAIL]",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=789147&view=logs&j=07e4961f-4232-5b4a-5e6a-31092474840f&t=dfe85eac-3de9-57b6-7c34-e995be202bc7&l=55
Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-106995-merge-c7701f6f34bd4379a0/System.Security.Cryptography.Tests/1/console.2a4f1b53.log?helixlogtype=result
Output:

     System.AggregateException : One or more errors occurred. (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False) (Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False)
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      ---- Assert.NotEqual() Failure: Values are equal
      Expected: Not False
      Actual:       False
      Stack Trace:
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(54,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(79,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(AiaResponseKind aiaResponseKind, Boolean mustIgnore)
           at System.Object.InvokeStub_AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(Object , Span`1 )
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(136,0): at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
        ----- Inner Stack Trace #1 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #2 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #3 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #4 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
        ----- Inner Stack Trace #5 (Xunit.Sdk.NotEqualException) -----
        /_/src/libraries/System.Security.Cryptography/tests/X509Certificates/RevocationTests/AiaTests.cs(88,0): at System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.<>c__DisplayClass1_1.<AiaAcceptsCertTypesAndIgnoresNonCertTypes>b__0()
        /_/src/libraries/Common/tests/TestUtilities/System/RetryHelper.cs(45,0): at System.RetryHelper.Execute(Action test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=789147
Error message validated: [System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes(aiaResponseKind: Cert, mustIgnore: False) [FAIL]]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/4/2024 5:20:44 PM UTC

Report

Build	Definition	Test	Pull Request
1044117	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#115583
1043274	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#115590
1043004	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.DisableAiaOptionWorks	#115583
1042092	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#115583
1041937	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#115583
1040292	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestUsageTests.RSACertificateNoPaddingMode	#115507
1028332	dotnet/runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes	#115022
2693496	dotnet-runtime	System.Security.Cryptography.X509Certificates.Tests.RevocationTests.AiaTests.AiaAcceptsCertTypesAndIgnoresNonCertTypes

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	6	8

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[bartonjs]

This failed when AIA was expected to work. Unfortunately, AIA tests can fail under high load (which RetryHelper hopes to outlast). Given that no other AIA tests failed it doesn't seem to be that AIA itself is not working, just that the requests timed out (or somehow the listener port got closed).