[PERF] Occasionally hitting CryptographicException on performance CI runs

[LoopedBard3]

Description

In the dotnet/performance repo ci runs, we are occasionally hitting errors while running some of the Security tests. This pops up in most of our overall ci runs but not consistently across all the tests of similar types. The failing test is also not consistently the same test. Being part of our ci runs, these are being run in containers in the Azure pipelines agents. The specific container is mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04 and is running on an Ubuntu 22.04 agent.

Part of this issue is to figure out if this is something that can somewhat be expected to be flaky on VMs and we should disable the tests for VM runs, or if there is an issue worth investigating. We are not seeing this on our full bare-metal testing, so it is only impacting our ci.

Example run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1001256&view=logs&j=ae025322-dfce-5e7d-abab-0043144d2912&t=af5e1e87-c830-5989-f0c8-c9ce5f9f1829, job "performance micro ubuntu 2204 Open 9.0" -> "send to helix" failing job.

Reproduction Steps

Untested repro (run on a VM):
Clone dotnet/performance.
From the top level the performance repo, run python3 ./scripts/benchmarks_ci.py --csproj ./src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture x64 -f net9.0 --dotnet-versions 9.0.105-servicing.25164.42 --bdn-arguments="--anyCategories Libraries Runtime --logBuildOutput --generateBinLog --filter *SslStreamTests*"

Expected behavior

Test runs successfully.

Actual behavior

Full benchmark run with error:

// Benchmark: SslStreamTests.HandshakeRSA4096CertAsync: Job-ZTJJFB(PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [protocol=Tls12]
// *** Execute ***
// Launch: 1 / 1
// Execute: dotnet MicroBenchmarks-Job-ZTJJFB-1.dll --anonymousPipes 224 225 --benchmarkName "System.Net.Security.Tests.SslStreamTests.HandshakeRSA4096CertAsync(protocol: Tls12)" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0" --benchmarkId 189 in /datadisks/disk1/work/AB3E0955/w/9BCA083E/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/MicroBenchmarks-Job-ZTJJFB-1/bin/Release/net9.0
// Failed to set up high priority (Permission denied). In order to run benchmarks with high priority, make sure you have the right permissions.
// BeforeAnythingElse

// Benchmark Process Environment Information:
// BenchmarkDotNet v0.14.1-nightly.20250107.205
// Runtime=.NET 9.0.3 (9.0.325.11113), X64 RyuJIT AVX2
// GC=Concurrent Workstation
// HardwareIntrinsics=AVX2,AES,BMI1,BMI2,FMA,LZCNT,PCLMUL,POPCNT VectorSize=256
// Job: Job-VKMUPW(PowerPlanMode=00000000-0000-0000-0000-000000000000, InvocationCount=1, IterationCount=1, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)

// BeforeActualRun

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
   at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.MapOpenSsl30Code(X509VerifyStatusCode code)
   at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.MapVerifyErrorToChainStatus(X509VerifyStatusCode code)
   at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.AddElementStatus(X509VerifyStatusCode errorCode, List`1 elementStatus, List`1 overallStatus, Boolean& overallHasNotSignatureValid)
   at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.AddElementStatus(ErrorCollection errorCodes, List`1 elementStatus, List`1 overallStatus, Boolean& overallHasNotSignatureValid)
   at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.BuildChainElements(WorkingChain workingChain, List`1& overallStatus)
   at System.Security.Cryptography.X509Certificates.OpenSslX509ChainProcessor.Finish(OidCollection applicationPolicy, OidCollection certificatePolicy)
   at System.Security.Cryptography.X509Certificates.ChainPal.BuildChainCore(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, X509Certificate2Collection customTrustStore, X509ChainTrustMode trustMode, DateTime verificationTime, TimeSpan timeout, Boolean disableAia)
   at System.Security.Cryptography.X509Certificates.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, X509Certificate2Collection customTrustStore, X509ChainTrustMode trustMode, DateTime verificationTime, TimeSpan timeout, Boolean disableAia)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Net.Security.SslStreamCertificateContext.Create(X509Certificate2 target, X509Certificate2Collection additionalCertificates, Boolean offline, SslCertificateTrust trust, Boolean noOcspFetch)
   at System.Net.Security.SslAuthenticationOptions.UpdateOptions(SslServerAuthenticationOptions sslServerAuthenticationOptions)
   at System.Net.Security.SslStream.AuthenticateAsServerAsync(SslServerAuthenticationOptions sslServerAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Security.Tests.SslStreamTests.HandshakeAsync(X509Certificate certificate, SslProtocols sslProtocol) in /datadisks/disk1/work/AB3E0955/w/9BCA083E/e/performance/src/benchmarks/micro/libraries/System.Net.Security/SslStreamTests.cs:line 189
   at BenchmarkDotNet.Helpers.AwaitHelper.GetResult(Task task)
   at BenchmarkDotNet.Autogenerated.Runnable_189.<.ctor>b__3_2(SslProtocols arg0) in /datadisks/disk1/work/AB3E0955/w/9BCA083E/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/MicroBenchmarks-Job-ZTJJFB-1/MicroBenchmarks-Job-ZTJJFB-1.notcs:line 33258
   at BenchmarkDotNet.Autogenerated.Runnable_189.WorkloadActionUnroll(Int64 invokeCount) in /datadisks/disk1/work/AB3E0955/w/9BCA083E/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/MicroBenchmarks-Job-ZTJJFB-1/MicroBenchmarks-Job-ZTJJFB-1.notcs:line 33332
   at BenchmarkDotNet.Engines.Engine.Measure(Action`1 action, Int64 invokeCount)
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineStage.RunIteration(IterationMode mode, IterationStage stage, Int32 index, Int64 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineActualStage.RunSpecific(Int64 invokeCount, IterationMode iterationMode, Int32 iterationCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineActualStage.Run(Int64 invokeCount, IterationMode iterationMode, Boolean runAuto, Int32 unrollFactor, Boolean forceSpecific)
   at BenchmarkDotNet.Engines.EngineActualStage.RunWorkload(Int64 invokeCount, Int32 unrollFactor, Boolean forceSpecific)
   at BenchmarkDotNet.Engines.Engine.Run()
   at BenchmarkDotNet.Autogenerated.Runnable_189.Run(IHost host, String benchmarkName) in /datadisks/disk1/work/AB3E0955/w/9BCA083E/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/MicroBenchmarks-Job-ZTJJFB-1/MicroBenchmarks-Job-ZTJJFB-1.notcs:line 33239
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /datadisks/disk1/work/AB3E0955/w/9BCA083E/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/MicroBenchmarks-Job-ZTJJFB-1/MicroBenchmarks-Job-ZTJJFB-1.notcs:line 57
// AfterAll
No Workload Results were obtained from the run.
// Benchmark Process 115213 has exited with code 255.

Regression?

No response

Known Workarounds

No response

Configuration

We see this happening on tests of both the latest net10.0 and the latest net9.0 dotnet versions. We are only seeing this on our Linux runs, Ubuntu 22.04.

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[vcsjones]

Most likely seems to be that OpenSSL is giving us an "I could not build the chain" error we don't know how to handle. Unfortunately release builds of the runtime don't tell us what the error was.

These can be tricky to reproduce because chain building can depend on a number of environmental factors, but I will give it a shot.

[vcsjones]

I'm having a hard time getting the performance tests to run with output like:

[2025/04/03 21:09:00][INFO] The command could not be loaded, possibly because:
[2025/04/03 21:09:00][INFO]   * You intended to execute a .NET application:
[2025/04/03 21:09:00][INFO]       The application 'restore' does not exist.
[2025/04/03 21:09:00][INFO]   * You intended to execute a .NET SDK command:
[2025/04/03 21:09:00][INFO]       A compatible .NET SDK was not found.
[2025/04/03 21:09:00][INFO]
[2025/04/03 21:09:00][INFO] Requested SDK version: 10.0.100-preview.3.25167.3
[2025/04/03 21:09:00][INFO] global.json file: /performance/global.json
[2025/04/03 21:09:00][INFO]
[2025/04/03 21:09:00][INFO] Installed SDKs:
[2025/04/03 21:09:00][INFO]
[2025/04/03 21:09:00][INFO] Install the [10.0.100-preview.3.25167.3] .NET SDK or update [/performance/global.json] to match an installed SDK.

@LoopedBard3 do I need to do anything once I drop in to a container, or do I need to be using a particular branch/tag from the dotnet/performance repository?

[LoopedBard3]

Ah, I forgot that we do some minor changes when running older sdks to make global.json happy. All you should need to do is replace the global.json file with the contents of global.net9.json file. This will get rid of the error requesting a 10.0.100* dotnet sdk version.