Some MonoAOT BDN individual tests are failing with PlatformNotSupportedException

[LoopedBard3]

There seem to be 40ish tests that are failing when running benchmark dotnet for MonoAOT. I am not seeing obvious commonalities between them as they are occurring in different test groups and may not be failing on the same test each run. A run with these failures is here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2414622&view=results, and an example log is:

 // Benchmark: SearchValuesCharTests.ContainsAny: Job-QLSPWG(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Values=abcdefABCDEF0123456789]
[2024/03/26 04:12:05][INFO] // *** Execute ***
[2024/03/26 04:12:05][INFO] // Launch: 1 / 1
[2024/03/26 04:12:05][INFO] // Execute: /home/helixbot/work/9AA00889/w/AC350967/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-QLSPWG/bin/Release/net9.0/linux-x64/publish/Job-QLSPWG --anonymousPipes 188 189 --benchmarkName "System.Buffers.Tests.SearchValuesCharTests.ContainsAny(Values: \"abcdefABCDEF0123456789\")" --job "PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Toolchain=MonoAOTLLVM, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1" --benchmarkId 19 in /home/helixbot/work/9AA00889/w/AC350967/e/performance/artifacts/bin/MicroBenchmarks/Release/net9.0/Job-QLSPWG/bin/Release/net9.0/linux-x64/publish
[2024/03/26 04:12:05][INFO] // Failed to set up high priority (Permission denied). In order to run benchmarks with high priority, make sure you have the right permissions.
[2024/03/26 04:12:05][INFO] // BeforeAnythingElse
[2024/03/26 04:12:05][INFO] 
[2024/03/26 04:12:05][INFO] // Benchmark Process Environment Information:
[2024/03/26 04:12:05][INFO] // BenchmarkDotNet v0.13.13-nightly.20240311.145
[2024/03/26 04:12:05][INFO] // Runtime=.NET 9.0.0 (42.42.42.42424) using MonoVM, X64 SSE4.2
[2024/03/26 04:12:05][INFO] // GC=Non-concurrent Workstation
[2024/03/26 04:12:05][INFO] // HardwareIntrinsics=SSE4.2,AES,BMI1,BMI2,LZCNT,PCLMUL,POPCNT VectorSize=128
[2024/03/26 04:12:05][INFO] // Job: Job-EXVQIU(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2024/03/26 04:12:05][INFO] 
[2024/03/26 04:12:05][INFO] OverheadJitting  1: 1 op, 4288.00 ns, 4.2880 us/op
[2024/03/26 04:12:05][INFO] 
[2024/03/26 04:12:05][INFO] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[2024/03/26 04:12:05][INFO]  ---> System.PlatformNotSupportedException: Operation is not supported on this platform.
[2024/03/26 04:12:05][INFO]    at System.Buffers.IndexOfAnyAsciiSearcher.IndexOfAnyCore[Boolean,DontNegate,Default,ContainsAnyResultMapper`1](Int16& searchSpace, Int32 searchSpaceLength, AsciiState& state)
[2024/03/26 04:12:05][INFO]    at System.Buffers.Tests.SearchValuesCharTests.ContainsAny()
[2024/03/26 04:12:05][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_19.WorkloadActionNoUnroll(Int64 invokeCount)
[2024/03/26 04:12:05][INFO]    at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
[2024/03/26 04:12:05][INFO]    at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
[2024/03/26 04:12:05][INFO]    at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
[2024/03/26 04:12:05][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_19.Run(IHost host, String benchmarkName)
[2024/03/26 04:12:05][INFO]    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
[2024/03/26 04:12:05][INFO]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
[2024/03/26 04:12:05][INFO]    --- End of inner exception stack trace ---
[2024/03/26 04:12:05][INFO]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
[2024/03/26 04:12:05][INFO]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2024/03/26 04:12:05][INFO]    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2024/03/26 04:12:05][INFO]    at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
[2024/03/26 04:12:05][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args)
[2024/03/26 04:12:05][INFO] // AfterAll
[2024/03/26 04:12:05][INFO] No Workload Results were obtained from the run.
[2024/03/26 04:12:05][INFO] // Benchmark Process 103460 has exited with code 255.

[LoopedBard3]

FYI: @matouskozak. From what I can tell, this shouldn't impact the large majority of the tests, but it will make the pipeline red.

[LoopedBard3]

More notes from some investigation:
This seems to be the range where it started based on what I found in the engsrvprod (helix) ADX:
Comparing 9c828a17c347118968d7a1a170178daf4a9ce126...27214a0787f9a0ee95c799639431d9d16c25d1ac · dotnet/runtime (github.com)

With this specific information:
Job ID's:
Bad: 24306758
Good: 24303771

Job Names:
Bad: b2e29a9a-c1ba-46dd-a7ab-14445717a680
Good: 75553a8a-11b2-4825-8720-18cbd9d6ed36

Build Number:
Bad: 20240311.4
Good: 20240311.2

Build Hashes:
Bad: 27214a0787f9a0ee95c799639431d9d16c25d1ac
Good: 9c828a17c347118968d7a1a170178daf4a9ce126

In this range, we also merged in a BDN update which may be related: #4044

[matouskozak]

@fanyang-mono do you know if dotnet/runtime@fd2f2a7 could cause these System.PlatformNotSupportedException: Operation is not supported on this platform errors?

[matouskozak]

Reproduced the issue with the latest Performance repo (2891b9ffab138ae81edd351480a7c56100356961) using sudo python3 benchmarks_local.py --commits "27214a0787f9a0ee95c799639431d9d16c25d1ac" --run-types MonoAOTLLVM --filter *SearchValuesCharTests.ContainsAny* on Ubuntu-x64.

Update: the issue is reproducible on the "good" build hash: sudo python3 benchmarks_local.py --commits "9c828a17c347118968d7a1a170178daf4a9ce126" --run-types MonoAOTLLVM --filter *SearchValuesCharTests.ContainsAny* as well, hinting that it might be the BDN update.

Update2: with the same Runtime commit, the issue is not present in BDNVersion: 0.13.13-nightly.20240228.135