GCStress=3 has gotten very slow on osx-arm64

[BruceForstall]

Sometime between January and now, test runs on osx-arm64 with DOTNET_GCStress=3 have gotten very, very slow, and now that leg of the runtime-coreclr gcstress0x3-gcstress0xc pipeline fails consistently with many timeouts. E.g.,

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1012958&view=ms.vss-test-web.build-test-results-tab

Unfortunately, we don't have log files going back far enough. The earliest failure listed in the pipeline is 20250312.1. The build before that, which succeeded, is 20250113.1 (I presume the pipeline keeps the latest successful build or two even if it ages out the log files and interim failing builds?)

One specific example is the test JIT/Directed/cmov/Bool_And_Op_cs_d/Bool_And_Op_cs_d.dll in test group Directed_2.0.1. On my M2 MacBook Pro, current runs of just this test, with DOTNET_TieredCompilation=0 and DOTNET_GCStress=3 take 6 minutes 50 seconds. When I sync to a January git hash (010e61a) (and pick up two build fixes), build it, and run, this test takes only about one minute.

@jkotas @janvorli @mangod9 Any ideas?

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[jkotas]

I would start by collecting some high-level perf metrics before/after so we can tell where to look. For example, add the following at the end of the test:

Console.WriteLine(JitInfo.GetCompilationTime(false));
Console.WriteLine(GC.GetGCMemoryInfo().Index);

This number will tell us whether the slow-down occurred when JITing vs. running the code, and whether we are doing more GCs or just wasting cycles on overhead.

[BruceForstall]

With the single Bool_And_Op.cs test extracted, built with dotnet.sh build -c Release, the OLD versus CURRENT source tree on osx-arm64 gives:

elapsed time: 37s vs. 2m31s
GC count: 378799 vs. 1568339
JIT time: 15s vs. 1m53s

so, 4x as many GC, ~7x as much JIT time.

[BruceForstall]

IL size: old: 330173, new: 330225
Generated code size: old: 167664, new: 175360 : ~4.6% more code. Maybe more inlining?

[BruceForstall]

Well, the timing of running just one test in a loop is extremely varied, so I'm not sure I can trust the numbers. With the OLD (January commit), out of 9 runs, I get a low of 39s and a high of 141s, a low of 466,028 GCs and a high of 1,478,406 GCs. I don't know why the number would be so varied, for a test case with no threading or I/O.

[BruceForstall]

I ran the single test 64 times in a row and got drastically varying results:

GCs:
min: 31125, max: 7879019, median: 2697683, stdev: 1627990

time (ms):
min: 3145, max: 712215, median: 232240, stdev: 144855

So I don't think it's worth continuing to investigate this test.

[BruceForstall]

I tried just increasing the timeout on the osx-arm64 GCStress jobs (I doubled it), but we still hit that timeout, with some jobs taking 8 hours. That is obviously unacceptable.

#114725