What steps can I take to identify if an observed issue is a regression or the intended system behavior?

[cw-balaji]

My company is in the process of upgrading our gRPC-based applications from .NET 6.0 to .NET 8.0. After the upgrade, we observed a significant change in CPU usage across different hardware platforms:

• On AMD-based machines, we saw a 20–30% reduction in CPU usage — a clear performance improvement.
• On ARM-based machines, however, we observed a 20–30% increase in CPU usage for the same workloads.

Our dotnet applications run inside Kubernetes and we selected ARM machines primarily for their cost-effectiveness, this increase is concerning. We would like to understand whether this behavior is:

1. A known performance regression on ARM architectures in .NET 8.0,
2. An intended side effect of architectural changes or optimizations targeted at x86/x64,
3. Or potentially the result of misconfiguration or environmental differences.

I tried running micro benchmarks from the performance repository, I ran the Serializers benchmarks and the results are:

AWS m6a.large x86_64 (AMD EPYC 7R13, AVX2) :

Method	Mean	Error	StdDev	Median	Min	Max	Gen0	Allocated
SystemTextJson_Reflection	262.0 ns	5.91 ns	6.80 ns	258.2 ns	256.9 ns	277.1 ns	0.0918	192 B
SystemTextJson_SourceGen	179.2 ns	0.29 ns	0.22 ns	179.3 ns	178.8 ns	179.4 ns	0.0911	192 B
JSON.NET	442.4 ns	1.77 ns	1.57 ns	442.4 ns	439.8 ns	445.0 ns	0.7187	1504 B
Jil	358.0 ns	1.90 ns	1.68 ns	358.1 ns	356.0 ns	360.7 ns	0.3787	792 B
Utf8Json	130.9 ns	0.67 ns	0.52 ns	130.8 ns	130.4 ns	132.4 ns	0.0913	192 B

AWS m6g.large ARM (Neoverse-N1, Arm64, AdvSIMD):

Method	Mean	Error	StdDev	Median	Min	Max	Gen0	Allocated
SystemTextJson_Reflection	437.6 ns	0.23 ns	0.21 ns	437.6 ns	437.2 ns	437.9 ns	0.0105	192 B
SystemTextJson_SourceGen	356.4 ns	0.23 ns	0.20 ns	356.4 ns	356.1 ns	356.7 ns	0.0101	192 B
JSON.NET	808.6 ns	9.17 ns	8.58 ns	811.3 ns	791.5 ns	817.8 ns	0.0888	1504 B
Jil	652.8 ns	3.73 ns	3.31 ns	651.5 ns	648.1 ns	658.4 ns	0.0452	792 B
Utf8Json	248.7 ns	0.39 ns	0.36 ns	248.7 ns	248.1 ns	249.3 ns	0.0109	192 B

Steps I followed to run this:

1. Launched AWS instance with the default Amazon Linux 2023 ami.
2. Installed dotnet using the command sudo yum update && sudo yum install dotnet-sdk-8.0.
3. Git clone performance benchmark and changed the global.json to global.net8.json.
4. Ran dotnet run -c Release -f net8.0 --filter MicroBenchmarks.Serializers*.

Could you please advise whether this performance change on ARM is expected in .NET 8.0, or if it warrants further investigation as a possible regression, or are we missing some configurations?

[huoyaoyuan]

You can try to compare if there's significant difference in compiled assembly code using [DisassemblyDiagnoser], or doing profiling to check where is the difference from.

[cw-balaji]

I used [DisassemblyDiagnoser] on some benchmarks and got assembly instructions for both ARM and AMD architectures, but I'm having trouble understanding them because there are many different instructions.

[huoyaoyuan]

Since the regressions are reproducible with official benchmarks, it can be investigated by our side. @eiriktsarpalis do you know who is the best one to handle this, or is there any known issue?

[huoyaoyuan]

Another tip is to try more recent version too, including 9.0 and 10.0 preview. If the performance is better for latest version, it's unlikely to be fixed for 8.0 specifically.

[eiriktsarpalis]

All official benchmarks are being run continuously on a fleet of many different types of hardware so any regressions we can detect are being automatically filed as issues. Serializers are very complex pieces of software, so I'm curious if you're able to detect similar regressions in simpler benchmarks from the official suite.

[cw-balaji]

As suggested by @huoyaoyuan I ran the serializers benchmark with 9.0 and 10.0 preview I got the following results:

Serializer	.NET 8.0 AMD64 Mean / Alloc	.NET 8.0 ARM64 Mean / Alloc	.NET 9.0 AMD64 Mean / Alloc	.NET 9.0 ARM64 Mean / Alloc	.NET 10.0 AMD64 Mean / Alloc	.NET 10.0 ARM64 Mean / Alloc
SystemTextJson_Reflection	262.0 ns / 192 B	437.6 ns / 192 B	275.5 ns / 192 B	442.6 ns / 192 B	241.4 ns / 192 B	460.9 ns / 192 B
SystemTextJson_SourceGen	179.2 ns / 192 B	356.4 ns / 192 B	171.6 ns / 192 B	337.8 ns / 192 B	182.1 ns / 192 B	344.8 ns / 192 B
JSON.NET	442.4 ns / 1504 B	808.6 ns / 1504 B	423.6 ns / 1504 B	734.8 ns / 1504 B	417.7 ns / 1504 B	861.7 ns / 1504 B
Jil	358 ns / 792 B	652.8 ns / 792 B	344.6 ns / 792 B	667.1 ns / 792 B	343.2 ns / 792 B	703.4 ns / 792 B
Utf8Json	130.9 ns / 192 B	248.7 ns / 192 B	117.9 ns / 192 B	240.3 ns / 192 B	143.4 ns / 192 B	244.7 ns / 192 B

Additionally, I ran a small simple benchmark on floor operation as suggested by @eiriktsarpalis , and observed these results:

Method	.NET 8.0 AMD64 Mean / Alloc	.NET 8.0 ARM64 Mean / Alloc	.NET 9.0 AMD64 Mean / Alloc	.NET 9.0 ARM64 Mean / Alloc	.NET 10.0 AMD64 Mean / Alloc	.NET 10.0 ARM64 Mean / Alloc
Floor	4.884 us / -	4.484 us / -	5.238 us / -	4.349 us / -	5.278 us / -	4.386 us / -

After reviewing both sets of results, I'm even more puzzled by the performance discrepancy. We use protobuf-net and Google.Protobuf for serialization, and we also use reflection to invoke methods dynamically based on function names represented as strings. I ran benchmarks on the reflection-related parts of the code, and they showed performance trends similar to those of the serialization benchmarks.