Skip to content

Improve Benchmark Accuracy #2336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 23 commits into
base: master
Choose a base branch
from
Open

Improve Benchmark Accuracy #2336

wants to merge 23 commits into from

Conversation

timcassell
Copy link
Collaborator

@timcassell timcassell commented Jun 21, 2023 •
edited
Loading

Fixes #1133
Fixes #2305
Fixes #1802
Fixes #2530

  • Drastically simplified benchmark type code gen.
  • Overhead return type is always void for a fair baseline for all benchmark return types.
  • Benchmark method is called directly instead of wrapped in a delegate.
  • Return value is popped off the stack instead of passed to Consumer or a NoInlining method.
  • Added an assembly weaver that automatically applies MethodImplOptions.NoInlining to benchmark methods.
  • Added additional tests.

@timcassell timcassell mentioned this pull request Jun 21, 2023
Closed
@timcassell timcassell marked this pull request as draft July 24, 2023 15:45
@timcassell timcassell marked this pull request as ready for review August 1, 2023 07:26
This was referenced Aug 9, 2023
Closed
Open
@timcassell timcassell added this to the v0.14.0 milestone Jan 14, 2024
@timcassell timcassell mentioned this pull request Jan 22, 2024
Closed
@timcassell timcassell modified the milestones: v0.15.x, v0.14.0 Mar 6, 2024
@timcassell

This comment was marked as outdated.

Sign in to view
@timcassell timcassell force-pushed the fair-types branch 2 times, most recently from 48266cd to 4abbb1f Compare March 11, 2024 14:08
@timcassell timcassell modified the milestones: v0.14.0, v0.15.x Aug 6, 2024
@timcassell timcassell mentioned this pull request Jan 11, 2025
Closed
@timcassell timcassell marked this pull request as draft January 13, 2025 00:58
@timcassell timcassell mentioned this pull request Feb 13, 2025
Merged
@timcassell timcassell changed the title Fair Return Types Improve Benchmark Accuracy Feb 13, 2025
@timcassell timcassell marked this pull request as ready for review February 13, 2025 21:53
@timcassell
Copy link
Collaborator Author

timcassell commented Feb 13, 2025 •
edited
Loading

Unfortunately, I no longer have an Intel CPU to test this, but I think the new results from this PR should address the issues you had in #2334 @AndreyAkinshin (results look good on my AMD 9800X3D CPU in .Net 8).

@timcassell
Copy link
Collaborator Author

Okay, I have gotten it to pass all tests, and tested it locally against dotnet/performance. The only caveat that should be mentioned is that AsmResolver doesn't support Arm64 Platform (for now). Washi1337/AsmResolver#640 (comment)
Arm64 Platform can still be benchmarked, as long as the user's project is compiled for AnyCpu (we don't make any assembly modifications to the benchmark wrapper project).

@timcassell timcassell marked this pull request as ready for review June 6, 2025 21:25
@timcassell timcassell mentioned this pull request Jun 8, 2025
Open
timcassell added 21 commits June 9, 2025 15:31
@timcassell
Call benchmark method directly instead of via delegate.
38e8cf8
@timcassell
Removed Consumer from benchmark actions.
c963dc9 
Overhead always returns `void`.
@timcassell
Removed workload return types from code gen.
562eb3c
@timcassell
Apply NoOptimization instead of AggressiveOptimization to loop me…
97d85df 
…thods.

Count down loops instead of count up.
Added IntroSmokeStringBuilder.
Added more return type test cases.
@timcassell
Added assembly weaver.
4b17ba1 
Reverted loop methods back to `AggressiveOptimization`.
Added `NoInlining` to `__Overhead` to match weaved benchmark method.
Updated ExpectedBenchmarkResultsTests.
@timcassell
Remove unused constant.
1f65946
@timcassell
Remove InProcess check from validation.
9b51937 
Update Weaver.
@timcassell
Fix comment.
256e91e
@timcassell
Zero threshold.
d1e431c
@timcassell
Update built package version.
7d42bb0
@timcassell
Fix test.
e58edca
@timcassell
Use AsmResolver instead of Mono.Cecil.
778648d 
Run task after CoreCompile instead of AfterBuild.
@timcassell
Improve pack-weaver task to auto-delete old package.
e64c285 
Updated comments.
@timcassell
Try latest beta version of AsmResolver.DotNet.
ecf85c4
@timcassell
Fix removal of AggressiveInlining.
ed496c9
@timcassell
Make AsmResolver non-transitive.
5e81b8e
@timcassell
Change target back to AfterBuild.
4521a7c
@timcassell
Make all packages non-transitive.
2b813dc
@timcassell
Update packages to latest after rebase.
c431200
@timcassell
Don't write module if it's unnecessary.
aea66e5 
Don't fail silently.
Remove unused property.
Embed the weaver into BenchmarkDotNet.Annotations package instead of another package dependency.
@timcassell
Rebuild after rebase.
c9acbc2
@AndyAyersMS
Copy link
Member

@dotnet/jit-contrib any thoughts on this?

@timcassell
Added comment.
238b6b6 
Fixed formatting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@AndreyAkinshin AndreyAkinshin Awaiting requested review from AndreyAkinshin

Assignees
No one assigned
Labels
Area:CodeGen
Projects
None yet
Milestone
v0.15.x
4 participants
@timcassell @AndyAyersMS @LoopedBard3 @AndreyAkinshin