Stop gracefully when benchmark fails with an exception that can not be caught by a catch block

[adamsitnik]

The original issue was reported in dotnet/performance#1701:

// Benchmark Process Environment Information:
// Runtime=.NET 6.0.0 (6.0.21.11801), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-AQTFSE(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

OverheadJitting  1: 1 op, 537000.00 ns, 537.0000 us/op
WorkloadJitting  1: 1 op, 6767100.00 ns, 6.7671 ms/op

OverheadJitting  2: 16 op, 731100.00 ns, 45.6938 us/op
WorkloadJitting  2: 16 op, 2527400.00 ns, 157.9625 us/op

WorkloadPilot    1: 16 op, 7800.00 ns, 487.5000 ns/op
// Benchmark Process 3336 has exited with code -1073741819
Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)

The benchmark process exited with -1073741819 which translates to 0xc0000005 which is Access Violation.

BenchmarkDotNet should:

• detect failures for exceptions that can not be caught (most probably this logic has to be modified)
• stop gracefully in such cases

Whoever is willing to work on that should most probably start by creating a benchmark that throws an exception that can not be caught by a catch block (AccessViolation would be the best), then create a test for it similar to this one and fix BenchmarkRunnerClean.Execute

[danmoseley]

I wonder whether Watson grabbed a dump.

[kevinsalimi]

I wanna take this if still available.

[adamsitnik]

@kevinsalimi I've assigned you. Please let me know if you need some help.

[kevinsalimi]

Hello @adamsitnik. Thank you for the assignment.
I spent ample time on the issue and learned a lot during the code exploration yet I have some questions.

The AccessViolation exception is this, am I right?

After creating a benchmark, I couldn't simulate any exceptions that cannot be caught by a catch block. I tried to threw variant exceptions and scenarios but I got nothing. In addition, I wrote code below in the benchmark:

unsafe
{
    IntPtr ptr = new IntPtr(100);
    int value = Marshal.ReadInt32(ptr);
}

I caught the AccessViolationException and the Benchmark stop gracefully like below.

// Benchmark Process Environment Information:
// Runtime=.NET Framework 4.8 (4.8.4300.0), X64 RyuJIT
// GC=Concurrent Workstation
// Job: DefaultJob

OverheadJitting  1: 1 op, 510500.00 ns, 510.5000 us/op

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.InteropServices.Marshal.ReadInt32(IntPtr ptr, Int32 ofs)
   at BenchmarkDotNet.Samples.IntroArguments.Benchmark() in C:\Users\Lenovo\Documents\OpenSourceProjects\BenchmarkDotNet\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\IntroArguments.cs:line 20
   at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount)
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args)
// AfterAll
// Benchmark Process 8632 has exited with code -1
Parse error in the following line:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
.
.
.
// ***** BenchmarkRunner: Finish  *****
.
.
.

Any ideas would be appreciated.

[kevinsalimi]

@adamsitnik

[timcassell]

I'm not sure how to generate an uncatchable AccessViolationException, but you could throw an uncatchable StackOverflowException:

void Kill()
{
    Kill();
}

[twallace27603]

You can also generate an uncatchable StackOverflowException by throwing it, although I love the elegant recursion. I found a great summary of "uncatchable errors on StackOverflow (the irony): https://stackoverflow.com/questions/7392783/list-of-exceptions-that-cant-be-caught-in-net. Basically there are conditions under which AccessViolationExceptions are sort of uncatchable, but it seems that the StackOverflowException is an easier candidate for testing the behavior.

[timcassell]

I believe if you manually throw a StackOverflowException it can be caught. It will only be un-catchable when the runtime throws it. At least that's what I observed when testing that in Unity a while ago. I haven't tested that behavior on the latest .Net runtime.

[twallace27603]

@timcassell you are 100% correct. I did something embarrassingly stupid for my first comment on this project. Your approach is the best test.