System.ExecutionEngineException creating Microsoft.ML.OnnxRuntime.SessionOptions

[DennyOne]

Describe the issue

This issue is reported by https://developercommunity.visualstudio.com/user/25004 and moved from https://developercommunity.visualstudio.com/t/SystemExecutionEngineException-creating/10794175 to onnx/onnx#6555

@justinchuby closed onnx/onnx#6555 and suggested creating the issue here.

---

Hi devs!

I have a C#.NET project that uses the Microsoft.ML.OnnxRuntime and Microsoft.ML.OnnxRuntime.Managed packages.

System.ExecutionEngineException error began while creating Microsoft.ML.OnnxRuntime.SessionOptions when the Microsoft.ML.OnnxRuntime and Microsoft.ML.OnnxRuntime.Managed Version=1.19.2 packages were updated to Version=1.20.0.

Do the 1.20.0 (or higher) packages include all required dependencies? Do I need to add any other NuGet packages? Previous version packages (1.19.2 or lower) work without issue.

The project uses the following ONNX packages:
Microsoft.ML.OnnxRuntime Version=1.19.2
Microsoft.ML.OnnxRuntime.Managed Version=1.19.2
TorchSharp Version=0.105.0
TorchSharp-cpu Version=0.105.0

Much Thanks!
-Denny

To reproduce

Unfortunately, I do not have a small sample that can reproduce the issue. If it might be helpful, a Visual Studio debug video was posted in the original issue (https://developercommunity.visualstudio.com/t/SystemExecutionEngineException-creating/10794175)

Urgency

Our project cannot use updated Microsoft.ML.OnnxRuntime versions higher than 1.19.2

Platform

Windows

OS Version

10 & 11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.20.0

ONNX Runtime API

C#

Architecture

X64

Execution Provider

Other / Unknown

Execution Provider Library Version

No response

[skottmckay]

Is there any log output?

If not, ensure that the OrtEnv instance has been created prior to doing anything with the session options. Simplest is to call OrtEnv.Instance() and ignore the return value.

[DennyOne]

@skottmckay Here's what happens when I try to call OrtEnv.Instance()

[DennyOne]

It looks like the same exception:

[yuslepukhin]

It is likely to be a dependency problem.
Also, you want to dispose of the objects as soon as they are no longer needed. Do not rely on the finalizers, they almost never run.

[DennyOne]

Thanks for the finalizer suggestion; I can fix that by implementing IDisposable. Are there missing dependencies in the released Microsoft.ML.OnnxRuntime(and .Managed) Version=1.20.0 packages? The project has worked for quite some time using 1.19.2 and previous versions, and continues to work when I roll back to 1.19.2 package version.

[yuslepukhin]

We are not aware of any missing dependencies.
With the debugger at hand, you are in a good position to find out what is going on.
Using dumpbin /DEPENDENTS and where in your environment can enable you quickly to pinpoint what is missing what needs to be either added to PATH or deployed in in the location where the system can find them.

[DennyOne]

Thanks. It looks like dependencies are the same for V1.19.2 and V1.20.1:

1.19.2

1.20.1

onnxruntime_providers_shared.dll dependencies look the same, too:

Diff

Is there a way to debug into the OnnxRuntime code? How can I load a .pdb for the OnnxRuntime into the debugger?

[yuslepukhin]

Typically, we publish PDBs for onnxruntime on MS symbol servers. Shared dll is normally not needed when CPU only is used.

_NT_SYMBOL_PATH=srv*c:\excache*https://msdl.microsoft.com/download/symbols

The alternative is to build on your own.

I do not think you need to debug to find out if a particular DLL can be loaded. Unless EPs other than CPU are involved, all dependents are build time so should be resolved when onnxruntime.dll is loaded.

You do need Managed dll though for C#.

[DennyOne]

I tried adding the symbol path "https://msdl.microsoft.com/download/symbols" to Visual Studio's Debugging -> Symbols options in addition to checking 'Microsoft Symbol Servers', but it still does not offer to debug into onnxruntime code.

FYI: I tried to clone the repo....I got this error a couple times - could be a problem with the GitHub server?

I downloaded the zip - Can I use build.bat to build and debug?

[yuslepukhin]

Gh had issues recently. I suggest you simply open a command line where you have trouble and run where on all the dependents. This is usually a simple problem to diagnose.

[DennyOne]

Reviewing details provided above leads me to suspect it is not a dependency problem, since all versions 1.19.2 - 1.20.1 have identical dependencies, and all are installed (in the same folder). The issue is SessionOptions cannot be created in released versions > 1.19.2.

Thanks for the suggestion, though!

[yuslepukhin]

SessionOptions is a wrapper around native object. Anything you do will cause a native library loaded first.