access violation when appending OpenVINO execution provider

[deischi]

Describe the issue

onnxruntime crashes in openvino_npu_plugin.dll when addingthe OpenVINO execution provider for CPU:

		std::unordered_map<std::string, std::string> options;
		options["device_type"] = "CPU";
		options["precision"] = "FP32";
		ortSessionOptions.AppendExecutionProvider("OpenVINO", options);

Callstack:

 	00007ffd287f29c0()	Unknown	No symbols loaded.
 	openvino_intel_npu_plugin.dll!00007ffd28e2ac5d()	Unknown	No symbols loaded.
 	openvino_intel_npu_plugin.dll!00007ffd28df8cfc()	Unknown	No symbols loaded.
 	openvino_intel_npu_plugin.dll!00007ffd28ccd9b8()	Unknown	No symbols loaded.
 	openvino_intel_npu_plugin.dll!00007ffd28cb5b42()	Unknown	No symbols loaded.
 	openvino_intel_npu_plugin.dll!00007ffd28ccc7a0()	Unknown	No symbols loaded.
 	openvino.dll!00007ffd377e1d18()	Unknown	No symbols loaded.
 	openvino.dll!00007ffd377e31f6()	Unknown	No symbols loaded.
 	openvino.dll!00007ffd377df3b3()	Unknown	No symbols loaded.
 	openvino.dll!00007ffd377c6511()	Unknown	No symbols loaded.
 	onnxruntime_providers_openvino.dll!onnxruntime::OVDevices::get_ov_devices() Line 22	C++	Symbols loaded.
 	onnxruntime_providers_openvino.dll!onnxruntime::OpenVINO_Provider::CreateExecutionProviderFactory(const void * void_params) Line 141	C++	Symbols loaded.
 	onnxruntime.dll!onnxruntime::OpenVINOProviderFactoryCreator::Create(const std::unordered_map<std::string,std::string,std::hash<std::string>,std::equal_to<std::string>,std::allocator<std::pair<std::string const ,std::string>>> * provider_options_map, const onnxruntime::SessionOptions * session_options) Line 1830	C++	Symbols loaded.
 	onnxruntime.dll!OrtApis::SessionOptionsAppendExecutionProvider(OrtSessionOptions * options, const char * provider_name, const char * const * provider_options_keys, const char * const * provider_options_values, unsigned __int64 num_keys) Line 111	C++	Symbols loaded.

To reproduce

Immediately crashed when adding the execution provider (see above).

onnxruntime build command:
build.bat --build --update --parallel --compile_no_warning_as_error --config RelWithDebInfo --use_openvino CPU --use_dml --build_shared_lib --skip_tests --cmake_extra_defines CMAKE_INSTALL_PREFIX=c:\dev\ort_1.20.1-install

As a workaround, I tried to remove the openvino_intel_npu_plugin.dll.
appending the execution provider is working, but creating the session later fails with this exception:

Exception during initialization: Exception from src\inference\src\cpp\core.cpp:265:
Exception from src\inference\src\dev\core_impl.cpp:567:
Device with "NPU" name is not registered in the OpenVINO Runtime

Also onnxruntime_perf_test.exe failes for OpenVINO - just without any error message.

C:\dev\onnxruntime-1.20.1\build\Windows\RelWithDebInfo\RelWithDebInfo>\dev\w_openvino_toolkit_windows_2024.6.0.17404.4c0f47d2335_x86_64\setupvars.batPython 3.12.8
[setupvars.bat] OpenVINO environment initialized

fine for CPU:

C:\dev\onnxruntime-1.20.1\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -m times -r 1 -I testdata\test_resize.onnx
Session creation time cost: 0.008206 s
First inference time cost: 0 ms
Total inference time cost: 5.77e-05 s
...

fine for DML:

C:\dev\onnxruntime-1.20.1\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e dml -m times -r 1 -I testdata\test_resize.onnx
Session creation time cost: 0.33271 s
First inference time cost: 47 ms
Total inference time cost: 0.0004038 s
Total inference requests: 1
...

no output for openvino:

C:\dev\onnxruntime-1.20.1\build\Windows\RelWithDebInfo\RelWithDebInfo>onnxruntime_perf_test.exe -e openvino -m times -r 1 -I testdata\test_resize.onnx

There is a similar issue openvinotoolkit/openvino#26567, but this should have been fixed with OpenVINO 2024.5

Urgency

No response

Platform

Windows

OS Version

Windows 10, 22H2

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.20.1

ONNX Runtime API

C++

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

openvino 2024.5

[jywu-msft]

@javier-intel @sfatimar @preetha-intel
can you advise?

[javier-intel]

@deischi I tried reproducing using tips from this repo and wasn't able to. Does your system have an NPU? Mine does and I wondering if that's the reason I see it working.

The code that references OVDevices::get_ov_devices() got refactored recently and should be landing on the ORT repo soon (~days). If it's ok I can ping you when it does and see if the issue is resolved.

[deischi]

My system does not have an NPU, also we are still on Windows 10. I only would like to use only the CPU.

Problems with the NPU plugin are not new for me. With older versions (probably version 2024.2 - I did not try each and every version) it is enough to remove all the "non-CPU" plugin DLLs. But with the latest version of openVino that workaround is no longer available. I get the exception (Device with "NPU" name is not registered in the OpenVINO Runtime) when creating the session.

@javier-intel, please let me know when the new code is in the repository.

[javier-intel]

Sorry to hear about the issue. We're getting a system without an NPU to try and reproduce. In the meantime could you try this commit? intel@506ee08

[javier-intel]

@deischi good news, a colleague of mine was able to reproduce the issue and find the fix. If you grab the ORT main tips (what I did) the issue doesn't happen. The commit in my previous message should also work but can you please confirm that your issue is gone with main?

If you happen to need the individual commit that fixed the issue it's here.

[deischi]

I have tried both fixes independently (intel@506ee08 and 0c4f811).

Both solutions are working fine for me.

When I include "openvino_intel_npu_plugin.dll" I still get the access violation, but if I only include the openvino_intel_cpu_plugin.dll everything is now working fine for me.

[javier-intel]

Hi @deischi I'm just circling back to let you know that commit 0c4f811 landed on main today (a6ea57b). If that addresses the issue please close the ticket.