Fix itt resource memory leak issue by sgbihu · Pull Request #34096 · openvinotoolkit/openvino

sgbihu · 2026-02-12T13:27:35Z

Details:

The ITT allocate memory from the below call stack.

000001d4`82618d30  00007ffd`12845ff9 vfbasics!AVrfpInitializeCriticalSectionCommon+0x13d
000001d4`82618d38  00007ffc`d277ac4c openvino!__itt_get_collection_state+0x2c [C:\Jenkins\workspace\private-ci\ie\build-windows-vs2022@2\b\repos\openvino\thirdparty\ittapi\ittapi\src\ittnotify\ittnotify_static.c @ 1665]
000001d4`82618d40  00007ffc`d1d68949 openvino!openvino::itt::internal::`dynamic initializer for 'state''+0x9 [src\common\itt\src\itt.cpp @ 22]
000001d4`82618d48  00007ffd`2f8de716 ucrtbase!initterm+0x36
000001d4`82618d50  00007ffc`d2782eea openvino!dllmain_crt_process_attach+0x9a [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 66]
000001d4`82618d58  00007ffc`d2783057 openvino!dllmain_dispatch+0x6f [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\dll_dllmain.cpp @ 276]
000001d4`82618d60  00007ffd`13d20ec4 verifier!AVrfpStandardDllEntryPointRoutine+0xf4
000001d4`82618d68  00007ffd`20dfb704 vrfcore!VfCoreStandardDllEntryPointRoutine+0x184
000001d4`82618d70  00007ffd`12848694 vfbasics!AVrfpStandardDllEntryPointRoutine+0xf4
000001d4`82618d78  00007ffd`322df86e ntdll!LdrpCallInitRoutineInternal+0x22
000001d4`82618d80  00007ffd`3218bcae ntdll!LdrpCallInitRoutine+0x10e
000001d4`82618d88  00007ffd`321897ac ntdll!LdrpInitializeNode+0x19c
000001d4`82618d90  00007ffd`322176ea ntdll!LdrpInitializeGraphRecurse+0x6a
000001d4`82618d98  00007ffd`32217716 ntdll!LdrpInitializeGraphRecurse+0x96

Need to call __itt_release_resources when unload openvino.dll.
The solutions
Here is the solution: we create a class used to store all resource deallocation methods, then create a static object. The release method will register to the static object; this object will be released when the dll unload, all release functions will be called in the destructor. In this way, we didn't need to change any code in DLLMain/unload_library. Just use a MACRO to define the function pointer, like the code below.

static void shutdown_frontend_resources() {
    google::protobuf::ShutdownProtobufLibrary();
}

OV_REGISTER_SHUTDOWN_CALLBACK(shutdown_frontend_resources)

Port #33887 to 2026.0

Tickets:

Change the public to private and try to fix linux configure issue Try to fix CI build issue Try to fix the CI Cmake issue Try another way to fix the CI Try to add a new machnism to release the DLL resource Add lost file Fix CI build errors try to fix MacOS link issue Fix the compile issue Update the configure to control build static/shared library Try to only enable shutdown on dynamic library(May need delete) Refactor some code for itt fix Try to fix the build issue Add install target Try to fix the issue try to fix the build warning Try to fix the sample build cmake error Try to add install target Try to fix the NVplugin cmake issue Try to install object target Try to fix NVplugin build issue Refactor the code to avoid release resource at DLLMain, instead in the static object destruct Fix lost file Fix review inputs Fix review inputs Update year

rkazants · 2026-02-13T05:57:33Z

src/frontends/common/shutdown_protobuf/CMakeLists.txt

    $<BUILD_INTERFACE:$<TARGET_PROPERTY:protobuf::libprotobuf,INTERFACE_INCLUDE_DIRECTORIES>>)
 set_target_properties(${TARGET_NAME} PROPERTIES INTERPROCEDURAL_OPTIMIZATION_RELEASE ${ENABLE_LTO})
 target_compile_features(${TARGET_NAME} PRIVATE $<TARGET_PROPERTY:protobuf::libprotobuf,INTERFACE_COMPILE_FEATURES>)
+target_link_libraries(${TARGET_NAME} PRIVATE openvino::shutdown)


why do we need to link additional library into frontends? Not sure that for fixing memory leaks we need to create the library?

We want to unify the resource deallocation logic. In this way, all new modules with resource deallocation requirements can use this interface. I found this case is similar to the ITT case, so change it.

In general, we need to talk about this deallocation logic. May be we need to have another approach.
We need to discuss it on arch meeting with @praasz.

This is small library allow clean resources and is linked with ITT library (plugins use so no additional linking required).
Is used in FE to use same way to release resources, only small library is used for protobuf release instead some bigger lib (if there will be no dedicated shutdown lib)

@sgbihu
Could explain how changes has been tested

You can find the duplicate steps in the https://jira.devtools.intel.com/browse/CVS-180657, I used the gflags/appverifier to monitor the memory leak issue. I can list the brief steps here:

setup the gflags/appverifier.

execute the onnxruntime_shared_lib_dlopen_test.exe with windbg tool (if the program break with memory leak info, it means program has memory leak issue. It's fine if it can exit)

The old way attempts to call DLLMain/library_unload to release resources when the DLL is unloaded.
This change wants to use a static object's destructor to release the resource. The static object will be released when DLL exit. In this way, it gets those benefits:

The shutdown_protobuf doesn't need to be an object target (I forgot to change this here, but this didn't impact the function, change the target from static to object is another memory leak fix in Fix the onnxfrontend didn't call DLLMain when exit #33641). This change can dismiss this platform dependency issue.

Reuse the OV core's code and make all releases aligned.

This also make the code more clean and reusable.

why do we need to link additional library into frontends? Not sure that for fixing memory leaks we need to create the library?

Linkage to the new library allows to ensure a modular approach. We can put the code directly to the frontend/backends/core as alternative solution but it is a bad practice.
If this shutdown static library is linked to any other shared lib, nothing changes until you implement OV_REGISTER_SHUTDOWN_CALLBACK(...) somewhere inside the shared lib.

So from frontend library perspective nothing is changed if PROTOBUF_REQUIRED=OFF.
If PROTOBUF_REQUIRED=ON (onnx, paddle, tensorflow), only order of sequence when google::protobuf::ShutdownProtobufLibrary(); is called could be changed

rkazants

Before changing frontends, we need careful review for this. It can be a critical changes in frontends.

Please add tests to show when memory leak happens. Thanks!

mlukasze · 2026-02-13T06:08:43Z

Thank you for that advice, we are working on it for some time already.

Copilot

Pull request overview

This PR introduces a centralized shutdown-callback registry to run cleanup code when OpenVINO shared libraries/modules are unloaded, addressing a reported ITT-related memory leak (and reworking protobuf shutdown to use the same mechanism).

Changes:

Added a new openvino::shutdown static library that provides ov::register_shutdown_callback and a OV_REGISTER_SHUTDOWN_CALLBACK macro.
Replaced platform-specific protobuf shutdown (DllMain / __attribute__((destructor))) with a shutdown callback registration.
Registered an ITT shutdown callback that calls __itt_release_resources() on unload, and wired openvino::shutdown into core/runtime, plugins, and frontends build targets as needed.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/frontends/common/shutdown_protobuf/shutdown_protobuf.cpp	Switch protobuf shutdown to the shutdown-callback mechanism.
src/frontends/common/shutdown_protobuf/CMakeLists.txt	Link protobuf shutdown object target with `openvino::shutdown`.
src/common/shutdown/shutdown.cpp	Implements shutdown callback registry and invokes callbacks on unload.
src/common/shutdown/include/openvino/shutdown.hpp	Declares shutdown API + registration macro.
src/common/shutdown/CMakeLists.txt	Adds/export/installs new `openvino_shutdown` static library.
src/common/itt/src/itt.cpp	Registers ITT resource-release callback on unload.
src/common/itt/include/openvino/itt.hpp	Adds `internal::shutdown()` declaration.
src/common/itt/CMakeLists.txt	Links `openvino::itt` with `openvino::shutdown`.
src/common/CMakeLists.txt	Adds `shutdown` subdirectory.
src/cmake/openvino.cmake	Links `openvino::shutdown` into `openvino` shared runtime build.
cmake/developer_package/plugins/plugins.cmake	Links plugins with `openvino::shutdown` when building shared libs.
cmake/developer_package/frontends/frontends.cmake	Links protobuf-required frontends with `openvino::shutdown`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-13T10:10:26Z

src/common/shutdown/include/openvino/shutdown.hpp

+#define OV_REGISTER_SHUTDOWN_CALLBACK(func) \
+    namespace { \
+        static bool ov_shutdown_register_##func = ov::register_shutdown_callback(func); \


OV_REGISTER_SHUTDOWN_CALLBACK expands to a namespace-scope static bool that is never referenced, which will trigger -Wunused-variable under -Wall (and can become a build break with warnings-as-errors). Also, using ov_shutdown_register_##func makes the macro fail for non-identifier arguments (e.g., &func, qualified names) and can cause redefinition errors if the macro is invoked twice with the same callback name in one TU. Consider generating a unique variable name (e.g., with __COUNTER__/__LINE__), and marking it [[maybe_unused]] (and/or const) so it is warning-free.

Suggested change

#define OV_REGISTER_SHUTDOWN_CALLBACK(func) \

namespace { \

static bool ov_shutdown_register_##func = ov::register_shutdown_callback(func); \

#define OV_CONCAT_IMPL(a, b) a##b

#define OV_CONCAT(a, b) OV_CONCAT_IMPL(a, b)

#define OV_REGISTER_SHUTDOWN_CALLBACK(func) \

namespace { \

[[maybe_unused]] const bool \

OV_CONCAT(ov_shutdown_register_, __COUNTER__) = \

ov::register_shutdown_callback(func); \

we referenced src\frontends\onnx\frontend\src\core\operator_set.hpp, #define ONNX_OP(name, range, ...) static bool onnx_op_reg = ONNX_OP_M(name, range, __VA_ARGS__)

Copilot · 2026-02-13T10:10:26Z

src/common/shutdown/shutdown.cpp

+    bool register_callback(const std::function<void()>& func) {
+        if (!func) {
+            return false;
+        }
+        m_callbacks.emplace_back(func);
+        return true;


ShutdownRegistry::register_callback mutates m_callbacks without any synchronization. Since register_shutdown_callback is a public function (and may be used from multiple threads), this introduces a data race and potential heap corruption. Please add thread-safety (e.g., guard m_callbacks with a mutex, and consider taking the std::function by value and moving it into the vector).

This was called at dll load stage, didn't have thread issue.

Copilot · 2026-02-13T10:10:26Z

src/common/shutdown/shutdown.cpp

+    ~ShutdownRegistry() {
+        for (auto& func : m_callbacks) {
+            func();
+        }
+        m_callbacks.clear();
+    }


ShutdownRegistry's destructor calls registered callbacks directly. If any callback throws, the process will std::terminate() during DLL unload / static deinitialization. Also, iterating m_callbacks directly can misbehave if callbacks register additional callbacks (vector reallocation/invalidated references). A safer pattern is to move/swap the callback list into a local container under a lock, then invoke callbacks (often in reverse registration order) inside a noexcept destructor with exceptions swallowed/logged.

One DLL only has one instance, so it doesn't need the mutex.

sgbihu requested review from a team as code owners February 12, 2026 13:27

sgbihu requested review from mlukasze and removed request for a team February 12, 2026 13:27

github-actions bot added category: inference OpenVINO Runtime library - Inference category: Core OpenVINO Core (aka ngraph) category: build OpenVINO cmake script / infra no-match-files labels Feb 12, 2026

praasz added this to the 2026.0 milestone Feb 12, 2026

praasz self-assigned this Feb 12, 2026

rkazants reviewed Feb 13, 2026

View reviewed changes

rkazants requested changes Feb 13, 2026

View reviewed changes

rkazants assigned mvafin and rkazants Feb 13, 2026

rkazants requested a review from mvafin February 13, 2026 05:58

praasz approved these changes Feb 13, 2026

View reviewed changes

mvafin requested a review from Copilot February 13, 2026 10:02

Copilot started reviewing on behalf of mvafin February 13, 2026 10:03 View session

Copilot AI reviewed Feb 13, 2026

View reviewed changes

mvafin approved these changes Feb 13, 2026

View reviewed changes

olpipi approved these changes Feb 13, 2026

View reviewed changes

mlukasze mentioned this pull request Feb 13, 2026

Fix itt resource memory leak issue #33887

Merged

sgbihu closed this Mar 10, 2026

Conversation

sgbihu commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

Uh oh!

rkazants Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

sgbihu Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

rkazants Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

praasz Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgbihu Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sgbihu Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olpipi Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rkazants left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlukasze commented Feb 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

sgbihu Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

sgbihu Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

sgbihu Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sgbihu commented Feb 12, 2026 •

edited

Loading

praasz Feb 13, 2026 •

edited

Loading

sgbihu Feb 13, 2026 •

edited

Loading

sgbihu Feb 13, 2026 •

edited

Loading

olpipi Feb 13, 2026 •

edited

Loading

rkazants left a comment •

edited

Loading