Memory Leak When Using ONNXRuntime With OpenVino EP

[narolski]

Description
Using the same model as in #102, the Triton Inference Server has a memory leak, as observed by docker stats, after adding:

  execution_accelerators {
    cpu_execution_accelerator : [ {
      name : "openvino"
    } ]
  }

to model config.

Without the openvino EP usage, there is no memory leak

Triton Information
What version of Triton are you using?
openvino==2022.1.0 with triton-onnxbackend==22.06 and onnxruntime==1.11.1.

Are you using the Triton container or did you build it yourself?

Custom container build.

To Reproduce

See #102 for model.

Expected behavior
A clear and concise description of what you expected to happen.

Provision of model configuration flags (like in #102) that will customize the memory handling of OpenVino EP.

[narolski]

After further investigation of this issue I've determined that there was a memory reusage solution implemented for OpenVino EP: openvinotoolkit/openvino#11667.

I will try to build the OpenVino master branch with the changes from the above PR to see if it resolves this issue.

Update:

• Building the OpenVino with changes from Dynamic shape mem reuse solution openvinotoolkit/openvino#11667 did not solve the issue for my model.
• I've also reported the bug to the OpenVino team [Bug] Significant Memory Leak When Using OpenVino EP With ONNX Runtime For CRAFT openvinotoolkit/openvino#12307

[riverlijunjie]

There is another PR to solve rnn cache increasing issue, could help to try it?
openvinotoolkit/openvino#12053

[luvwinnie]

I have a same problem on CRAFT model, even I converted CRAFT to openvino IR format, does this to be fix?

[riverlijunjie]

You can try the openvino 2022.2 or latest master branch.

[zhuangxiaopi]

@narolski Has this problem been solved?

[ogencoglu]

Any updates?

[mezotaken]

I think that the memory reuse PR was meant to merely improve the consumption, it's not openvino itself causing the problem, it's on the onnxruntime side. See this: microsoft/onnxruntime#7271
I have encountered similar memory leak, and on a much later version of onnxruntime-openvino EP, which already has the memory reuse PR from 3 years ago