Skip to content

[NPU] New vcl_allocator_3 to support multiple allocations and robust memory ownership transfer#35623

Open
ShaojieZhuIntel wants to merge 3 commits intoopenvinotoolkit:masterfrom
ShaojieZhuIntel:change_allocator
Open

[NPU] New vcl_allocator_3 to support multiple allocations and robust memory ownership transfer#35623
ShaojieZhuIntel wants to merge 3 commits intoopenvinotoolkit:masterfrom
ShaojieZhuIntel:change_allocator

Conversation

@ShaojieZhuIntel
Copy link
Copy Markdown
Contributor

@ShaojieZhuIntel ShaojieZhuIntel commented Apr 30, 2026

Details:

  • Added vcl_allocator_3 to use a std::vector<std::pair<uint8_t*, size_t>> m_info registry to track all allocated memory blocks independently.
  • Updated compileWsOneShot to iterate over all allocated blobs in m_info, wrap them into ov::Tensor via make_tensor_from_aligned_addr, and cleanly transfer ownership using m_info.clear().
  • Fixed ownership transfer in compile: Instead of arbitrarily using the first blob or wiping out the entire registry, it now searches (std::find_if) for the specific blob address returned by the VCL compiler. It accurately retrieves its aligned size, erases only that blob from m_info to transfer its ownership to the ov::Tensor deleter, and leaves any temporary allocations safely inside m_info.

Tickets:

  • ticket-id

AI Assistance:

  • AI assistance used: no / yes
  • If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

@github-actions github-actions Bot added the category: NPU OpenVINO NPU plugin label Apr 30, 2026
@ShaojieZhuIntel ShaojieZhuIntel marked this pull request as ready for review April 30, 2026 10:06
@ShaojieZhuIntel ShaojieZhuIntel requested review from a team as code owners April 30, 2026 10:06
Copy link
Copy Markdown
Collaborator

@rkazants rkazants left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we use 'unordered_map' for storing these pairs? The search will be more efficient rather than std::vector.

Also, is it possible to provide a test case with multiple allocations?

Signed-off-by: Zhu, Shaojie <shaojie.zhu@intel.com>
Signed-off-by: Zhu, Shaojie <shaojie.zhu@intel.com>
Signed-off-by: Zhu, Shaojie <shaojie.zhu@intel.com>
@ShaojieZhuIntel
Copy link
Copy Markdown
Contributor Author

Can't we use 'unordered_map' for storing these pairs? The search will be more efficient rather than std::vector.

Also, is it possible to provide a test case with multiple allocations?

Thanks for the review and the suggestions!

Regarding unordered_map:
I considered using std::unordered_map to optimize lookup. However, we must preserve the insertion order of allocated blobs, because in compileWsOneShot(), initMainTensors strictly relies on the order in which blobs were created to map nicely onto the model's inputs/outputs.
To use unordered_map but still maintain order, we would need to maintain an auxiliary vector alongside it, adding unnecessary complexity. Besides, the actual number of allocations performed by VCL here is extremely small (single-digits typically), therefore finding/erasing from a contiguous std::vector is extremely fast and cache-friendly. So I keep the std::vector implementation here.

Regarding the test case:
Good point! I will add a unit test mocking multiple allocations to ensure the blobs are safely transferred to ov::Tensor and the allocator's memory tracking and lifecycle are behaving correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants