Skip to content

Update allocation and execution plan handling for CPU EP MemcpyToHost/MemcpyFromHost#27272

Open
chilo-ms wants to merge 7 commits intomainfrom
chi/update_exe_plan_for_default_plugin_ep_memcpy
Open

Update allocation and execution plan handling for CPU EP MemcpyToHost/MemcpyFromHost#27272
chilo-ms wants to merge 7 commits intomainfrom
chi/update_exe_plan_for_default_plugin_ep_memcpy

Conversation

@chilo-ms
Copy link
Contributor

@chilo-ms chilo-ms commented Feb 6, 2026

Description

The current CPU EP provides the MemcpyToHost/MemcpyFromHost kernel implementations and registry for plugin EPs, however, the allocation and execution plan does not account for the case that the producer/consumer nodes of MemcpyToHost/MemcpyFromHost nodes (on cpu) are on other non-cpu devices.

MemcpyToHost and MemcpyFromHost nodes provided by CPU EP require special handling in planning:

  1. Determining output device for MemcpyFromHost:
    MemcpyFromHost node provided by the CPU EP requires special handling.
    As per MemcpyFromHost kernel registration uses default memory type for output which means it uses CPU memory for output as it's run on CPU, but it actually may produce output on the device specific to its consumer node's EP.
    So, we need to check the consumer node's EP and set the output device accordingly.

  2. Make sure MemcpyToHost use WaitNotificationOnHost:
    There are typically two types of wait functions defined in the Notification class for plugin EPs or other provider-bridge EPs (e.g., CUDA EP and TRT EP):
    (1) WaitNotificationOnDevice and (2) WaitNotificationOnHost

    Note: MemcpyToHost nodes provided by the CPU EP require special handling.
    If a MemcpyToHost node (running on the host) consumes a tensor produced by a device node, MemcpyToHost must use WaitNotificationOnHost, because the CPU device does not have a stream, which is required by WaitNotificationOnDevice.

Motivation and Context

#26088

@chilo-ms chilo-ms marked this pull request as ready for review February 9, 2026 22:23
@chilo-ms chilo-ms requested a review from yuslepukhin February 13, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant