You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/execution-providers/plugin-ep-libraries.md
+32-22Lines changed: 32 additions & 22 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -328,8 +328,8 @@ ONNX Runtime discovered during initialization.
328
328
The factory returns `OrtEpDevice` instances from `OrtEpFactory::GetSupportedDevices()`.
329
329
Each `OrtEpDevice` instance pairs a factory with a hardware device that the factory supports.
330
330
For example, if a single factory instance supports both CPU and NPU, then the call to `OrtEpFactory::GetSupportedDevices()` returns two `OrtEpDevice` instances:
331
-
- ep_device_0: (factory_0, CPU)
332
-
- ep_device_1: (factory_0, NPU)
331
+
- ep_device_0: (factory_0, CPU)
332
+
- ep_device_1: (factory_0, NPU)
333
333
334
334
<br/>
335
335
<p align="center"><img width="100%" src="../../images/plugin_ep_sd_lib_reg.png" alt="Sequence diagram showing registration and unregistration of a plugin EP library"/></p>
@@ -519,7 +519,7 @@ Opaque type that contains a compiled/fused node's name and host memory allocatio
519
519
OrtNodeComputeInfo
520
520
</td>
521
521
<td>
522
-
Struct that contains the computation function for a compiled `OrtGraph` instance. Initialized by an `OrtEp` instance.
522
+
Struct that contains the computation function for a compiled <code>OrtGraph</code> instance. Initialized by an <code>OrtEp</code> instance.
523
523
</td>
524
524
</tr>
525
525
@@ -528,7 +528,7 @@ Struct that contains the computation function for a compiled `OrtGraph` instance
528
528
OrtEpGraphSupportInfo
529
529
</td>
530
530
<td>
531
-
Opaque type that contains information on the nodes supported by an EP. An instance of `OrtEpGraphSupportInfo` is passed to `OrtEp::GetCapability()` and the EP populates the `OrtEpGraphSupportInfo` instance with information on the nodes that it supports.
531
+
Opaque type that contains information on the nodes supported by an EP. An instance of <code>OrtEpGraphSupportInfo</code> is passed to <code>OrtEp::GetCapability()</code> and the EP populates the <code>OrtEpGraphSupportInfo</code> instance with information on the nodes that it supports.
532
532
</td>
533
533
</tr>
534
534
@@ -546,7 +546,7 @@ Enumerates the operator data layouts that could be preferred by an EP. By defaul
546
546
OrtMemoryDevice
547
547
</td>
548
548
<td>
549
-
Opaque type that represents a combination of a physical device and memory type. A memory allocation and allocator are associated with a specific `OrtMemoryDevice`, and this information is used to determine when data transfer is required.
549
+
Opaque type that represents a combination of a physical device and memory type. A memory allocation and allocator are associated with a specific <code>OrtMemoryDevice</code>, and this information is used to determine when data transfer is required.
550
550
</td>
551
551
</tr>
552
552
@@ -582,11 +582,11 @@ Struct of functions that an EP implements if it needs to support Streams.
582
582
<a href="#ort-ep-factory">OrtEpFactory</a>
583
583
</td>
584
584
<td>
585
-
A plugin EP library provides ORT with one or more instances of `OrtEpFactory`. An `OrtEpFactory` implements functions that are used by ORT to query device support, create allocators, create data transfer objects, and create instances of an EP (i.e., an `OrtEp` instance).<br/>
585
+
A plugin EP library provides ORT with one or more instances of <code>OrtEpFactory</code>. An <code>OrtEpFactory</code> implements functions that are used by ORT to query device support, create allocators, create data transfer objects, and create instances of an EP (i.e., an <code>OrtEp</code> instance).<br/>
586
586
587
-
An `OrtEpFactory` may support more than one hardware device (`OrtHardwareDevice`). If more than one hardware device is supported by the factory, an EP instance created by the factory is expected to internally partition any graph nodes assigned to the EP among its supported hardware devices.<br/>
587
+
An <code>OrtEpFactory</code> may support more than one hardware device (<code>OrtHardwareDevice</code>). If more than one hardware device is supported by the factory, an EP instance created by the factory is expected to internally partition any graph nodes assigned to the EP among its supported hardware devices.<br/>
588
588
589
-
Alternatively, if an EP library author needs ONNX Runtime to partition the graph nodes among different hardware devices supported by the EP library, then the EP library must provide multiple `OrtEpFactory` instances. Each `OrtEpFactory` instance must support one hardware device and must create an EP instance with a unique name (e.g., MyEP_CPU, MyEP_GPU, MyEP_NPU).
589
+
Alternatively, if an EP library author needs ONNX Runtime to partition the graph nodes among different hardware devices supported by the EP library, then the EP library must provide multiple <code>OrtEpFactory</code> instances. Each <code>OrtEpFactory</code> instance must support one hardware device and must create an EP instance with a unique name (e.g., MyEP_CPU, MyEP_GPU, MyEP_NPU).
590
590
</td>
591
591
</tr>
592
592
@@ -595,7 +595,7 @@ Alternatively, if an EP library author needs ONNX Runtime to partition the graph
595
595
<a href="#ort-ep">OrtEp</a>
596
596
</td>
597
597
<td>
598
-
An instance of an Ep that can execute model nodes on one or more hardware devices (`OrtHardwareDevice`). An `OrtEp` implements functions that are used by ORT to query graph node support, compile supported nodes, query preferred data layout, set run options, etc. An `OrtEpFactory` creates an `OrtEp` instance via the `OrtEpFactory::CreateEp()` function.
598
+
An instance of an Ep that can execute model nodes on one or more hardware devices (<code>OrtHardwareDevice</code>). An <code>OrtEp</code> implements functions that are used by ORT to query graph node support, compile supported nodes, query preferred data layout, set run options, etc. An <code>OrtEpFactory</code> creates an <code>OrtEp</code> instance via the <code>OrtEpFactory::CreateEp()</code> function.
599
599
</td>
600
600
</tr>
601
601
@@ -604,7 +604,7 @@ An instance of an Ep that can execute model nodes on one or more hardware device
604
604
OrtRunOptions
605
605
</td>
606
606
<td>
607
-
Opaque object containing options passed to the `OrtApi::Run()` function, which runs a model.
607
+
Opaque object containing options passed to the <code>OrtApi::Run()</code> function, which runs a model.
608
608
</td>
609
609
</tr>
610
610
@@ -613,7 +613,7 @@ Opaque object containing options passed to the `OrtApi::Run()` function, which r
613
613
OrtGraph
614
614
</td>
615
615
<td>
616
-
Opaque type that represents a graph. Provided to `OrtEp` instances in calls to `OrtEp::GetCapability()` and `OrtEp::Compile()`.
616
+
Opaque type that represents a graph. Provided to <code>OrtEp</code> instances in calls to <code>OrtEp::GetCapability()</code> and <code>OrtEp::Compile()</code>.
617
617
</td>
618
618
</tr>
619
619
@@ -622,11 +622,11 @@ Opaque type that represents a graph. Provided to `OrtEp` instances in calls to `
622
622
OrtValueInfo
623
623
</td>
624
624
<td>
625
-
Opaque type that contains information for a value in a graph. A graph value can be a graph input, graph output, graph initializer, node input, or node output. An `OrtValueInfo` instance has the following information.<br/>
625
+
Opaque type that contains information for a value in a graph. A graph value can be a graph input, graph output, graph initializer, node input, or node output. An <code>OrtValueInfo</code> instance has the following information.<br/>
626
626
<ul>
627
-
<li>Type and shape (e.g., `OrtTypeInfo`)</li>
628
-
<li>`OrtNode` consumers</li>
629
-
<li>`OrtNode` producer</li>
627
+
<li>Type and shape (e.g., <code>OrtTypeInfo</code>)</li>
628
+
<li><code>OrtNode</code> consumers</li>
629
+
<li><code>OrtNode</code> producer</li>
630
630
<li>Information that classifies the value as a graph input, graph output, initializer, etc.</li>
631
631
</ul>
632
632
</td>
@@ -637,7 +637,7 @@ Opaque type that contains information for a value in a graph. A graph value can
637
637
OrtExternalInitializerInfo
638
638
</td>
639
639
<td>
640
-
Opaque type that contains information for an initializer stored in an external file. An `OrtExternalInitializerInfo` instance contains the file path, file offset, and byte size for the initializer. Can be obtained from an `OrtValueInfo` via the function `ValueInfo_GetExternalInitializerInfo()`.
640
+
Opaque type that contains information for an initializer stored in an external file. An <code>OrtExternalInitializerInfo</code> instance contains the file path, file offset, and byte size for the initializer. Can be obtained from an <code>OrtValueInfo</code> via the function <code>ValueInfo_GetExternalInitializerInfo()</code>.
Unregister an EP library with ORT. Caller <b>MUST</b> ensure there are no <code>OrtSession</code> instances using the EPs created by the library before calling this function.
0 commit comments