Refactor providers into separate libraries #1190
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Details: Add a DML DeviceInterface and DML DeviceBuffer handler. Remove #if blocks that are doing memory copies between device/cpu memory and use the DeviceSpan interface.
Remove as many #if USE_CUDA/USE_DML as possible
aciddelgado
reviewed
Jan 21, 2025
aciddelgado
reviewed
Jan 22, 2025
aciddelgado
reviewed
Jan 22, 2025
natke
reviewed
Jan 27, 2025
There are some #if USE_CUDA in our tests, this shouldn't be a problem The rough edges are just expected simple bugs we'll find and easily fix that I can't find in advance. |
edgchen1
reviewed
Jan 29, 2025
aciddelgado
reviewed
Jan 30, 2025
baijumeswani
previously approved these changes
Feb 4, 2025
baijumeswani
approved these changes
Feb 13, 2025
baijumeswani
added a commit
that referenced
this pull request
Jun 27, 2025
Before the device interface was introduced in #1190, the dml objects were tied to the model. The device interface abstraction decoupled the device specific objects and the `OgaModel`. For dml, this meant that the dml objects now lived in a global scope (they were previously owned by the `OgaModel` and hence had the Model scope). These dml objects upon instantiation create background threads that retain hardware resources and prevent the driver threads from terminating. Since these are now in a global scope, the background threads continue living beyond the lifetime of the Model and can cause issues since driver threads may be able to terminate correctly leading to issues in application layers. Another pull-request #1378 made it so that device allocators are cached and tied to a global ort session. As a result, this device allocator is also linked to the dml objects. Making it hard to control the lifetime of the dml objects. This pull request special cases the dml device type so that it destroys all linked globally scoped variables when the model is destroyed and re-creates them when a new model is initialized. This way, the dml threads terminate when the model is destroyed and release driver threads so they can do their own thing.
baijumeswani
added a commit
that referenced
this pull request
Jun 27, 2025
Before the device interface was introduced in #1190, the dml objects were tied to the model. The device interface abstraction decoupled the device specific objects and the `OgaModel`. For dml, this meant that the dml objects now lived in a global scope (they were previously owned by the `OgaModel` and hence had the Model scope). These dml objects upon instantiation create background threads that retain hardware resources and prevent the driver threads from terminating. Since these are now in a global scope, the background threads continue living beyond the lifetime of the Model and can cause issues since driver threads may be able to terminate correctly leading to issues in application layers. Another pull-request #1378 made it so that device allocators are cached and tied to a global ort session. As a result, this device allocator is also linked to the dml objects. Making it hard to control the lifetime of the dml objects. This pull request special cases the dml device type so that it destroys all linked globally scoped variables when the model is destroyed and re-creates them when a new model is initialized. This way, the dml threads terminate when the model is destroyed and release driver threads so they can do their own thing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This removes most of the #if USE_CUDA and #if USE_DML blocks for the model handling code. Device memory management is also handled through the DeviceSpan structure and now all data copying is done in a device independent manner.
It's a huge change, and there will be some rough edges when submitted. Goal is to unblock other people needing the changes and then to make larger improvements in future prs.