Skip to content

AutoSelect EP Infrastructure Part 2. #24455

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

skottmckay
Copy link
Contributor

@skottmckay skottmckay commented Apr 17, 2025

Description

NOTE: Moving out of Draft state so the CIs run.

Diffs are against #24430 and will be a follow-up PR once that is checked in.

Proposed changes. Can split out the 3 main areas if it's easier/quicker to review.
Assumption: We want all these changes. Please feel free to disagree with some/all and to suggest alternatives.

Update SessionOptionsAppendExecutionProvider_V2 to make OrtEpDevices as input to support all known scenarios.

  • supports needing to pick an EP that supports multiple devices to run on a specific device
    • e.g. WebGPU could run on discrete or integrated GPU and we want to register it for one
  • supports needing to pick an EP to run on multiple devices it supports
    • e.g. OpenVINO could be some combination of CPU/GPU/NPU. not necessarily all. not necessarily one.
  • update EpFactoryInternal to support multiple devices

Re-work setup for provider bridge EPs being registered.

  • removes all hardcoding of EP name to identify provider bridge EP
  • add CreateEpFactories/ReleaseEpFactory to provider bridge EP
  • add CreateIExecutionProvider to Provider
  • add implementation of the pieces that are now on the provider bridge side to the CUDA EP.
  • wire up EpLibraryProviderBridge to use new setup

Add enhance device discovery

  • try and find/add more meaningful data so there's always a vendor name and description
  • add some other arbitrary metadata. TBD what we really want/need
  • I am really just fumbling my way around here to see what's possible. Happy to have all of this replaced with a better approach.

Minor cleanups

  • use uint32_t for vendor_id and device_id as that's the real type
  • rename bus_id to device_id as that matches the fields in the hardware id we're reading from
    • the new device discovery can lookup BusNumber and add to the metadata which makes bus_id confusing
    • open to suggestions on naming. we don't have a field that represents 'unique id' (not sure we need one). 'device_id' could potentially be confused as being a unique id for the device.

Motivation and Context

…as input to support all known scenarios.

  - also update EpFactoryInternal to support multiple devices

Re-work setup for provider bridge EP
  - add CreateEpFactories/ReleaseEpFactory to provider bridge EP
  - add CreateIExecutionProvider to Provider
  - wire up EpLibraryProviderBridge to use new setup
  - removes all hardcoding of EP name

Add enhance device discovery
  - try and find/add more meaningful data so there's always a vendor name and description
  - add some other arbitrary metadata. TBD what we really want/need
@skottmckay skottmckay marked this pull request as ready for review April 17, 2025 23:00
@skottmckay
Copy link
Contributor Author

Merged into #24430

@skottmckay skottmckay closed this Apr 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants