Reduce compute io overhead by TedThemistokleous · Pull Request #107 · onnxruntime/onnxruntime-ep-amdgpu

TedThemistokleous · 2026-09-02T19:37:40Z

Update MIGraphX EP to further refine hip graph, direct bind modes and other items to reduce overall latency

Was running into cases where we ran into lots of allocs and other items regarding the hot compute path.

Doing things to ensure we're preserving zero-copy for outputs, direct binding to allow hip graph to just run replay and performing prewarming for the hip graph combined with caching for IO, captures and state.

Simplified the compute to a direct bind - > staging -> eager path similar to the built in EP to further reduce overall latency.

…pile_at_load flag

With the regular hipMemCpy we're just syninc to the host slowing things down.

Removes item from hot path that causes overhead of a bunch of CPU bound work stalling before we can send anything to the GPU for compute

…rhead for large input models Effects batch < concurrency runs.

Removes the lookup overhead and movs most thing sto a unordeed_map lookup instead of Map. Clean up existing logic for cached values based on hash

Correctnes guard for compute threads

… lookups We're doing 3-4 lookups of O(n) for inputs, now things are reduced to one O(n) lookup now in compute. Done via caching staging inputs and outputs.

Saves constant O(n) lookups each inference run. Instead we reuse teh result until otherwise reset

… launches and output scatters in the hot compute path

Checks to heavy handed and not needed since inputs should be GPU resident at the point this is checked

…put/get_shaoe/length() lookup Cleans up items we don't need to keep checking per inference and just rely on a single compare + lookup. Reduces overhead per inference useful for smaller models in the steady state path

Make this more maintainble for each mode of direct, staging and eager. Cleanup some of the copy/zeroing logic as well for hipgraph runs to reuse shared code

…unction and seperate paths We have direct bind, stagged and eager mode now in the compute path

Reuse buffers, and drop extra lookup calls to getShape() by looking for initial dimension when we're in dynamic batching mode

Only do memset on startup so we're not constantly blockign CPU hot path so we aren't wasting cycles memsetting regions we just end up slicing out. This matters for first run/capture since we want a known state but that can be handled on initial allocation/startup prior to the hot compute() path

remove hip enqueue on hot path via hipMemsetAsyc for scratch Eliminate hash pass on every batch size change remove 1 alloc + free call for GatherInputshapesandBatch

Remove 1 heap alloc for extra outputs Remove 1 call per input for tensor data precompute bound_output_bytes and use that instead of calling shape::bytes

Remove PadInputTensor and associated signature in header simplify no_padding flag creation Mirror CopyStagingOutputstoOrt to use hipMemCpyAsync instead of hipmemCpyWithstream

leverage indexing so we don't have to do a bunch of map lookups

…ry for H2D DMA Ensure we're leveraging pinned memory from io_bound inputs so the DMA from H2D avoids a memcpy reducing overall latency The added reductions here reduce pointer checks as well as the shape scans to reduce overhead if we know IO is pinned and we're in a steady state. This avoids O(N) calls and just performs a lookup

…ned memory Make the logic to coalesce trigger based on host/device memory and not whether the inputs are paged or pinned. This allows us to always collapse the N dim input copies to one larger H2D copy

…r output

…during concurrent runs

TedThemistokleous added 30 commits August 21, 2026 09:13

Always load but allow for compile on session creation based on precom…

fd754c3

…pile_at_load flag

Fix missng include

34634a8

Update to use async non blocking copies

b0e3542

With the regular hipMemCpy we're just syninc to the host slowing things down.

Add legacy stream handling when user_compute_stream is enabled

ac6a6ae

Add in direct binding

bccaec6

Add direct bind caching and ultra fast path

980ed0f

Ensure async hipMemCpy for hipGraph and coalsce arena memory

a88ae42

Remove get_parameter_shapes() lookup each cpute run

43e4d19

Removes item from hot path that causes overhead of a bunch of CPU bound work stalling before we can send anything to the GPU for compute

Cache pointer and other setup for parameters to avoid direct bind ove…

14975cb

…rhead for large input models Effects batch < concurrency runs.

Clean up per-call strign work and move to lookups

f941501

Removes the lookup overhead and movs most thing sto a unordeed_map lookup instead of Map. Clean up existing logic for cached values based on hash

Add guard IFF thread is not on current device, reset it.

741e412

Correctnes guard for compute threads

Remove additional IO traversals in ot compute path and cache input IO…

f5983f7

… lookups We're doing 3-4 lookups of O(n) for inputs, now things are reduced to one O(n) lookup now in compute. Done via caching staging inputs and outputs.

Only check once and store result for coalesced inputs

44b70b0

Saves constant O(n) lookups each inference run. Instead we reuse teh result until otherwise reset

Additional optmizations to cache and remove heap allocs, map lookips,…

69d6b9f

… launches and output scatters in the hot compute path

Cleanup and remove eligible checks

c8d4b43

Checks to heavy handed and not needed since inputs should be GPU resident at the point this is checked

Modify lookup to use int64 instead of string lookup

0210b75

Simpify compute path

33efea9

Make this more maintainble for each mode of direct, staging and eager. Cleanup some of the copy/zeroing logic as well for hipgraph runs to reuse shared code

Update to cleanup code to make things more maintainable for compute f…

206d363

…unction and seperate paths We have direct bind, stagged and eager mode now in the compute path

Improve the slice/pad batching to reduce overhead

3603766

Reuse buffers, and drop extra lookup calls to getShape() by looking for initial dimension when we're in dynamic batching mode

Remove additional working during hot path with allocs, and hip calls

06170e8

remove hip enqueue on hot path via hipMemsetAsyc for scratch Eliminate hash pass on every batch size change remove 1 alloc + free call for GatherInputshapesandBatch

Handle additioanl hot compute bath items

1001a0f

Remove 1 heap alloc for extra outputs Remove 1 call per input for tensor data precompute bound_output_bytes and use that instead of calling shape::bytes

Cleanup redundant paths

5e24ba2

Remove PadInputTensor and associated signature in header simplify no_padding flag creation Mirror CopyStagingOutputstoOrt to use hipMemCpyAsync instead of hipmemCpyWithstream

Cache per input shapre for stready state

21e4693

leverage indexing so we don't have to do a bunch of map lookups

Add hot path logging

1755412

Add hipgraph warmup and capture outside of the hot path

6078475

Ensure coalesced IO always collapses to 1 H2D regardless of paged/pin…

b6ed816

…ned memory Make the logic to coalesce trigger based on host/device memory and not whether the inputs are paged or pinned. This allows us to always collapse the N dim input copies to one larger H2D copy

Remove hybird path and do direct bind output to save copy overhead pe…

9f18961

…r output

cache slot split vs padded and directg bind paths - avoids thrashing …

1080091

…during concurrent runs

TedThemistokleous requested review from apwojcik, tperry-amd and urpetkov-amd September 2, 2026 19:37

Update project and build scripts

7dd201f

urpetkov-amd approved these changes Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

Reduce compute io overhead - #107

Reduce compute io overhead#107
TedThemistokleous wants to merge 32 commits into
mainfrom
reduce_compute_io_overhead

TedThemistokleous commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

TedThemistokleous commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants