Using separate cuda streams for one session

[cozeybozey]

Describe the issue

I have multiple threads that are calling session.run on one session. I recently made it so I am using pinned memory and asynchronous mem copies, which is working great. However, to do this I am using separate cuda streams for the mem copies. I noticed that session.run does not work with these cuda streams. I can link one cuda stream to a session via the options, but I want to use multiple cuda streams and a different one for every run call. How can I achieve this? Or should I just use multiple sessions instead? But then I will have multiple instances of the same model in memory, which doesn't seem great.

To reproduce

--

Urgency

No response

Platform

Windows

OS Version

10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

16.2

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

No response

[YT-Chow]

I have the same issue. In the multiple threads that are calling session.run on one session. when calling
Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);
it will throw an error like this:
CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=Windows11 ; file=D:\a\_work\1\s\onnxruntime\core\providers\cuda\gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));

[don-reba]

We have the same problem as @cozeybozey. We are currently loading our model into two sessions, in order to use two CUDA streams and overlap our memory transfers. This does work, but it doubles the GPU memory requirements and loading time. Ideally, user_compute_stream would be moved from a session option to a run option.

[don-reba]

I don't see anything referencing streams in recent commits. How is this issue completed?

[andrea-agostini-snkeos]

I have the same issue. In the multiple threads that are calling session.run on one session. when calling Ort::MemoryInfo memoryInfo = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault); it will throw an error like this: CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=Windows11 ; file=D:\a\_work\1\s\onnxruntime\core\providers\cuda\gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));

Did you ever figure out why the error was thrown? I'm facing the same right now, wondering if it has anything to do with me concurrently synchronizing the default stream on another thread...

[microsoft-github-policy-service]

Applying stale label due to no activity in 30 days. Please react to the issue by assigning it or labeling it as 'contributions welcome', 'documentation', 'feature request', 'regression', or 'no stale'.