orion-interference-aware-fine-grained-gpu-sharing-for-ml-applications.md

Orion: Interference-aware, fine-grained GPU sharing for ML applications

Meta Info

Presented in EuroSys 2024.

Understanding the paper

• Orion — a system that transparently intercepts GPU kernel launches from multiple clients sharing a GPU
• It schedules work on the GPU at the granularity of individual operators and minimizes interference by taking into account each operator’s compute and memory requirements
• Integrated into PyTorch

Technical details

• Influence the behavior of the hardware scheduler by using CUDA stream priorities
• CUDA Events to monitor the progress of each stream in the GPU
• Schedule each cudaMemcpy operation by considering its PCIe bandwidth requirements and current bus bandwidth utilization
• Use NVIDIA Nsight Compute and NVIDIA Nsight Systems to collect the compute throughput, memory throughput, and execution time of each kernel

Evaluation

• Baselines
  • Temporal sharing — time-slice the GPU by executing one job’s request at a time
  • NVIDIA MPS
  • CUDA Streams
  • REEF