Idle power usage
#6272
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Are there any improvements coming to ROCm regarding idle power usage?
I the situation that models are loaded both VLLM and llamacpp keep the GPU in the D0 state on ROCm, when using Vulkan on llamacpp with the RADV drivers it goes to D3Hot (I didn't expect a low idle with VLLM as I know that has some idle looping designed to keep cards awake)
So, for system power, this is with a R9700 32GB:
Idle, no models loaded: approx 24w D3hot
Idle, models loaded llamacpp on rocm: 70+w D0
Idle, models loaded llamacpp on vulkan: approx 24w D3hot.
More than doubling (almost trippling) the system power draw pushes me away from ROCm since my local models won't have anything like constant traffic, but loading them and unloading them after every use is also not ideal. I read somewhere this was for responsiveness - as long as a compute context was live the card will be kept fully awake, could we have the option to sacrifice a few milliseconds (if that) to save the power draw? or should this work and I just need to change some settings?
All reactions