Add NUMA node ID attribute for NUMA aware allocation between CPU, GPU and NICs #690
Conversation
github-project-automation
bot
moved this to Backlog
in Planning Board: k8s-dra-driver-gpu
… GPU and NICs Signed-off-by: Shiva Krishna, Merla <[email protected]>
shivamerla
force-pushed
the
add_numa_node_attribute
branch
from
0fafb15 to
9ebf2a4
Compare
| if id, ret := device.GetNumaNodeId(); ret == nvml.SUCCESS { | ||
| numaNodeID = &id | ||
| } else { | ||
| klog.Warningf("error getting NUMA node ID for device %d, continuing without attribute: %v", index, ret) |
There was a problem hiding this comment.
Interesting! Would this be entirely exceptional, or is this an expected scenario on some systems?
This leads me to thinking:
-
On non-NUMA systems this should probably not be a warning. On those systems, I suppose this code path is expected, and maybe it's worth emitting an info-level log message.
-
On NUMA systems this should maybe be an error. At least I'd like to hear our thoughts about this.
-
How can we reliably distinguish a NUMA from a non-NUMA system (as a reference point)?
There was a problem hiding this comment.
ref docs for nvmlDeviceGetNumaNodeId() indicate that we should distinguish three types of errors:
NVML_SUCCESS if the NUMA node is retrieved successfully
NVML_ERROR_NOT_SUPPORTED if request is not supported on the current platform
NVML_ERROR_INVALID_ARGUMENT if device node is invalid
I don't want to answer my own questions from above, but it looks like we probably want to give NVML_ERROR_NOT_SUPPORTED its own code path (not emitting a warning/error log), and then NVML_ERROR_INVALID_ARGUMENT is handled separately (and maybe that one should be fatal?).
There was a problem hiding this comment.
@jgehrcke good point, the not supported error can be logged as info instead of warning. After discussing with Kevin, we actually want to use the nvmlDeviceGetMemoryAffinity() . This will give the bitmask of NUMA nodes as the GPU can belong to multiple within the given socket. This means the current matchAttibute is not sufficient for this. Instead i will propose a new parameter to allow attributes to be matched from a list instead of a single element.
In that case it will be new set of error codes that we need to handle and i will update based on that.
3 participants
No description provided.