DRA: CPU Placement Bit String

[johnbelamaric]

Enhancement Description

One of the key goals of DRA is to help with alignment of devices in the intra-node topology. In the current DRA incarnation, this is done via the matchAttributes constraint. This allows you to require that specified attributes for all devices satisfying the request have the same value. For example, the NIC and GPU that are selected must share the same PCIe root complex.

One issue with this is that it does not include CPU alignment. Another issue is that intra-node topologies vary widely and are often quite complex; simple attribute matching may not be sufficient in many cases. It also requires a fair bit of knowledge on the part of users as to how to align these different devices.

The approach taken by Slurm, an HPC scheduler, is a bit different. Rather than requiring an understanding of specific attributes, Slurm calculates a standardize CPU-placement bit string for every device. That is, it normalizes based up on the number of CPUs in the node, and publishes a bit string for each device that represents which CPU(s) that device is aligned with. This localizes the alignment logic to the node, rather than requiring users to understand it in depth.

This bit string is calculable based on data in /proc and similar places on Linux machines, and the folks building Slurm have agreed to develop the necessary code to calculate this bit string in OSS to share with Kubernetes. This common library could be used by DRA plugin authors to publish the common placement bit string, and an alignment constraint option added to the ResourceClaim constraints. This will allow users to require alignment between their devices without having to understand the details of intra-node topology. It would also allows us to optimally align by default (that is, without the user asking, we would align if possible; with the constraint, we would fail scheduling if we cannot align).

• One-line enhancement description (can be used as a release note): Enable DRA drivers to publish CPU alignment data, and allow DRA users to require that alignment during scheduling
• Kubernetes Enhancement Proposal: TBD
• Discussion Link: Experimenting with managing CPU alignment with DRA cncf/maintainer-summit#15
• Primary contact (assignee): @johnbelamaric @catblade
• Responsible SIGs: Node, Scheduling
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.34
  • Beta release target (x.y): 1.35
  • Stable release target (x.y): 1.36
• Alpha
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[johnbelamaric]

/sig node
/sig scheduling
/wg device-management

/cc @pohly @klueska @mrunalp