Skip to content

[Bug]: Health check misses devices that share a physical GPU #1929

Description

@tryuuu

1. Quick Debug Information

  • OS/Version(e.g. RHEL8.6, Ubuntu22.04): Amazon Linux 2023
  • Kernel Version: Kernel-6.1
  • Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd
  • K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): EKS(v1.34)

2. Issue or feature description

  • Without MIG (time slicing or MPS). When multiple logical GPUs (replicas) correspond to one physical GPU UUID, an Xid error on that UUID should report all of them as unhealthy to the kubelet. Currently only one logical GPU is reported. Since the physical GPU itself is broken, every logical GPU tied to it should become unhealthy.
  • With MIG. Even when multiple MIG instances inside one physical GPU are broken, at most one is reported as unhealthy, and in some cases none at all. If the value of parentToDeviceMap[uuid] happens to be a pointer to a device that is not broken, the GI and CI check rejects the event and nothing is reported.

Metadata

Metadata

Assignees

Labels

bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions