How to make NVIDIA MIG Manager restart custom (pre-installed) device plugin daemonset during MIG reconfiguration?

[shachijun]

Hello,

I have a pre-installed NVIDIA device plugin running as a daemonset in my Kubernetes cluster, and now I'm trying to perform some MIG operations.

Current Behavior:

When MIG configuration changes, MIG Manager automatically restarts GPU operator components
MIG Manager also restarts system services defined in default-gpu-clients ConfigMap
However, my pre-installed device plugin daemonset doesn't get restarted

What I've Tried:

Added labels to match GPU operator:

labels:

  app: nvidia-device-plugin
  nvidia.com/device-plugin: "true"

Question:
How can I make MIG Manager restart my custom device plugin daemonset when MIG configuration changes?

[cdesiniotis]

@shachijun we don't currently have a solution for this. It may make sense for mig-manager to use a common label, i.e. nvidia.com/management-container, to identify which pods are GPU management pods (like device-plugins, monitoring agents, etc) and therefore need to be restarted across MIG reconfigurations.