Feature Request: Blocking device plugin startup when another instance is already runniung

[djsly]

We are looking to augment the device plugin with a semaphore that would prevent the process from starting when another process is already running.

for our own case, we will be running the device plugin natively through systemd, and we would like to ensure that if a daemonset is deployed for the device plugin, that it will force the pods to go into a crashloopbackoff start due to recurrent start error.

[elezar]

As an alternative, could one update a node label / taint on nodes that have the device plugin running as a systemd unit and have the device plugin not schedule on these nodes using Affinity / Anti-Affinity?

[djsly]

@elezar as a Kubernetes provider (AKS) this is too intrusive to the user's clusters. AKS cannot alter the user node taints, since this would have a major repercussion to their other workload. As far as the label go with anti-affinity, this will probably have 0 effect for most of the users who are using a generic chart to install the daemonset today. We cannot ask ALL of the users to upgrade their chart versions or their in-house manifest to add those new anti-affinities, unfortunately.

Blocking the second instance seems to be the safest way to ensure all of the users (the ones using a version >= the version that will introduce this change) will automatically see failures with their daemonset once AKS add the managed device plugin on their nodes.

Note that we will be telling users they should disable their daemonset themselves, add taints/label and what not, but knowing that not everyone will follow the instructions we are looking for a catch all solution.

[djsly]

@elezar friendly ping.