[Enhancement] Check for PDBs status before draining Node

[viveksyngh]

Is this a BUG REPORT or FEATURE REQUEST?:

enhancement

What happened:

When node reaper tries to drain a node, it does not check if all pods can be disrupted and node can be drained. It keeps trying to drain the node until command does not timesout.

What you expected to happen:

It should check before trying to drain a node.

How to reproduce it (as minimally and precisely as possible):

Try to drain a node which is having pods with 0 disruption allowed.

Anything else we need to know?:

Environment:

• Kubernetes version:

kubectl version -o yaml

Other debugging information (if applicable):

• relevant logs:

kubectl logs <governor-pod>

[eytan-avisror]

This would make a great enhancement, however would be somewhat tricky as we would have to evaluate "disruptions allowed" - since there is no way to link an existing PDB CR to a node we are about to terminate - this would mean:

• List PDBs in the cluster
• List Pods in the cluster
• Map all pods to nodes that they are scheduled on
• Before terminating a node, evaluate whether a disruption is likely allowed (e.g. all pods are running / enough pods are running)

We can probably add a flag to enable/disable this feature

[shrinandj]

Another option might be to do kubectl drain --dry-run and see if that call succeeds. If there are PDBs that might get violated, the dry-run would fail (need to test this though).

[eytan-avisror]

Another option might be to do kubectl drain --dry-run and see if that call succeeds. If there are PDBs that might get violated, the dry-run would fail (need to test this though).

Good idea worth a check

[viveksyngh]

kubectl drain node --dry-run=client

does not check if it can disrupt the pod or not. It just print evicting for all pods on that node regardless of their pdb status.

I tried this on one of the node which node-reaper was failed to drain and has some pods which can be disrupted.

@eytan-avisror @shrinandj