static-pod initialization for proper resource accounting

[ajaysundark]

Static Pod Readiness

Static pods are managed directly by the Kubelet on a specific node, without the API server's involvement in their creation. To make them visible to the control plane, the Kubelet creates a Mirror Pod for each static pod in the API server. However, if there's a delay before Mirror Pods are created, the scheduler is unaware of the resources (cpu/memory) consumed by these static pods and could potentially over-commit on the node[1].

This race condition could be avoided with NRC to ensure a node only accepts workloads after all static pods are successfully synchronized with the API server.

This doc issue is to demonstrate how a npd custom-plugin script could be utilized to work with NRC using a 'StaticPodsSynced' node condition.

[1] Ref:

More details here: kubernetes/kubernetes#115325, kubernetes/website#47264 and kubernetes/kubernetes#126870

[ajaysundark]

Notes:

1. Demonstrate also as an example on how node-readiness-controller can be integrated into NPD eco-system with a custom plugin
2. Deliverables on this feature would be
   • a npd plugin script that reads static-manifests directory "/etc/kubernetes/manifests" on the node and checks against API server that the corresponding mirror-pods are found.
   • add a custom plugin configuration to set the 'StaticPodsSynced' condition based on the custom npd-plugin evaluation
   • create a bootstrap-only Node Readiness Rule so user can test this in their cluster
3. This can possibly be tested in a Kind cluster by leveraging extraMounts feature to mount a local directory (eg: ./my-static-manifests) into the kind node container. I expect one also need to set 'enforceNodeAllocatable' on the kind node to repro this issue. This needs some more investigation from the contributor to understand the issue and handle it.

[ajaysundark]

cc @yashasvimisra2798

[ajaysundark]

removing the doc label as this is more of a feature than a quick doc update.

[GGh41th]

Hello @ajaysundark if you currently don't have the bandwidth to work on this , i can work on it !

[ajaysundark]

Sure, Thanks @GGh41th!

[ajaysundark]

/assign @GGh41th