Karpenter should handle when it is creating nodes that aren't used

[GnatorX]

Description

What problem are you trying to solve?

• I want Karpenter to notify me if it is creating nodes that aren't being used by pods
• I want Karpenter to stop creating nodes after some time of creating nodes that no pod is using

In cases like #1928 and some times user error such as bad AMI or bad requirements or incorrect kubelet configurations, Karpenter would create nodes that may not ever fit pods it created the nodes for. As operators we want to be able to see this and possibly configure a way to stop this behavior.

How important is this feature to you?
This is pretty important because we often see node churn that isn't necessary

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[GnatorX]

Will keep an eye against #1910 to further include other conditions such as this one

[jonathan-innis]

As operators we want to be able to see this and possibly configure a way to stop this behavior

Can you specify how you are thinking about this @GnatorX? As I'm just thinking about this at a high-level, it seems like we should just fix the underlying problem, which is that we shouldn't be launching capacity when there are pods that aren't currently able to be drained.

[jonathan-innis]

/triage needs-information
/priority awaiting-more-evidence

[GnatorX]

Can you specify how you are thinking about this @GnatorX? As I'm just thinking about this at a high-level, it seems like we should just fix the underlying problem, which is that we shouldn't be launching capacity when there are pods that aren't currently able to be drained.

Ya that fixes #1928 however I think in the first case we still would unnecessarily churn nodes. Specifically if we either incorrectly configured nodepools or nodes aren't configured properly. Similar to #1910 but beyond registration. I want to essentially be able to say NodePool A tried to create a new for pending Pod X N times but all N times the node wasn't used by the pod and was consolidated, maybe something is wrong between the NodePool and Pod and we should emit a metric or stop trying to create a node for this pod.