Emit DisruptionBlocked events on affected pod or pdb resource

[cnmcavoy]

Description

What problem are you trying to solve?
As a cluster admin, I use Karpenter's events to understand and triage when disruption is occurring less frequently than expected. Karpenter emits DisruptionBlocked events when a node can not be disrupted, and if it is because of a pod (with a do-not-disrupt annotation) or pdb, the resource and namespace is in the event message:

karpenter/pkg/controllers/disruption/events/events.go

Lines 95 to 115 in c0e7299

	func Blocked(node *corev1.Node, nodeClaim *v1.NodeClaim, msg string) (evs []events.Event) {
	if node != nil {
	evs = append(evs, events.Event{
	InvolvedObject: node,
	Type: corev1.EventTypeNormal,
	Reason: events.DisruptionBlocked,
	Message: msg,
	DedupeValues: []string{string(node.UID)},
	})
	}
	if nodeClaim != nil {
	evs = append(evs, events.Event{
	InvolvedObject: nodeClaim,
	Type: corev1.EventTypeNormal,
	Reason: events.DisruptionBlocked,
	Message: msg,
	DedupeValues: []string{string(nodeClaim.UID)},
	})
	}
	return evs
	}

Because the InvolvedObject is the node + nodeclaim, the DisruptionBlocked events always end up in the default namespace, rather than the user's namespace with the pod or pdb. This means that the message has to be parsed by tools in order to extract the namespace from the event, which is burdonsome (and really hurts the ability to index these events in tools like datadog). Either emitting these events on the affected resource, or emitted a second duplicate event on the affected resource would satisfy our use-case.

How important is this feature to you?

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Interesting -- what's the problem that you are running into here that causes you to need these events? Are users creating faulty PDBs and then you need to inform them that they are fully blocking the node from being disrupted? Same thing with karpenter.sh/do-not-disrupt?

[cnmcavoy]

Right. We have spark users that annotate their worker pods with the do-not-disrupt annotation (or less commonly, misconfigure a pdb) that end up blocking disruption. We use datadog (which has integration into kubernetes events) and have dashboards set up for users so they can monitor their workload and namespace. Because the events are outside of the user namespace, it's not visible to them in this kind of presentation.

[engedaam]

/assign @GnatorX

[GnatorX]

Hey, I think this make sense however I think we wouldn't changed "Involved Object" but rather "Related" instead, would that work? Involved object is used to directly call out the object involved so it won't make sense to send consolidation block events to the pod or PDB

https://kubernetes.io/docs/reference/kubernetes-api/cluster-resources/event-v1/
https://www.pulumi.com/registry/packages/kubernetes/api-docs/core/v1/event/

[jonathan-innis]

Interesting, didn't know this existed @GnatorX! That's pretty cool! Seems completely reasonable to me if we could get that there

[jonathan-innis]

/triage accepted
/priority important-longterm

[GnatorX]

Ya! Initially I thought this was only supported in events/v1 but it seems like it was supported back in core/v1 events so we didn't need to upgrade Karpenter