Scale based on k8s native events

[Omar-Bishtawi]

Proposal

the idea of this scaler is to scale workloads based on k8s native event.

Usecase

a sample use case for this scaler is with DeprovisioningBlocked event . this event is thrown when k8s isn't able to drain a node, there are multiple reasons for this one of them is due to PDB (Pod disruption budges) .

here is a sample event content that can be taken from kubectl get events -o json:

{
            "apiVersion": "v1",
            "count": 1,
            "eventTime": null,
            "firstTimestamp": "2023-08-13T08:47:00Z",
            "involvedObject": {
                "apiVersion": "karpenter.sh/v1alpha5",
                "kind": "Machine",
                "name": "RESOURCE_NAME",
                "resourceVersion": "1069644099",
                "uid": "ad90b678-0693-4ef1-929b-1091fa68e4ea"
            },
            "kind": "Event",
            "lastTimestamp": "2023-08-13T08:47:00Z",
            "message": "Cannot deprovision machine due to pdb {NAMESPACE}/{PDB_BANE} prevents pod evictions",
            "metadata": {
                "creationTimestamp": "2023-08-13T08:47:00Z",
                "name": "RESOURCE_NAME",
                "namespace": "default",
                "resourceVersion": "1070337960",
                "uid": "da73d9a0-7ecb-48f6-a9e0-47d359d9aee4"
            },
            "reason": "DeprovisioningBlocked",
            "reportingComponent": "",
            "reportingInstance": "",
            "source": {
                "component": "karpenter"
            },
            "type": "Normal"
        },

the scaler will be useful to resolve this issue and add new pod when this event is thrown which will allow the node to be drained since there is enough pods to maintain the PDB condition.

Scaler Source

K8s Events API

Scaling Mechanics

we can scale based on the response of k8s Events API .

here is a suggested format for the keda scaled object trigger:

triggers:
- type: kubernetes-events
  metadata:
    eventsFilters:
    - field: reason
      type: exact
      value: 'DeprovisioningBlocked'
    - field: message
      type: regex
      value: '.*due to pdb {NAMESPACE}.*}'
    eventCountThreshold: 3

Authentication Source

KEDA Service account (same as Kubernetes Workload scaler)

Anything else?

I'm willing to work on this scaler and create the PR for it if you don't have any notes on this.

[tomkerkhove]

I like this idea!

[kangteng525]

I like this idea as well!

Meanwhile I have another requirement described in #4883
It would be great if this scaler could be designed to support this requirement as well.

Thanks,
Kevin

[Omar-Bishtawi]

@kangteng525 for what is described in #4883 it should be doable if you have k8s event for the update event.
I did a quick test, the deployment only fire an event when the number of replicas change , but there is no event fired when other values are updated (ex: image tag) .
if other k8s resources fire an event when the update happens to the deployment then this is doable using this scaler(ex: ci/cd solutions) . but if not I think your best bet will be an admission hook .

[tomkerkhove]

Are you open to contribute it if we agree with the idea @Omar-Bishtawi?

Any thoughts @kedacore/keda-maintainers?

[Omar-Bishtawi]

Hi @tomkerkhove yes I don't have any issue in this.
and Sorry I thought I tagged Kevin in the last comment

[kangteng525]

hi @Omar-Bishtawi , thanks for looking into this requirement!
Yes, you are right, I have also done similar test, when the deployment replicas scaled to 0, and update the image for this deployment, there will be no events fired, also the status of the deployment will not be changed.
So the admission webhook might be a possible solution.

@tomkerkhove @JorTurFer do you have any suggestion?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.