Replica Count Fluctuation at Cron Trigger Transition (start/end overlap) on KEDA v2.16.1

[devkoriel]

Report

We are experiencing an issue with the KEDA cron trigger where, at the overlapping point between the end of one cron schedule and the start of the next, the actual replica count fluctuates instead of being exactly set to the specified desiredReplicas. This fluctuation occurs even though the workload’s readinessProbe is correctly configured, ultimately causing intermittent 5xx errors.

Expected Behavior

We expected that during the transition between cron triggers, the replica count would immediately and precisely match the desiredReplicas value defined in the active cron trigger. This stable state is critical for ensuring that the application remains healthy and does not encounter readinessProbe failures or resulting 5xx errors.

Actual Behavior

Despite applying all recommended workarounds (such as aligning the cooldownPeriod with stabilizationWindowSeconds), the replica count does not consistently adjust to the exact desiredReplicas at the transition period. For example, instead of settling at 10 replicas, we observe fluctuations (e.g., 10 → 12 → 11), which leads to readinessProbe failures and sporadic 5xx errors.

Steps to Reproduce the Problem

1. Deploy the Workload:
   • Use a Deployment that has a properly configured readinessProbe.
2. Configure the ScaledObject with Multiple Cron Triggers:
   • Create a KEDA ScaledObject targeting the workload with at least two cron triggers where the end of one schedule overlaps with the start of the next.
   • For instance, one trigger with desiredReplicas of 40 from 08:00 to 10:00 and another with desiredReplicas of 10 from 10:00 onward.
3. Set Timing Parameters:
   • Configure the cooldownPeriod and set advanced.horizontalPodAutoscalerConfig.behavior.scaleDown.stabilizationWindowSeconds to the same value (e.g., 300 seconds) to attempt to stabilize the scaling behavior.
4. Monitor the Scaling Behavior:
   • Observe that during the overlapping period, the replica count does not exactly match the defined desiredReplicas and instead fluctuates between values (for instance, 10 → 12 → 11).
5. Notice the Impact:
   • These fluctuations cause readinessProbe failures, leading to intermittent 5xx errors from the application.

Logs from KEDA operator

2025-04-01 17:42:10.228 2025-04-01T08:42:10Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "03a2273f-5c74-4683-ba8e-cab1b26083d8", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 17:42:10.211 2025-04-01T08:42:10Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "03a2273f-5c74-4683-ba8e-cab1b26083d8"} 2025-04-01 17:42:10.198 2025-04-01T08:42:10Z INFO Initializing Scaling logic according to ScaledObject Specification {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "a7f448ce-5ad9-45e5-a2de-b577eb458d40"} 2025-04-01 17:42:10.198 2025-04-01T08:42:10Z INFO Updated HPA according to ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "a7f448ce-5ad9-45e5-a2de-b577eb458d40", "HPA.Namespace": "[redacted-namespace]", "HPA.Name": "[redacted-hpa]"} 2025-04-01 17:42:10.191 2025-04-01T08:42:10Z INFO Updated HPA according to ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "a7f448ce-5ad9-45e5-a2de-b577eb458d40", "HPA.Namespace": "[redacted-namespace]", "HPA.Name": "[redacted-hpa]"} 2025-04-01 17:42:10.159 2025-04-01T08:42:10Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "a7f448ce-5ad9-45e5-a2de-b577eb458d40", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 17:42:10.135 2025-04-01T08:42:10Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "a7f448ce-5ad9-45e5-a2de-b577eb458d40"} 2025-04-01 16:31:12.231 2025-04-01T07:31:12Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "17564aab-29f5-46b2-ae93-46e29be0497a", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 16:31:12.214 2025-04-01T07:31:12Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "17564aab-29f5-46b2-ae93-46e29be0497a"} 2025-04-01 16:31:12.199 2025-04-01T07:31:12Z INFO Updated HPA according to ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "9617bbcb-1f2c-4ec3-8af0-67d6f3711e07", "HPA.Namespace": "[redacted-namespace]", "HPA.Name": "[redacted-hpa]"} 2025-04-01 16:31:12.199 2025-04-01T07:31:12Z INFO Initializing Scaling logic according to ScaledObject Specification {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "9617bbcb-1f2c-4ec3-8af0-67d6f3711e07"} 2025-04-01 16:31:12.191 2025-04-01T07:31:12Z INFO Updated HPA according to ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "9617bbcb-1f2c-4ec3-8af0-67d6f3711e07", "HPA.Namespace": "[redacted-namespace]", "HPA.Name": "[redacted-hpa]"} 2025-04-01 16:31:12.153 2025-04-01T07:31:12Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "9617bbcb-1f2c-4ec3-8af0-67d6f3711e07", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 16:31:12.135 2025-04-01T07:31:12Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "9617bbcb-1f2c-4ec3-8af0-67d6f3711e07"} 2025-04-01 16:28:28.615 2025-04-01T07:28:28Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "1ca12434-a65c-4872-93fe-f3992a9082df", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 16:28:28.573 2025-04-01T07:28:28Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "1ca12434-a65c-4872-93fe-f3992a9082df"} 2025-04-01 16:28:28.560 2025-04-01T07:28:28Z INFO Updated HPA according to ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "e26f406d-7e1c-47ec-ba18-df855e3f9e25", "HPA.Namespace": "[redacted-namespace]", "HPA.Name": "[redacted-hpa]"} 2025-04-01 16:28:28.560 2025-04-01T07:28:28Z INFO Initializing Scaling logic according to ScaledObject Specification {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "e26f406d-7e1c-47ec-ba18-df855e3f9e25"} 2025-04-01 16:28:28.522 2025-04-01T07:28:28Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "e26f406d-7e1c-47ec-ba18-df855e3f9e25", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 16:28:28.505 2025-04-01T07:28:28Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "e26f406d-7e1c-47ec-ba18-df855e3f9e25"} 2025-04-01 14:18:04.438 2025-04-01T05:18:04Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "cd59f49f-b695-40ba-ab4c-7d8f3a9c44c5", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 14:18:04.415 2025-04-01T05:18:04Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "cd59f49f-b695-40ba-ab4c-7d8f3a9c44c5"} 2025-04-01 14:18:04.396 2025-04-01T05:18:04Z INFO Initializing Scaling logic according to ScaledObject Specification {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "613da6ca-587f-4fd0-b009-f3e0975d153c"} 2025-04-01 14:18:04.361 2025-04-01T05:18:04Z INFO Creating a new HPA {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "613da6ca-587f-4fd0-b009-f3e0975d153c", "HPA.Namespace": "[redacted-namespace]", "HPA.Name": "[redacted-hpa]"} 2025-04-01 14:18:04.350 2025-04-01T05:18:04Z INFO Detected resource targeted for scaling {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "613da6ca-587f-4fd0-b009-f3e0975d153c", "resource": "apps/v1.Deployment", "name": "[redacted-deployment]"} 2025-04-01 14:18:04.332 2025-04-01T05:18:04Z INFO Reconciling ScaledObject {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name": "[redacted-scaledobject]", "namespace": "[redacted-namespace]"}, "namespace": "[redacted-namespace]", "name": "[redacted-scaledobject]", "reconcileID": "613da6ca-587f-4fd0-b009-f3e0975d153c"}

KEDA Version

2.16.1

Kubernetes Version

1.30

Platform

Google Cloud

Scaler Details

Cron

Anything else?

• We have confirmed this behavior on KEDA version 2.16.1 (the latest release).
• Despite employing all known configuration adjustments and workarounds, the issue persists.
• We are interested in understanding if this behavior is already known and if there are any existing workarounds.
• If not, it might indicate an underlying bug that requires a fix.
• I am prepared to contribute by submitting a PR to address this issue if the community agrees on the approach.

[rickbrouwer]

Hello!

Step 2 in the "to Reproduce the Problem" section, can you share your scaledObject? I don't quite understand the example mentioned there and with the scaledObject I can view it directly and possibly reproduce it.

[devkoriel]

Sure! Here's the example ScaledObject where some sensitive info's redacted.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: REDACTED
  namespace: REDACTED
  labels:
    deploymentName: REDACTED
spec:
  minReplicaCount: 6
  maxReplicaCount: 40
  cooldownPeriod: 300
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: REDACTED
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Pods
            periodSeconds: 30
            value: 1
          - type: Percent
            value: 20
            periodSeconds: 30
          selectPolicy: Max
        scaleUp:
          stabilizationWindowSeconds: 300
          policies:
          - type: Pods
            periodSeconds: 30
            value: 5
          - type: Percent
            value: 20
            periodSeconds: 30
          selectPolicy: Max
  triggers:
  - type: cron
    metadata:
      timezone: Asia/Seoul
      start: 0 8 * * 1-5
      end: 0 10 * * 1-5
      desiredReplicas: "40"
  - type: cron
    metadata:
      timezone: Asia/Seoul
      start: 0 10 * * 1-5
      end: 0 18 * * 1-5
      desiredReplicas: "10"
  - type: cron
    metadata:
      timezone: Asia/Seoul
      start: 0 12 * * 6
      end: 30 4 * * 1
      desiredReplicas: "2"

[rickbrouwer]

At the moment I can't reproduce it yet. I used your scaledObject and adjusted the times a bit so I could see how the transition would be. In the event logging I see SuccessfulRescale events. What I do notice is the 3rd trigger. This has a desiredReplicas of 2 while the minReplicaCount is set to 6. This gives an event New size: 6; reason: Current number of replicas below Spec.MinReplicas.

I am curious if others in the community can reproduce it.