Cluster warnings when deploying

[TR-RH]

Currently I have a bunch of services that have some extra objects to them, OnePasswordItems that create secrets based off of secret server vault items, and sometimes it takes a second or so to create those secrets.

I seem to get Keda warnings every time install one of these services KedaScalerFailed because those secrets aren't ready instantly. Keda is working fine and everything sorts itself out once those secrets are all created and the pod fully comes to life, but I'd just rather not have those warnings come up at all if possible. It's become just something I gloss over and I don't want to miss a real error mixed up in all those warnings when I'm deploying 10-15 services at a time.

Curious to know if there's any sort of waiting delay that I can use before Keda starts trying to deal with it's scaler. I've tried adding startup probes to the services to no avail.

[rickbrouwer]

I think you can use Argo CD sync waves when you use Argo CD (https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/), or Helm hooks (https://v2.helm.sh/docs/charts_hooks/).

This allows you to determine the order of deployment of resources.

[TR-RH]

I thought about using helm hooks, however from the documentation it sounds like any objects created during hooks aren't managed by the app, ie, when the service is uninstalled, the objects don't go with it.

[rickbrouwer]

Yes, you're right. We're having this issue on our cluster too.
We use Argo CD ourselves and I think we can solve this with sync-waves (but that requires some additional configuration).

But I'll think about it some more to see if I can come up with something.

[rickbrouwer]

Okay. As mentioned, I recognize the issue. I work with SealedSecrets that need to be converted to Secrets. During deployment, the ScaledObject is available before the SealedSecret converts it to a Secret, and that's why i have error logs and error events, same as you. As soon as the Secret is available, it starts scaling.

You mentioned whether there's a waiting delay within KEDA. As far as I know, there isn't one....yet, but perhaps introducing one wouldn't be such a bad idea. Quickly thought, maybe we can introduce an "initialStartUpPeriod" that prevents the ScaledObject from being processed at all until that period has elapsed.

If that sounds like a good idea, could you create an issue here with this proposal? Then we can discuss it first, and see if this is an acceptable solution for this use case.

[TR-RH]

Issue created.

I marked myself as Maybe for wanting to work on the implementation. I'm fairly new to Kubernetes and I think this might be an interesting exercise, but I don't know if I'd have the knowledge to be confident in a solution, but once there is some discussion on the issue I may take a stab at it, whether I submit it or not.