Monitoring application health flapping

[pstefka]

Hey there,

I found out, that one of our apps Health is flapping, i.e. 99% of time it is Healthy, but for a very short period it changes to Degraded (up to 3 times a minute), which is not even visible in the ArgoCD GUI.
Our current alarms require the Degraded state to last for a few minutes, so a very short <1 sec Degraded state won't be detected.

After investigation, I don't think flapping can be identified using Prometheus metrics, as the base argocd_app_info metrics is a gauge, i.e. Prometheus would need to check in the exact moment the application is degraded, which is not very probable.

Is it possible to alarm on application health flapping?

Thank you!

:P.