workflow controller keeps crashing under load

[static-moonlight]

Scenario: we are using Argo to run smaller workflows, lots of them, during normal operation ~500 per hour.

After a small outage, Argo gets flooded with 1000+ workflows. It seems the workflow controller can't handle that.

    State:          Running
      Started:      Thu, 27 Feb 2025 09:43:32 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 27 Feb 2025 09:34:22 +0000
      Finished:     Thu, 27 Feb 2025 09:38:21 +0000
    Ready:          True
    Restart Count:  47
    Liveness:       http-get http://:6060/healthz delay=90s timeout=30s period=60s #success=1 #failure=3

Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Normal   Started    51m (x3 over 59m)     kubelet  Started container workflow-controller
  Normal   Killing    47m (x4 over 59m)     kubelet  Container workflow-controller failed liveness probe, will be restarted
  Normal   Created    47m (x4 over 59m)     kubelet  Created container workflow-controller
  Normal   Pulled     26m (x8 over 59m)     kubelet  Container image "quay.io/argoproj/workflow-controller:v3.5.11" already present on machine
  Warning  Unhealthy  10m (x32 over 61m)    kubelet  Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff    6m51s (x65 over 35m)  kubelet  Back-off restarting failed container workflow-controller in pod workflow-controller-5875f4b9c4-lplx6_argo(870d7cbf-3b47-45bf-93df-7a32fd479c06)

This is a serious problem. I need to know why the workflow controller keeps crashing. How do I find out? Where do I need to look?

On that note: I also need ideas how to make it more stable/resilient/reliable.

Any ideas?

[static-moonlight]

related to death by workflow never reconciled