Multiple alerts for HA on VM and host

[shwstppr]

problem

HA jobs for VM on host disconnection/down are not synchronized. CloudStack creates multiple HA jobs for the same VM when host goes into the Down state

versions

4.19.2.0-SNAPSHOT, other 4.19 and above versions as wel

The steps to reproduce the bug

1. Create a cluster with 2 hypervisor hosts
2. Create a HA enabled VM
3. Make the host running VM go into down state (used https://linuxconfig.org/how-to-crash-your-linux-system-with-fork-bomb)
4. After sometime when VM is started on a different hosts observe multiple HA alerts are generated for the same

What to do about it?

No response

[btzq]

We actually face this alot too! Ive noticed this but wasnt sure if this was normal.

In your test, does CS creating multiple jobs also reduce the likelihood of a sucessful HA?

Asking cause we often encounter situations where:

• In a large hypervisor, it it dies and HA is started, only a portion if VMs start up normally. The rest fails.
• When CS is busy, i encounter situations where a VM starts normally, but CS thinks its timedout, and stops the VM. It causes the VM to flap on and off for 10-15 minutes until it stops.

[btzq]

I am also now wondering. In CS i remember theres a global setting for max worker threads or ha threads.

Will fixing the duplicate HA jobs also improve the efficiency of these threads? 🤔

[shwstppr]

@btzq In my testing with 2x KVM hosts and limited VMs, it didn't fail HA for those VMs.

Fixing this should help the performance of HA in my opinion as it would spend less time working on duplicate jobs

[DaanHoogland]

@shwstppr , I tried to reproduce both with your fork-bomb method z(){ z|z& }; z& and with boldly powering off the host. I did get HA to kick in but only one job and one alert.
One extra thing I noticed is that the host going to disconnected state does not yield an alert, though I think that is worth one as well. We could include that in this work or create a new issue for it.

[DaanHoogland]

One extra thing I noticed is that the host going to disconnected state does not yield an alert, though I think that is worth one as well. We could include that in this work or create a new issue for it.

I spoke too early:
it actually happens multiple times. I did not get that for the VM HA job (yet)

[DaanHoogland]

reproduced both outputs now

I am noticing some serious changes in the alert manager between 4.19 and main, so not sure on which to address this yet.