Maintenance mode caused stopped vms

[BartJM]

problem

We had a vm that was migrated from host A to host B after putting host A into maintenance mode. The first attempt at maintenance mode had a Errorinmaintenance so maintenance was canceled and started again.

After maintenance was done and maintenance mode canceled on host A, host B was set in maintenance. This caused the vm to be migrated to host A again. Right after the migration to host A was finished Cloudstack attempted to migrate the vm to host B for maintenance. But due to host B being in maintenance the com.cloud.agent.api.PrepareForMigrationCommand failed with com.cloud.exception.AgentUnavailableException. This caused Cloudstack to stop the vm.

The time between maintenance cancel on Host A and the migrations of the vm was around 2 hours.

versions

Cloudstack 4.19.3

The steps to reproduce the bug

Currently trying to reproduce on our testing environment but the steps would be

1. Host A: Put in maintenance
2. Host A: Cancel maintenance
3. Host A: Put in maintenance
4. Host A: Wait for host to be in maintenance
5. Host A: Cancel maintenance
6. Host B: Put in maintenance

What to do about it?

We do not expect Cloudstack to stop the vm due to a failed prepare for migration.

[BartJM]

I have been unable to reproduce the issue. I do have some additional info found:

• Not all vms that were migrated from Host B to Host A had the same issue.
• Found that the vms that were stopped had the migration started by c.c.h.HighAvailabilityManagerImpl, but I did not find the scheduling.

[DaanHoogland]

@BartJM , any updates on this. Have you considdered a ntp issue?
another cause may have been that multiple migrate async jobs were created. Any chance you can do any forensic work on the DB?
🤞

[BartJM]

I had not considered ntp issues, but we keep sync with chrony, so not expecting that to be an issue.

I think the cause is related to the multiple async maintenance jobs, since the stopped vms were not originally on Host A. I however expect that cancelling the maintenance and having the enabled state on the hypervisor does not start migrations due to a still present async job.

The main issue however is that the vm was stopped by Cloudstack after the migration to the host in maintenance failed.

Currently I do not have much time to further investigate, but if there are specific places in the db (or elsewhere) to look I will be able to check. Sadly was still not able to reproduce.

[DaanHoogland]

Currently I do not have much time to further investigate, but if there are specific places in the db (or elsewhere) to look I will be able to check. Sadly was still not able to reproduce.

well, you’d need historic data of the async job table to get any meaningful info, i think.

keep us updated,