HA : Hosts persist in the Suspect state in HA cluster with ShareMountPoint

[Luskan777]

ISSUE TYPE

• Bug Report

COMPONENT NAME

HA, KVM

CLOUDSTACK VERSION

4.20

CONFIGURATION

Zone type : Advanced Network
Primary Storage: ShareMountPoint

OS / ENVIRONMENT

Hosts OS: Ubuntu 22.04 (HPE ProLiant BL460c Gen10)
Management Server OS: Ubuntu 22.04
out-of-band management driver: IPMI

SUMMARY

Hello, I configured out-of-band management on my hosts, however, the HA status of my hosts is always between Suspect or DEGRADED, I have already checked the IPMI communication and everything is working, my servers are also on and operational.

STEPS TO REPRODUCE

Configure Hosts KVM
Configure HA provider with KVMHAProvider
Configure out-of-band management with IPMI driver
Enable HA and see HA State

EXPECTED RESULTS

HA hosts with AVAILABLE state

ACTUAL RESULTS

Managemente Server logs:

@MSLOG@:2025-01-07 00:29:25,698 DEBUG [o.a.c.h.HAManagerImpl] (pool-4-thread-21:[]) HA state post-transition:: new state=[Suspect], old state=[Checking], for resource id=[3], status=[true], ha config state=[Suspect].
@MSLOG@:2025-01-07 00:29:25,707 DEBUG [o.a.c.h.HAManagerImpl] (pool-4-thread-21:[]) Transitioned host HA state from:Checking to:Suspect due to event:TooFewActivityCheckSamples for the host id:3
@MSLOG@:2025-01-07 00:29:41,622 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:[ctx-28440d8d]) HA state post-transition:: new state=[Checking], old state=[Suspect], for resource id=[2], status=[true], ha config state=[Checking].
@MSLOG@:2025-01-07 00:29:41,629 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:[ctx-28440d8d]) Transitioned host HA state from:Suspect to:Checking due to event:PerformActivityCheck for the host id:2

2025-01-07 15:44:06,928 DEBUG [o.a.c.u.p.ProcessRunner] (pool-2-thread-11:[]) Process standard output for command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 10.16.20.21 -p 623 -U cloudstack -P ***** chassis power status]: [Chassis Power is on
].
2025-01-07 15:44:06,928 DEBUG [o.a.c.u.p.ProcessRunner] (pool-2-thread-11:[]) Process standard error output command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 10.16.20.21 -p 623 -U cloudstack -P ***** chassis power status]: [Running Get PICMG Properties my_addr 0x20, transit 0, target 0x20
Error response 0xc1 from Get PICMG Properities
Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20
Invalid completion code received: Invalid command
Discovered IPMB address 0x0
].
2025-01-07 15:44:06,929 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (pool-2-thread-11:[]) The command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 10.16.20.21 -p 623 -U cloudstack -P PASSWORD  chassis power status] was successful and got the result [Chassis Power is on].

KVM hosts logs:

2025-01-07 15:49:52,534 DEBUG [kvm.resource.KVMHAChecker] (pool-1067-thread-1:[]) (logid:) Checking heart beat with KVMHAChecker for host IP [IP_SERVER] in pools []
2025-01-07 15:49:52,534 WARN  [kvm.resource.KVMHAChecker] (pool-1067-thread-1:[]) (logid:) All checks with KVMHAChecker for host IP [IP_SERVER] in pools [] considered it as dead. It may cause a shutdown of the host.

[DaanHoogland]

@Luskan777 , can you try the ipmitool from the MS log by hand and examine the output?

The value kvm.ha.activity.check.max.attempts is used to decide how often the HA provider will try before deciding to fence or reinstate the host. The default is 10. The line

@MSLOG@:2025-01-07 00:29:25,707 DEBUG [o.a.c.h.HAManagerImpl] (pool-4-thread-21:[]) Transitioned host HA state from:Checking to:Suspect due to event:TooFewActivityCheckSamples for the host id:3

should be there 10 times as a consequence.

Also maybe you can try a force reconnect for the host.

[slavkap]

Hi @Luskan777, as far as I know, Host HA is only available for NFS, StorPool and Linstor storage pools. That's why there aren't any listed pools in your log

2025-01-07 15:49:52,534 DEBUG [kvm.resource.KVMHAChecker] (pool-1067-thread-1:[]) (logid:) Checking heart beat with KVMHAChecker for host IP [IP_SERVER] in pools []
2025-01-07 15:49:52,534 WARN  [kvm.resource.KVMHAChecker] (pool-1067-thread-1:[]) (logid:) All checks with KVMHAChecker for host IP [IP_SERVER] in pools [] considered it as dead. It may cause a shutdown of the host.

[Luskan777]

Hi @Luskan777, as far as I know, Host HA is only available for NFS, StorPool and Linstor storage pools. That's why there aren't any listed pools in your log

2025-01-07 15:49:52,534 DEBUG [kvm.resource.KVMHAChecker] (pool-1067-thread-1:[]) (logid:) Checking heart beat with KVMHAChecker for host IP [IP_SERVER] in pools []
2025-01-07 15:49:52,534 WARN  [kvm.resource.KVMHAChecker] (pool-1067-thread-1:[]) (logid:) All checks with KVMHAChecker for host IP [IP_SERVER] in pools [] considered it as dead. It may cause a shutdown of the host.

Hi @slavkap ,

Thanks for your reply, I'm using ShareMountPoint with GFS2, maybe that explains the problem.

I would like to develop support for pools that use ShareMountPoint, looking at the code it seems possible, however, I'm new to Cloudstack development, @slavkap and @DaanHoogland , could you tell me where I can start to solve this problem? Maybe pointing to some files or some documentation, I would be grateful for any help :)

[DaanHoogland]

I would like to develop support for pools that use ShareMountPoint, looking at the code it seems possible, however, I'm new to Cloudstack development, @slavkap and @DaanHoogland , could you tell me where I can start to solve this problem? Maybe pointing to some files or some documentation, I would be grateful for any help :)

@Luskan777, there are several start points;

• the cwiki,
• especially the dev and
• marvin test
• the hackerbook

git blame is also a great help to look for help ;)

As a code entry point, I would start looking in the HAManagerImpl

hope this helps.

[slavkap]

@Luskan777, as @DaanHoogland mentioned cwiki, those two guides could help:
Host HA
High Availability Developer guide

Probably there is only a need for a change here

cloudstack/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java

Line 318 in fadb39e

public boolean isPoolSupportHA() {

to support SharedMountPoint if the kvm heartbeat script works for SharedMountPoint storage

[Luskan777]

Hi @slavkap @DaanHoogland ,

Thanks for your reply, it helps me a lot, I'm going to start developing HA support for ShareMountPoint.

I found an issue with the same problem #9750 , I believe this will solve this issue too.

[Luskan777]

Hello everyone,

With some changes, I managed to add HA support for ShareMountPoint volumes. My changes are in this fork I made.

I tested it and it worked perfectly, however, I believe these commits are not ready for Pull Request, because during development, I identified a limitation in the way Cloudstack handles the ShareMountPoint storage type in Libvirt.

Cloudstack uses the same storage pool format for ShareMountPoint and Filesystem, that is, both StoragePoolTypes use Libvirt Directory Pool.

Therefore, in lower layers of Cloudstack, especially when dealing with Libvirt StoragePools, the ShareMountPoint type becomes the Filesystem type, which is a problem, since the Filesystem storage type is not shared and does not allow HA because it is a local storage.

cloudstack/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

Lines 318 to 324 in 55c8115

	private StoragePool createSharedStoragePool(Connect conn, String uuid, String host, String path) {
	String mountPoint = path;
	if (!_storageLayer.exists(mountPoint)) {
	logger.error(mountPoint + " does not exists. Check local.storage.path in agent.properties.");
	return null;
	}
	LibvirtStoragePoolDef spd = new LibvirtStoragePoolDef(PoolType.DIR, uuid, uuid, host, path, path);

cloudstack/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java

Lines 827 to 828 in 55c8115

	} else if (type == StoragePoolType.SharedMountPoint || type == StoragePoolType.Filesystem) {
	sp = createSharedStoragePool(conn, name, host, path);

To solve this (temporarily), I created some conditions to enable HA on the Filesystem storage, as per the code below:

    @Override
    public boolean isPoolSupportHA() {
        String kvmLocalUuid = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.LOCAL_STORAGE_UUID);
        return type == StoragePoolType.NetworkFilesystem || type == StoragePoolType.Filesystem && !kvmLocalUuid.equals(this.uuid);
    }

To avoid conflicts between what is the Filesystem storage and what is the ShareMountPoint, I created the condition that checks if the storage path is the same as the LOCAL_STORAGE_PATH configuration variable. THIS IS FAR FROM THE BEST SOLUTION, but it solves the problem for now.

As a definitive solution, I think it would be interesting to open a discussion to change the Libvirt StoragePool format used for ShareMountPoint. My recommendation is to change it to the Filesystem pool format. Using this format makes sense for the ShareMountPoint storage type, and I also haven't found any other StoragePoolType that uses this Libvirt storage format.

Since this is a change that I consider relevant, I decided to consult the community before developing it. If the community decides that it makes sense to change the Libvirt storage format from ShareMountPoint, I'll be happy to develop it.

[DaanHoogland]

@Luskan777 , your proposal looks sensible and I doubt anyone would want the old Directory Pool , once this is implemented. We do however have to deal with legacy and I would make it a option (default on for new SharedMountPoints. Other than such backwards compatibility considerations, I'd say; go ahead. (and keep sharing as you go, thanks)

[Luskan777]

@Luskan777 , your proposal looks sensible and I doubt anyone would want the old Directory Pool , once this is implemented. We do however have to deal with legacy and I would make it a option (default on for new SharedMountPoints. Other than such backwards compatibility considerations, I'd say; go ahead. (and keep sharing as you go, thanks)

Perfect, I will work on it, I will keep you posted on my updates