Wrong local primary storage is added on host maintenance/restart

[akrasnov-drv]

problem

I enabled local storage for guest VMs.
When I place a host to maintenance, or if I just restart a host (without maintenance), CloudStack creates additional local storage instances with names hostname-local-random
In virsh I see just 1 pool corresponding to local storage but in primary storage in UI I see several.
If I try to start a VM on such a host, CS fails to deploy VM because it does not find local storage.
Another rather strange thing, is that if I put to maintenance wrong storage (intending to remove it), I see in virsh pool name changes to the one specified in disabled (still single local pool but with another name).
It happens also for empty pools and hosts without any VM.

versions

CloudStack 4.20.0.0
Ubuntu 22.04.5 LTS
QEMU 1:6.2+dfsg-2ubuntu6.24
libvirt 8.0.0-1ubuntu7.10
local storage for guests is enabled

The steps to reproduce the bug

1.Enable local storage for guests
2.Add host(s)
3.Ensure local primary storage was created for each host
4.Place host to maintenance / restart
5. After host is back check that there are 2 or more primary storage IDs are in primary storage list for the same host
6. Check that there is only 1 virsh pool corresponding to 1 of primary storage instances
7. Disable some of non-existing pools and see that virsh pool was renamed
...

What to do about it?

To workaround this, after restart/maintenance I have to disable all existing local storage for a host, then enable 1 really existing, and then remove all others.
Cloudstack should not keep/add non-existing local storage. At least it should not affect VM deployment (CS should try to use the proper one)
Adding local.storage.uuid to agent config did not help

[harikrishna-patnala]

@akrasnov-drv may I know what are the values of "local.storage.uuid and local.storage.path" in /etc/cloudstack/agent/agent.properties on the KVM host.

[akrasnov-drv]

I just checked and found no of this 2 in my agent.properties
I don't think I had path, but I believe I had uuid previously automatically added.
Must be it was removed when I removed additional stores from UI.
My cluster works well now, I mean local primary storage is functional and VMs are created, even though I have not storage related vars in agent.properties

[DaanHoogland]

@akrasnov-drv , is there still an issue to address or can we close this?

[harikrishna-patnala]

@akrasnov-drv during my testing while fixing the cases of local storages in github.com//pull/10231 I found that the values of "local.storage.uuid and local.storage.path" in /etc/cloudstack/agent/agent.properties on the KVM host are the source to create corresponding primary storages in CloudStack. This happens whenever the kvm agent is restarted.

In your case I believe since you have removed all the storages except the one you wanted, same is reflecting now in the agent.properties.

If in case you find any issue with your environment, please feel free to reopen this or create new issue with more details needed for investigation. Thanks for all the information.

[akrasnov-drv]

Sorry missed the updates.
I believe I have not removed anything related to local.storage from agent properties, it just initially was not there.
I'm checking now if the issue is solved by fixing local.storage.uuid in agent properties. If yes, it can be considered as a workaround. But as agent creates local storage after it already up, and this uuid is kind of crucial, I'd expect agent to set it after local storage is created.

[akrasnov-drv]

No, it did not help, at least not fully.
I have a cluster with nodes and local storage working.
I added local.storage.uuid= to /etc/cloudstack/agent/agent.properties with uuid of existing local pool and restarted agents.
Then I restarted management and found doubled storage for a server. Existing virsh pool still matches local.storage.uuid=
So I decided to remove the extra one. But when I placed the extra one to maintenance, virsh pool was rename (if not replaced) and does not match local.storage.uuid= anymore.
Here is the relevant output, no other commands was executed in that shell session:

root@host:~# virsh pool-list
 Name                                   State    Autostart
------------------------------------------------------------
 4798c891-2d66-35c8-b0e3-706fedb51483   active   no
 e6b7e1ea-9cfb-409a-8daf-fb7bdc124d2e   active   no

root@host:~# grep storage /etc/cloudstack/agent/agent.properties
local.storage.uuid=e6b7e1ea-9cfb-409a-8daf-fb7bdc124d2e
#### here I placed local storage for 53529f92-5fea-4ad0-b908-f0d95630ff3d  to maintenance
root@host:~# virsh pool-list
 Name                                   State    Autostart
------------------------------------------------------------
 4798c891-2d66-35c8-b0e3-706fedb51483   active   no
 53529f92-5fea-4ad0-b908-f0d95630ff3d   active   no

root@host:~# grep storage /etc/cloudstack/agent/agent.properties
local.storage.uuid=e6b7e1ea-9cfb-409a-8daf-fb7bdc124d2e

[DaanHoogland]

reopening on request by user (see #10763 (comment))

[harikrishna-patnala]

@akrasnov-drv local.storage.uuid should not be empty and as far as I know this has to be entered manually by the operator.

when I checked the document also it says the same https://docs.cloudstack.apache.org/en/latest/adminguide/storage.html#manually-adding-local-storage-pool

In your case, may I ask you to do the following to make it a cleaner slate

1. remove all the local storages from CloudStack
2. make sure you have "local.storage.uuid" and "local.storage.path" in all the hosts
3. Restart cloudstack agent on all the KVM hosts

(you can quickly test the same steps on one of the hosts and repeat the same for other if it works)

I hope this helps.

[akrasnov-drv]

Hi,

• Removed local storage
• stopped cloudstack-agent service
• added local storage path (uuid already was configured)

local.storage.uuid=a21cb1ec-9148-40cc-a54b-0e9f9924a31a
local.storage.path=/var/lib/libvirt/images

• ensured that there is no local pool in virsh
• started cloudstack-agent

And now I really see that the pool is created with proper uuid from agent.properties.

The main 2 differences with what I did before are

• storage path
• no existing pool (previously I tried to save existing pool uuid to agent properties)

Looks like local.storage agent properties are must to have
Why not to add them on the 1st agent start when it re-writes agent.properties anyway?

[akrasnov-drv]

Without path it also works.
So uuid should be configured beforehand. If I add uuid of existing pool and restart agent, it does not work well.