Multiple `libvirt-provider` instances share same storage

[lukas016]

Problem description

Current implementation of store uses filesystem for storage api.Machine objects. We are using mutex for avoid of race condition during modified api.Machine. But we don't have any protection between running multiple instances of libvirt-provider.

It can create race condition during upgrade of libvirt-provider, when multiple instances of libvirt-provider run simultaneously.

I think we need to fix this special case. One of possible solution is use lock base on file (same logic as flock). Maybe: https://pkg.go.dev/github.com/theckman/go-flock

What do you think?

[lukas016]

we decided only one instance of pod can run at the time.

[lukasfrank]

There should be never a second instance of the libvirt-provider on one host. Rather fixing this issue by locking the store files I would propose to use the kubernetes way to do so.
@lukas016 @balpert89 Do you set leder-elect in your landscape?
This should already solve your issue in a proper way.

ref: https://github.com/ironcore-dev/ironcore/blob/main/poollet/machinepoollet/cmd/machinepoollet/app/app.go#L111