doc: production readiness of howto/cluster_vip

[nobuto-m]

First of all, a big round of applause to having a vip / high-availability doc finally! 🎉 🎉 🎉 (#16645)

https://documentation.ubuntu.com/lxd/latest/howto/cluster_vip/

May I request some updates to cover more topics on top, to be more production ready?

No health check

lxd/doc/howto/cluster_vip.md

Lines 38 to 47 in 08814e3

	vrrp_instance VI_1 {
	state MASTER
	interface enp5s0
	virtual_router_id 41
	priority 200
	advert_int 1
	virtual_ipaddress {
	192.0.2.50/24
	}
	}

The current configuration doesn't monitor LXD port so VIP can sit on a node where LXD service may have already crashed in the worst case scenario. It would be good to check the status of the backend service and a vip follows the status of it. i.e. the vip should always run on a node where LXD service is available.

One example is to check the TCP port, and you can see some examples such as in the charmed-keepalived config.

"chk_svc_port" part in:
https://github.com/charmed-kubernetes/charm-keepalived/blob/main/src/templates/keepalived.conf

HAProxy example

Further more, it would be great if it checks the actual API HTTP(S) response from the LXD endpoint instead of relying on the TCP connection only. To do so, it's a good idea to employ HAProxy as mentioned in the following section, which is good.

lxd/doc/howto/cluster_vip.md

Line 158 in 08814e3

Alternatively, consider combining Keepalived with an implementation of [HAProxy](https://www.haproxy.org/). HAProxy is a reverse proxy that can redirect traffic for both TCP and HTTP protocols, which means that it can handle load balancing both API and UI traffic for LXD clusters.

What we are missing here and would like to have is an actual example to setup HAProxy for production usage. One example is to have a sample haproxy.cfg to load-balance the traffic to multiple backend servers with a health check not to redirect traffic to non-functional backend. Also, it would be essential to configure a vip for HAProxy with pacemaker/corosync so that the vip can follow the status of HAProxy so the traffic still works even if one of the 3 HAProxy instances fails.

---

Document: howto/cluster_vip.md

[minaelee]

Thanks! These are great suggestions, and I've added them to the docs backlog.

[vosdev]

Sharing my setup here:

Instead of a tcp check on port 8443, my keepalived healthcheck script checks if the current node is the database-leader:

#!/bin/bash
set -euo pipefail

# local short name must match the cluster server_name
LOCAL=$(hostname -s)

# Query the local LXD API for this member
OUT=$(lxc query "/1.0/cluster/members/$LOCAL" 2>/dev/null || true)
if [ -z "$OUT" ]; then
  # LXD API not responding or not a cluster member
  exit 1
fi

# Ensure node is Online
STATUS=$(echo "$OUT" | jq -r '.status // empty')
if [ "$STATUS" != "Online" ]; then
  exit 1
fi

# Ensure node has the database-leader role
if echo "$OUT" | jq -r '.roles[]?' | grep -qx 'database-leader'; then
  exit 0
else
  exit 1
fi

Keepalived does the health checking instead of haproxy.

I've modified the haproxy example here: https://documentation.ubuntu.com/lxd/latest/authentication/#tls-server-certificate to proxy traffic to only the local LXD instance.

When haproxy receives a request, TLS SNI inspection happens to see if my haproxy has to proxy tcp to LXD, or proxy towards a https listener so that it can then forward the request to Ceph's UI or Ceph's monitoring stack (Prometheus,Grafana,Alertmanager).

I use LXD with a selfmanaged Ceph stack. If Microcloud has also enabled Ceph's UI and it's monitoring stack then my haproxy config is even more relevant :)

Happy to share if you're interested.