terminal sockets timeouts in cluster

[vosdev]

Hey!

After a long time of debugging I think I have it pinpointed down to the issue and am able to reproduce it.

My terminal connections get a timeout of 60s (the timeout configured in haproxy) when the instance is running on a different cluster member than the lxd server serving the web ui

No errors in my F12 console.

I have a 4 node LXD 6.5 cluster, node[1-4] for simplicity.

I have a keepalived script that chooses the database-leader as cluster member. That node will serve the web UI via haproxy. This is currently node4. haproxy only has the localhost lxd as backend configured.

Instances running on node4 can have their terminal open in the web UI without any issues. But instances running on node1, node2 or node3 timeout after 60s.

Haproxy is the one killing the session. If I change the haproxy timeout to 5 minutes it happens after 5 minutes.

If I migrate the instance to node4 and open a terminal it stays open indefinitely. But when I migrate the instance to node3 again and open a terminal, it dies after the configured timeout in haproxy.

Perhaps similar to canonical/lxd-ui#1193 ?

[vosdev]

To confirm, I've just configured node1 as (the only) haproxy backend on node4's haproxy and now the instances that live on node2/3/4 have their terminal killed after 60s. The VM that lives on node1 still has it's terminal open.

[edlerd]

I tested this on a 3 node microcloud, and can't reproduce it. Connecting to the first node also gives me a terminal for an instance running on node-2 or node-3.

Can you connect to the LXD-UI on node-1 directly, without the proxy?

I also tested it with haproxy in front of the cluster. Below minmal config worked well for me. I could open a terminal for an instance on node-2, where the haproxy config routes all requests to node1:

global
  daemon

defaults
  mode  http
  timeout connect 50000
  timeout client  500000
  timeout server  500000

frontend lxd_frontend
  bind 0.0.0.0:8407 ssl verify optional crt keys/lxd-ui.pem ca-file keys/lxd-ui.crt
  default_backend lxd_core

backend lxd_core
  server lxd_https 10.131.25.10:8443 ssl verify none crt keys/lxd-ui.pem

[vosdev]

That's a timeout of 500 seconds (8 min 20 sec), can you keep the terminal open for 10 minutes? I can open a terminal just fine but keeping the terminal open for longer than the timeout is the problem.

note: setting the timeout to 60s makes testing easier :)

[edlerd]

I see, the problem is the timeout. It is surprising the location of the instance makes a difference here. Doesn't the 60s timeout apply to any instance on any cluster member?

[vosdev]

Well that's the funny part. I can keep terminals open without issues when they run on the LXD instance that haproxy is connected to.

In the previous similar issue we had with console there was no keepalive pings over the websocket. (canonical/lxd-ui#1193 )

I don't know how traffic is routed for terminal sockets internally. Maybe there are keepalive pings for terminal sessions but those are lost when they have to travel across nodes. I think @tomponline can probably enlighten us here

[tomponline]

@edlerd does lxd-ui generate websocket pings, or does it expect LXD to generate them and it just responds back to them?

Possibly what is happening is that when the web socket is forwarded between cluster members that the websocket ping/pongs are not making it all the way to the UI (and thus haproxy is seeing idle timeout).

[edlerd]

Well that's the funny part. I can keep terminals open without issues when they run on the LXD instance that haproxy is connected to.

Interesting, for me, it times out independent on which cluster member the instance is running. But I was testing with LXD 5.21 and the server initiated ping fix is likely only available in the 6 series, which @vosdev is running as mentioned above (fix was in #15491).

Does lxd-ui generate websocket pings, or does it expect LXD to generate them and it just responds back to them?

@tomponline The UI can't create the ping messages, as that is only possible in the browser. Though, the browser will reply to the ping messages a server sends. The browser is not exposing any of this ping protocol to a web application, nor to a JavaScript API that the LXD-UI could hook into.

Possibly what is happening is that when the web socket is forwarded between cluster members that the websocket ping/pongs are not making it all the way to the UI (and thus haproxy is seeing idle timeout).

Yes, the description by @vosdev suggests that the LXD cluster internal forwarding is not including the WebSocket pings.

[edlerd]

I can confirm the behavior with a 2 member cluster running LXD 6.5-ccdfb39 from the 6/stable channel.

I put HAProxy in front connecting to node-1, with all timeouts set to 15 seconds.

Open a terminal in the UI with a container on node 1. Terminal stays open and functional.
Open a terminal in the UI with a container on node 2. Terminal will close connection after 15 seconds.

Swapping HAProxy in front, connecting to node-2 also swaps the observed behavior. The instance terminal on node 2 continues working while the one on node 1 times out after 15 seconds.

Supporting the previous idea of LXD cluster member forward of the WebSocket ping messages is not working as expected.

[vosdev]

And I just tested to confirm that it's not only the terminal but also the graphical console that breaks after the haproxy timeout is hit.