Do timeouts (proxy-read, proxy-send) need to be adjusted for HUB behind a Traefik proxy + websocket?

[luckman212]

Related to Pull Request #25 · henrygd/beszel-docs

and

https://beszel.dev/guide/advanced-deployment#note-on-websocket-timeouts

I have my HUB deployed behind a Traefik proxy. I'm not experiencing the disconnection issue mentioned by @romanzipp in that PR, but was curious when I came across that in the Docs. Does this apply to Traefik as well? If it does, does anyone know how to configure that?

[luckman212]

Tack-on question: is there a wss:// endpoint we can query with tool like wscat in order to test and verify that all is working properly?

When I tried this simple test, I get an "error" (although HTTP 200 == OK ?)

$ wscat -c wss://ip.of.beszel.hub/api
error: Unexpected server response: 200

[romanzipp]

Hey! I'm not using Traefik in my k8s cluster so I can not give a comment on that unfortunately.

[henrygd]

Check your agent logs for disconnection events. It's normal for WebSocket to drop and reconnect every once in a while, but if it's very frequent then there may be an issue w/ the proxy.

I don't use Traefik, but from a quick search it looks like you could do something like this:

labels:
  - "traefik.http.services.my-service.loadbalancer.transport.respondingtimeouts.idleTimeout=360s"
  - "traefik.http.services.my-service.loadbalancer.transport.respondingtimeouts.readTimeout=360s"
  - "traefik.http.services.my-service.loadbalancer.transport.respondingtimeouts.writeTimeout=360s"

This could also help make the SSE connection more stable if you have any issues with data not updating in real time in the web UI.

Testing with a tool like wscat would be difficult because we use a custom handshake that exchanges binary CBOR data. You could do something like this with a valid token to trigger the connection upgrade, but it will likely kick you off after a few seconds because you'd still need to complete the rest of the handshake.

wscat -H "X-Token:c6d2019c-ecea-4538-a237-24c5a8895023" -H "X-Beszel:0.18.2" -c wss://example.com/api/beszel/agent-connect

[luckman212]

Thanks @henrygd

sadly, my testing and research leads me to conclude that Traefik does not support per-service transport timeouts. When I attempted them, the proxy failed to start.

I can (and did) test with a global 360s timeout for the entire HTTPS transport:

  websecure:
    address: ":443"
    asDefault: true
    http:
      tls:
        certResolver: letsencrypt
    transport:
      respondingTimeouts:
        readTimeout: "360s"
        writeTimeout: "360s"
        idleTimeout: "360s"

This still causes my 1m chart not to load. So I don't think this is a timeout issue. I also don't see any disconnection events in my agent logs.

I also tested the websocket connection which seems fine. Both from the Docker host itself:

# websocat -v -H="X-Token:a014dee6-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -H="X-Beszel:0.18.2" ws://172.20.0.2:8090/api/beszel/agent-connect
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::net_peer] Connected to TCP 172.20.0.2:8090
[INFO  websocat::ws_client_peer] Connected to ws
��X@hC���oа�Ha @�7鋵F�@F5�F���{E=��G:��"����T �8&�(u�fK{�
[INFO  websocat::ws_peer] Received WebSocket close message
[INFO  websocat::sessionserve] Reverse finished

As well as from a client outside of the proxy:

$ wscat -H "X-Token:a014dee6-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -H "X-Beszel:0.18.2" --connect wss://beszel.foo.com/api/beszel/agent-connect
Connected (press CTRL+C to quit)
< ��X@hC���oа�Ha @�7鋵F�@F5�F���{E=��G:��"����T �8&�(u�fK{�
Disconnected (code: 1000, reason: "context deadline exceeded")

So somehow it seems this is the fault of the Fail2ban middleware. I will file an issue there.

[luckman212]

Filed tomMoulard/fail2ban#241 to continue troubleshooting this.