pfSense HaProxy Reverse Proxy and Portainer Agent

[Salvora]

Hello,
I could not decide where to open this issue but I could not get the agent to connect to the portainer instance behind haproxy.
Portainer 2.18.4
Agent 2.18.4

I deployed portainer with ports 9443 and 8000.
Portainer instance has signed certs which work fine.
Reverse proxy works for the portainer works as expected. I can access the UI through the proxy.
However, when I create the environment for the agent and deploy the agent to another docker instance using the reverse proxy URL, the healthcheck shows fine but fails to establish connection when I try to manage it.

I tried following this guide but it might be outdated.
https://portainer-notes.readthedocs.io/en/latest/faq.html

frontend www-http
    bind    *:80
    stats   enable
    mode    http
    option  http-keep-alive

    acl portainer   hdr_end(host)   -i portainer.127.0.0.1.xip.io

    use_backend     portainer       if portainer

backend portainer
    stats   enable
    option  forwardfor
    option  http-keep-alive
    server  portainer    127.0.0.1:9000 check

I get these errors when I try to connect to the portainer host.

023/07/13 00:12:00 client: Connecting to ws://portainer.mydomain.org:8000
2023/07/13 00:12:10 client: Connection error: dial tcp 10.10.10.10:8000: operation was canceled
2023/07/13 00:12:10 client: Give up

10.10.10.10 is the reverse proxy ip.
Agent tries to connect to the port 8000 which is a closed port. Only port 443 is listening on the reverse proxy.

What is wrong here?

[jamescarppe]

Those docs are definitely out of date. We recommend docs.portainer.io as the source of up to date documentation.

Portainer uses port 8000 to provide a secure TCP tunnel for the remote Edge Agent to communicate with the Portainer server, via websockets. This process is described in more detail here. For this to work via HAproxy, you'll need to add a configuration for port 8000 as well to support the websocket communication from the Edge Agent.

[Salvora]

As advised, I've setup a TCP mode reverse proxy for port 8000, created another backend for the portainer host post 8000. It successfully connects, however, after a while the connection closes with this error "Connection error: websocket: close 1006 (abnormal closure): unexpected EOF".
To fix this, I've tried

• increased client timeout to 300000ms on frontend
• used "http-server-close" on frontend
• increased Connection timeout to 300000ms on backend
• increased Server timeout to 300000ms on backend
• tried enabling stick tables for source ipv4.

I've also used a direct IP connection for comparison, and I don't have this issue when agent is connected directly to host by IP instead of reverse proxy.

[jamescarppe]

If it works without HAproxy then the issue is clearly in the HAproxy config, so it's just a matter of figuring out the special sauce to get it working.

This documentation might help - it suggests setting http-server-close on the backend and setting the tunnel timeout to 1 hour.

I don't have a lot of experience with HAproxy myself so I might not be much help, but I'm checking with the wider team to see whether there's anything else we need to be aware of here.

[Salvora]

Tried a bunch of different configs but just couldn't get it to work reliably. Is there a way to use ws on the same port as 9443 instead of 8000. I noticed that using ws on the same port reduces lots of issues. That is being

• You don't need to make arrangements for a separate port
• HAproxy reliably works and can switch between ws and http with almost no config.