-
Notifications
You must be signed in to change notification settings - Fork 29
docs: cover deploying behind a reverse proxy that already owns 443 #296
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
3bfb2e0
b37814e
b02e35f
a86b261
980d3f3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,6 +27,10 @@ The internal API listener defaults to `4017/tcp` and is not published by the | |
| bundled Compose deployment. Root-host traffic reaches it through Portal's own | ||
| SNI router. | ||
|
|
||
| This topology assumes Portal owns public `443/tcp`. If that port already | ||
| belongs to something else on the host, see | ||
| [Running Behind an Existing Reverse Proxy](#running-behind-an-existing-reverse-proxy). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - A public Linux server with Docker and Docker Compose. | ||
|
|
@@ -95,15 +99,20 @@ also required when wildcard tunnel names terminate TLS at Portal. | |
| ```bash | ||
| mkdir -p ./.portal-certs | ||
| docker compose pull portal | ||
| docker compose up -d --force-recreate --remove-orphans portal | ||
| docker compose up -d --force-recreate portal | ||
| ``` | ||
|
|
||
| For a local source build: | ||
|
|
||
| ```bash | ||
| docker compose up -d --build --force-recreate --remove-orphans portal | ||
| docker compose up -d --build --force-recreate portal | ||
| ``` | ||
|
|
||
| `--remove-orphans` is deliberately absent. It is useful once, to clear the | ||
| containers a superseded topology left behind, and dangerous in a project shared | ||
| with unrelated services. Remove those containers by name instead — see | ||
| [Migration From the Split Stack](#migration-from-the-split-stack). | ||
|
|
||
| The Compose stack publishes: | ||
|
|
||
| | Port | Purpose | | ||
|
|
@@ -136,6 +145,141 @@ The `/admin` request must return the SPA entry rather than `404`. Registered | |
| subdomains must continue to reach their tunnel targets through the same public | ||
| 443 listener. | ||
|
|
||
| ## Running Behind an Existing Reverse Proxy | ||
|
|
||
| Portal expects to own public `443/tcp`. On a host that already serves other | ||
| sites from that port, it cannot simply be pointed at: Portal's SNI router | ||
| **closes any hostname it has no lease for**, so a shared socket would drop | ||
| every request meant for those other sites. | ||
|
|
||
| The proxy keeps the port and hands Portal the hostnames that belong to it. How | ||
| it hands them over is not uniform, and the two halves are split for opposite | ||
| reasons. | ||
|
|
||
| ### Split by SNI, not by path | ||
|
|
||
| ```text | ||
| :443 -> SNI inspection | ||
| portal.example.com -> terminate TLS, proxy to Portal's API listener | ||
| *.portal.example.com -> pass through untouched to Portal's SNI listener | ||
| anything else -> whatever the host already served | ||
| ``` | ||
|
|
||
| **Lease hostnames must pass through.** Terminating TLS for them breaks tunnels: | ||
| clients started with `--ban-mitm` probe for termination and drop a relay that | ||
| does it. It also disables keyless TLS and Encrypted Client Hello, both of which | ||
| need the handshake itself to reach Portal. | ||
|
|
||
| **The root host should be terminated.** Portal derives the client address from | ||
| `X-Forwarded-For` and `X-Real-IP` only; it does not speak the PROXY protocol. | ||
| Passing the root host through raw is simpler, but then every visitor reaches | ||
| Portal as the proxy's own address, `TRUST_PROXY_HEADERS` has nothing to read, | ||
| and IP policy under `/api/policy/ips` matches everyone or no one. | ||
|
|
||
| ### Client addresses across the loopback hop | ||
|
|
||
| An SNI router that forwards to a local port opens a new connection, so the | ||
| terminating listener sees the router rather than the visitor. Carry the | ||
| original address explicitly: | ||
|
|
||
| ```nginx | ||
| stream { | ||
| server { | ||
| listen 443; | ||
| ssl_preread on; | ||
| proxy_pass $portal_backend; | ||
| proxy_protocol on; # adds the PROXY header | ||
| } | ||
| } | ||
|
|
||
| http { | ||
| set_real_ip_from 127.0.0.1; # trust only the loopback hop | ||
| real_ip_header proxy_protocol; # recover the address from it | ||
| } | ||
| ``` | ||
|
|
||
| Then set `TRUST_PROXY_HEADERS=true`. Leave `TRUSTED_PROXY_CIDRS` empty to trust | ||
| the default private and loopback ranges, which covers container networks. | ||
|
|
||
| `proxy_protocol` on a `listen` directive is a socket option, so every server | ||
| block on that port receives the header. Other sites keep a plain `listen ... ssl` | ||
| and still see real client addresses. | ||
|
|
||
| A complete, tested configuration is at | ||
| `docs/static/examples/reverse-proxy/nginx.conf`. HAProxy expresses the same | ||
| split with `tcp-request inspect-delay` plus `req_ssl_sni` ACLs, and Traefik with | ||
| a TCP router using `HostSNI` rules and TLS passthrough. | ||
|
|
||
| ### Publishing Portal's ports | ||
|
|
||
| With a proxy in front, Portal should not publish `443/tcp` on the host. Bind | ||
| its SNI listener to loopback or leave it on the container network, and keep the | ||
| UDP ports published: | ||
|
|
||
| ```yaml | ||
| services: | ||
| portal: | ||
| ports: !override | ||
| - "127.0.0.1:8443:443" | ||
| - "${WIREGUARD_PORT:-51820}:${WIREGUARD_PORT:-51820}/udp" | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Preserve every enabled transport mapping in the replacement list
|
||
| ``` | ||
|
|
||
| Do not change `SNI_PORT`. It is fixed at 443 because Portal reaches its own API | ||
| listener through its SNI router, and because that port is published in the ECH | ||
| `HTTPS` record. Only the host-side mapping moves. | ||
|
|
||
| `!override` replaces the ports list rather than appending to it; without it | ||
| Compose merges both entries and still tries to bind `443`. It requires Docker | ||
| Compose 2.24.4 or newer. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Sharing a Compose project with unrelated services | ||
|
|
||
| A host like this usually runs Portal alongside services that have nothing to do | ||
| with it. Compose commands operate on the whole project by default, so name the | ||
| service explicitly every time: | ||
|
|
||
| ```bash | ||
| docker compose up -d portal # not: docker compose up -d | ||
| docker compose stop portal # not: docker compose down | ||
| ``` | ||
|
|
||
| Never pass `--remove-orphans` on a shared project. It deletes every container | ||
| in the project that the current file does not define, which includes services | ||
| that belong to other stacks. Remove superseded containers by name instead: | ||
|
|
||
| ```bash | ||
| docker stop portal-api portal-frontend | ||
| docker rm portal-api portal-frontend | ||
| ``` | ||
|
|
||
| ### Verifying without a regression hunt | ||
|
|
||
| Record how the host answers **before** changing anything, so that an error found | ||
| afterwards can be attributed rather than investigated: | ||
|
|
||
| ```bash | ||
| # One line per host, with a path its clients actually use. | ||
| cat > probe.txt <<'PROBE' | ||
| portal.example.com /api/healthz | ||
| other-site.example / | ||
| PROBE | ||
|
|
||
| probe() { | ||
| while read -r host path; do | ||
| [ -z "$host" ] && continue | ||
| printf '%-32s %-16s %s\n' "$host" "$path" \ | ||
| "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 10 "https://$host$path")" | ||
| done < probe.txt | ||
| } | ||
|
|
||
| probe > baseline.txt | ||
| ``` | ||
|
|
||
| Compare with `probe | diff baseline.txt -` afterwards. Judge by the difference, | ||
| not by whether a code looks healthy: `/` is not a valid request for every host, | ||
| and a WebSocket-only endpoint answers a plain `GET` with nothing at all, which a | ||
| proxy correctly reports as `502`. | ||
|
|
||
| ## Automated Updates | ||
|
|
||
| Production deployments should follow the v2 release track: | ||
|
|
@@ -154,11 +298,21 @@ chmod +x watch_and_deploy.sh | |
|
|
||
| ## Migration From the Split Stack | ||
|
|
||
| 1. Confirm `./.portal-certs` contains the Portal certificate and state. | ||
| 1. Confirm `./.portal-certs` contains the Portal certificate and state. It holds | ||
| the relay identity: losing it makes this a different relay. | ||
| 2. Pull or build the new single Portal image. | ||
| 3. Stop the old stack so it releases public port 443. | ||
| 4. Start `portal` with `--remove-orphans` to remove the old edge and frontend | ||
| containers. | ||
| 4. Remove the superseded edge and frontend containers, then start `portal`. | ||
| Name them rather than reaching for `--remove-orphans`, which deletes every | ||
| container in the project that the current file does not define — including | ||
| services that belong to other stacks when the project is shared: | ||
|
|
||
| ```bash | ||
| docker stop portal-api portal-frontend | ||
| docker rm portal-api portal-frontend | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Target the actual Compose containers during migration The superseded Compose stack did not set |
||
| docker compose up -d portal | ||
| ``` | ||
|
|
||
| 5. Verify the SPA, relay APIs, and at least one wildcard tunnel. | ||
|
|
||
| The old edge configuration and its separate browser certificate are no longer | ||
|
|
@@ -169,13 +323,25 @@ for root-host requests. | |
|
|
||
| ### Port 443 Is Already Allocated | ||
|
|
||
| Stop the previous edge container or host service before starting Portal: | ||
| Identify what holds the port first: | ||
|
|
||
| ```bash | ||
| docker compose down --remove-orphans | ||
| sudo ss -tlnp | grep ':443\b' | ||
| ``` | ||
|
|
||
| If it is a superseded Portal edge container, stop and remove it by name, then | ||
| start Portal: | ||
|
|
||
| ```bash | ||
| docker stop <old-edge-container> | ||
| docker rm <old-edge-container> | ||
| docker compose up -d portal | ||
| ``` | ||
|
|
||
| If it is something that has to keep serving — an nginx fronting other sites, for | ||
| example — Portal cannot take the port from it and must not try. See | ||
| [Running Behind an Existing Reverse Proxy](#running-behind-an-existing-reverse-proxy). | ||
|
|
||
| ### SPA Routes Return 404 | ||
|
|
||
| When `PORTAL_FRONTEND_DIR` is empty, use an image built after the embedded | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| # Portal behind an nginx that already owns :443. | ||
| # | ||
| # Portal cannot share that port. Its SNI router closes any hostname it has no | ||
| # lease for, so putting it on a socket that also serves other sites would drop | ||
| # every request meant for them. nginx keeps the port and hands Portal the | ||
| # hostnames that belong to it. | ||
| # | ||
| # Substitute: | ||
| # portal.example.com PORTAL_URL's host | ||
| # portal:443 Portal's SNI listener | ||
| # portal:4017 Portal's API listener | ||
| # 127.0.0.1:8443 the loopback port this file's http block listens on | ||
| # | ||
| # The two halves are split for opposite reasons. Read the "Running Behind an | ||
| # Existing Reverse Proxy" section of the deployment guide before editing either. | ||
|
|
||
| events { | ||
| worker_connections 4096; | ||
| } | ||
|
|
||
| stream { | ||
| # Docker's embedded DNS. Use whatever resolver the platform provides. | ||
| # | ||
| # Upstreams are reached through variables plus a resolver rather than static | ||
| # upstream blocks: a static name is resolved once at boot and cached, so | ||
| # nginx would refuse to start while Portal is down and would hold a stale | ||
| # address after the container is recreated. | ||
| resolver 127.0.0.11 ipv6=off valid=10s; | ||
|
|
||
| map $ssl_preread_server_name $portal_backend { | ||
| # The relay's own hostname is terminated by the http block below, which | ||
| # is what lets the client address survive. Portal reads that address | ||
| # from X-Forwarded-For and does not speak the PROXY protocol, so a raw | ||
| # passthrough here would make every visitor look like this proxy. | ||
| portal.example.com 127.0.0.1:8443; | ||
|
|
||
| # Lease hostnames pass through byte for byte. Terminating TLS here | ||
| # breaks tunnels: clients running with --ban-mitm probe for exactly | ||
| # that and drop the relay when they find it. It also disables keyless | ||
| # TLS and ECH, both of which need the handshake to reach Portal. | ||
| *.portal.example.com portal:443; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| # Everything else this box already served. | ||
| default 127.0.0.1:8443; | ||
| } | ||
|
|
||
| server { | ||
| listen 443; | ||
| ssl_preread on; | ||
| proxy_pass $portal_backend; | ||
|
|
||
| proxy_socket_keepalive on; | ||
| proxy_connect_timeout 5s; | ||
| # Tunnels are long-lived. The 10 minute default cuts idle sessions. | ||
| proxy_timeout 86400s; | ||
|
|
||
| # Carries the client address across the loopback hop. Without it every | ||
| # request reaches the http block as 127.0.0.1, which makes | ||
| # TRUST_PROXY_HEADERS pointless and IP policy match every visitor | ||
| # identically. | ||
| proxy_protocol on; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This directive applies to every upstream selected by this |
||
| } | ||
| } | ||
|
|
||
| http { | ||
| resolver 127.0.0.11 ipv6=off valid=10s; | ||
|
|
||
| # Recover the real address from the PROXY header. Trust only the loopback | ||
| # hop to supply it. | ||
| set_real_ip_from 127.0.0.1; | ||
| real_ip_header proxy_protocol; | ||
|
|
||
| include /etc/nginx/mime.types; | ||
| default_type application/octet-stream; | ||
|
|
||
| map $http_upgrade $connection_upgrade { | ||
| default upgrade; | ||
| '' close; | ||
| } | ||
|
|
||
| server { | ||
| # proxy_protocol is a listen-socket option, so it applies to every | ||
| # server block on this port. Other sites on this box keep a plain | ||
| # `listen 8443 ssl` and still receive real client addresses. | ||
| listen 8443 ssl proxy_protocol; | ||
| server_name portal.example.com; | ||
| server_tokens off; | ||
|
|
||
| # Portal's own ACME issues this certificate under IDENTITY_PATH. | ||
| ssl_certificate /etc/nginx/certs/fullchain.pem; | ||
| ssl_certificate_key /etc/nginx/certs/privatekey.pem; | ||
| ssl_protocols TLSv1.2 TLSv1.3; | ||
|
|
||
| # Portal's API listener speaks TLS using the relay certificate, which | ||
| # is not the browser-facing one, so it is not verified here. | ||
| set $portal https://portal:4017; | ||
|
|
||
| # The tunnel control plane: long-lived and bidirectional, so buffering | ||
| # and the 60s default read timeout both have to go. | ||
| location = /sdk/connect { | ||
| proxy_pass $portal; | ||
|
|
||
| proxy_ssl_verify off; | ||
| proxy_ssl_server_name on; | ||
| proxy_set_header Host $host; | ||
| proxy_set_header X-Real-IP $remote_addr; | ||
| proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Overwrite
|
||
| proxy_set_header X-Forwarded-Proto https; | ||
|
|
||
| proxy_http_version 1.1; | ||
| proxy_set_header Upgrade $http_upgrade; | ||
| # $connection_upgrade, not $http_connection: a client may send | ||
| # "keep-alive, Upgrade", and forwarding that verbatim fails the | ||
| # upgrade. That is what the map above exists for. | ||
| proxy_set_header Connection $connection_upgrade; | ||
|
|
||
| proxy_buffering off; | ||
| proxy_request_buffering off; | ||
| proxy_read_timeout 86400s; | ||
| proxy_send_timeout 86400s; | ||
| } | ||
|
|
||
| # Everything else Portal owns: the embedded dashboard, /api, the rest | ||
| # of /sdk, /discovery and /v1/sign. | ||
| location / { | ||
| proxy_pass $portal; | ||
|
|
||
| proxy_ssl_verify off; | ||
| proxy_ssl_server_name on; | ||
| proxy_set_header Host $host; | ||
| proxy_set_header X-Real-IP $remote_addr; | ||
| proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
| proxy_set_header X-Forwarded-Proto https; | ||
| } | ||
| } | ||
|
|
||
| # Other sites this box serves. They keep a plain `listen 8443 ssl`. | ||
| include /etc/nginx/sites-enabled/*.conf; | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Account for Portal's root ECH record before terminating this TLS connection
With a managed DNS provider,
prepareAPITLSpublishes an HTTPSech=record forPORTAL_URLand installs the corresponding derived ECH key only on Portal's API TLS listener. This nginx terminator has neither that key nor an ECH configuration, so ECH-capable clients that consume the advertised record can reject the connection before any HTTP request reaches Portal. Passing wildcard leases through preserves tenant ECH, but it does not preserve the relay root's ECH. Please either keep root TLS on Portal, add a proxy mode that stops advertising root ECH, or document/provide a front end that can terminate it with the same material.