diff --git a/docs/src/routes/deployment/+page.md b/docs/src/routes/deployment/+page.md index 1e163c93..30f28d3c 100644 --- a/docs/src/routes/deployment/+page.md +++ b/docs/src/routes/deployment/+page.md @@ -27,6 +27,10 @@ The internal API listener defaults to `4017/tcp` and is not published by the bundled Compose deployment. Root-host traffic reaches it through Portal's own SNI router. +This topology assumes Portal owns public `443/tcp`. If that port already +belongs to something else on the host, see +[Running Behind an Existing Reverse Proxy](#running-behind-an-existing-reverse-proxy). + ## Prerequisites - A public Linux server with Docker and Docker Compose. @@ -99,15 +103,20 @@ also required when wildcard tunnel names terminate TLS at Portal. ```bash mkdir -p ./.portal-certs docker compose pull portal -docker compose up -d --force-recreate --remove-orphans portal +docker compose up -d --force-recreate portal ``` For a local source build: ```bash -docker compose up -d --build --force-recreate --remove-orphans portal +docker compose up -d --build --force-recreate portal ``` +`--remove-orphans` is deliberately absent. It is useful once, to clear the +containers a superseded topology left behind, and dangerous in a project shared +with unrelated services. Remove those containers by name instead — see +[Migration From the Split Stack](#migration-from-the-split-stack). + The Compose stack publishes: | Port | Purpose | @@ -141,6 +150,216 @@ The `/admin` request must return the SPA entry rather than `404`. Registered subdomains must continue to reach their tunnel targets through the same public 443 listener. +## Running Behind an Existing Reverse Proxy + +Portal expects to own public `443/tcp`. On a host that already serves other +sites from that port, it cannot simply be pointed at: Portal's SNI router +**closes any hostname it has no lease for**, so a shared socket would drop +every request meant for those other sites. + +The proxy keeps the port and hands Portal the hostnames that belong to it. +Complete, tested configurations are in +`docs/static/examples/reverse-proxy/`. + +### One topology: nginx in a container beside Portal + +nginx runs as a service on the same Compose network as Portal. Only nginx +publishes host ports. Portal publishes no TCP port at all and is reached as +`portal:443` and `portal:4017` over that network. + +```text +host :443 -> nginx container + :443 stream, ssl_preread, sends PROXY protocol + *.portal.example.com -> :8444 -> strips PROXY -> portal:443 + portal.example.com -> :8444 -> strips PROXY -> portal:443 + anything else -> :8443 -> nginx http, other sites +``` + +Mixing this with a host-loopback port mapping does not work: if Portal also +published `127.0.0.1:8443`, nginx's own listener on that address could not +bind. Pick this topology or a host nginx reaching Portal over published ports — +not both. + +`SNI_PORT` stays `443` inside the container. Portal reaches its own API listener +through its SNI router, and that port is what goes into the ECH `HTTPS` record. + +### Lease hostnames must pass through, unmodified + +Terminating TLS for `*.portal.example.com` breaks tunnels: clients started with +`--ban-mitm` probe for termination and drop a relay that does it, and it +disables keyless TLS and Encrypted Client Hello, both of which need the +handshake itself to reach Portal. + +Two things in an nginx `stream` block break this quietly. + +**The map needs `hostnames;`.** Without it, `map` compares keys as literal +strings and `*.portal.example.com` matches nothing, so every lease hostname +falls through to `default` and is answered by the HTTP terminator instead of +Portal: + +```nginx +map $ssl_preread_server_name $portal_backend { + hostnames; # required for the wildcard to match + *.portal.example.com 127.0.0.1:8444; + portal.example.com 127.0.0.1:8444; + default 127.0.0.1:8443; +} +``` + +**The PROXY header must be stripped before Portal.** `proxy_protocol on` is a +server-level directive, so the `:443` listener sends the header to *every* +destination it selects. Portal does not parse the PROXY protocol: it would read +`PROXY TCP4 ...` where it expects a TLS ClientHello and close the connection. +Send the lease path through a stage that consumes the header first: + +```nginx +server { + listen 127.0.0.1:8444 proxy_protocol; # consumes it + set $portal_sni portal:443; # variable, so it resolves per request + proxy_pass $portal_sni; # no proxy_protocol on: plain TLS onward +} +``` + +### Client addresses, and the trust boundary + +An SNI router forwarding to a local port opens a new connection, so the +terminating listener sees the router rather than the visitor. `proxy_protocol on` +carries the original address, and the http block recovers it: + +```nginx +set_real_ip_from 127.0.0.1; # trust only the loopback hop +real_ip_header proxy_protocol; +``` + +This is what gives the *other sites* on the box their real client addresses. For +Portal itself it only matters if you terminate the root host — see below. + +If you do, **overwrite `X-Forwarded-For` rather than appending to it**: + +```nginx +proxy_set_header X-Forwarded-For $remote_addr; # not $proxy_add_x_forwarded_for +``` + +`$proxy_add_x_forwarded_for` keeps whatever the visitor sent and appends the +peer. Portal trusts the *first* entry, so a request carrying +`X-Forwarded-For: 10.0.0.9` from the Internet arrives as `10.0.0.9, ` and +is read as `10.0.0.9` — an `/api/policy/ips` bypass. `$remote_addr` has already +been restored from the PROXY header, so it is both correct and unspoofable. + +Set `TRUSTED_PROXY_CIDRS` to **the proxy's own address as a `/32`**, not the +default private ranges. The default trusts every RFC 1918 address, which on a +Docker host means every container. + +That address has to be *fixed*. Compose assigns container addresses +dynamically, so a `/32` matching whatever nginx got today stops matching the +next time it is recreated — and the failure is silent: everything still works, +Portal just ignores the forwarded address and starts applying IP bans and rate +limits to nginx instead of to visitors. Give the network its own IPAM and pin +nginx into it: + +```yaml +services: + nginx: + networks: + edge: + ipv4_address: 172.31.240.2 # TRUSTED_PROXY_CIDRS=172.31.240.2/32 + portal: + networks: [edge] + +networks: + edge: + ipam: + config: + - subnet: 172.31.240.0/24 +``` + +Compose's implicit default network does not accept `ipv4_address`, which is why +the network is declared. Check the subnet does not overlap something already on +the host with +`docker network inspect $(docker network ls -q) --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}}{{end}}'`. + +### Terminating the root host conflicts with ECH + +Passing the root host through leaves Portal with no client address at all: it +reads `X-Forwarded-For` and `X-Real-IP` only, and a pass-through carries no HTTP +layer to put them in. Terminating it recovers that, but check one thing first. + +When a DNS provider is configured, Portal publishes an `HTTPS` record carrying +`ech=` for its own hostname and installs the matching key **only on its own API +listener**. An nginx terminator has neither, so ECH-capable clients that read +the record can fail the connection before any request arrives. + +`SyncECHConfig` is a no-op when no DNS provider is configured, so: + +| `ACME_DNS_PROVIDER` | Root host | +|---|---| +| set (managed issuance) | pass through — terminating breaks the ECH it advertises | +| empty (manual certificates) | may be terminated, which is what recovers client addresses | + +Pass-through is the default in the example for that reason. + +### Publishing Portal's ports + +```yaml +services: + portal: + ports: !override + - "${WIREGUARD_PORT:-51820}:${WIREGUARD_PORT:-51820}/udp" +``` + +`!override` replaces the base `ports` list rather than appending to it; without +it Compose merges both and still tries to bind `443`. It requires Docker Compose +2.24.4 or newer. + +Because it *replaces*, this list must carry **every mapping the deployment had +enabled**. The bundled file publishes TCP 443 and the WireGuard port and +comments out three more — `443/udp` for QUIC backhaul, the `MIN_PORT`–`MAX_PORT` +UDP range, and the same range for raw TCP leases. Anything left out here stops +being published, silently, and tunnels that used it stop working. + +### Sharing a Compose project with unrelated services + +A host like this usually runs Portal alongside services that have nothing to do +with it. Compose commands operate on the whole project by default, so name the +service explicitly every time: + +```bash +docker compose up -d portal # not: docker compose up -d +docker compose stop portal # not: docker compose down +``` + +Never pass `--remove-orphans` on a shared project. It deletes every container +in the project that the current file does not define, which includes services +that belong to other stacks. + +### Verifying without a regression hunt + +Record how the host answers **before** changing anything, so that an error found +afterwards can be attributed rather than investigated: + +```bash +# One line per host, with a path its clients actually use. +cat > probe.txt <<'PROBE' +portal.example.com /api/healthz +other-site.example / +PROBE + +probe() { + while read -r host path; do + [ -z "$host" ] && continue + printf '%-32s %-16s %s\n' "$host" "$path" \ + "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 10 "https://$host$path")" + done < probe.txt +} + +probe > baseline.txt +``` + +Compare with `probe | diff baseline.txt -` afterwards. Judge by the difference, +not by whether a code looks healthy: `/` is not a valid request for every host, +and a WebSocket-only endpoint answers a plain `GET` with nothing at all, which a +proxy correctly reports as `502`. + ## Automated Updates Production deployments should follow the v2 release track: @@ -159,11 +378,37 @@ chmod +x watch_and_deploy.sh ## Migration From the Split Stack -1. Confirm `./.portal-certs` contains the Portal certificate and state. +1. Confirm `./.portal-certs` contains the Portal certificate and state. It holds + the relay identity: losing it makes this a different relay. 2. Pull or build the new single Portal image. 3. Stop the old stack so it releases public port 443. -4. Start `portal` with `--remove-orphans` to remove the old edge and frontend - containers. +4. Remove the superseded `nginx`, `portal-api` and `portal-frontend` services, + then start `portal`. + + Do this through the **old** Compose file. Those services did not set + `container_name`, so their containers are named `-portal-api-1` + rather than `portal-api`, and `docker stop portal-api` fails with + `No such container`. Keep the old file until this step is done: + + ```bash + docker compose -f docker-compose.old.yml ps # confirm the real names + docker compose -f docker-compose.old.yml stop nginx portal-api portal-frontend + docker compose -f docker-compose.old.yml rm -f nginx portal-api portal-frontend + docker compose up -d portal + ``` + + If the old file is already gone, resolve the names through Compose's own + labels instead of guessing: + + ```bash + docker ps -a --filter label=com.docker.compose.service=portal-api \ + --format '{{.Names}}' + ``` + + Not `--remove-orphans`: it deletes every container in the project that the + current file does not define, including services belonging to other stacks + when the project is shared. + 5. Verify the SPA, relay APIs, and at least one wildcard tunnel. The old edge configuration and its separate browser certificate are no longer @@ -174,13 +419,25 @@ for root-host requests. ### Port 443 Is Already Allocated -Stop the previous edge container or host service before starting Portal: +Identify what holds the port first: ```bash -docker compose down --remove-orphans +sudo ss -tlnp | grep ':443\b' +``` + +If it is a superseded Portal edge container, stop and remove it by name, then +start Portal: + +```bash +docker stop +docker rm docker compose up -d portal ``` +If it is something that has to keep serving — an nginx fronting other sites, for +example — Portal cannot take the port from it and must not try. See +[Running Behind an Existing Reverse Proxy](#running-behind-an-existing-reverse-proxy). + ### SPA Routes Return 404 When `PORTAL_FRONTEND_DIR` is empty, use an image built after the embedded diff --git a/docs/static/examples/reverse-proxy/compose.override.yaml b/docs/static/examples/reverse-proxy/compose.override.yaml new file mode 100644 index 00000000..0eb5909b --- /dev/null +++ b/docs/static/examples/reverse-proxy/compose.override.yaml @@ -0,0 +1,80 @@ +# Port arrangement for running Portal behind a containerised nginx. +# +# Save as docker-compose.override.yml next to the bundled docker-compose.yml. +# Compose loads that name automatically. +# +# Only nginx publishes host ports. Portal keeps its TCP listeners on the Compose +# network, where nginx reaches them as portal:443 and portal:4017, so nothing +# competes for :443 on the host and no loopback mapping is needed. +# +# SNI_PORT stays 443 inside the container. Portal reaches its own API listener +# through its SNI router, and that port is what goes into the ECH HTTPS record. + +services: + nginx: + image: nginx:1.27-alpine + ports: + - "80:80" + - "443:443" + networks: + edge: + # Only needed when the root host is terminated and Portal is therefore + # configured with TRUST_PROXY_HEADERS=true. Compose assigns container + # addresses dynamically, so TRUSTED_PROXY_CIDRS pointing at whatever + # nginx happened to get would stop matching the next time the container + # is recreated. Portal would then ignore the forwarded address and apply + # IP bans and rate limits to nginx instead of to visitors -- silently, + # because everything still works. + # + # Set TRUSTED_PROXY_CIDRS=172.31.240.2/32 to match. + ipv4_address: 172.31.240.2 + volumes: + - ./nginx.conf:/etc/nginx/nginx.conf:ro + - ./sites-enabled:/etc/nginx/sites-enabled:ro + # The relay's own ACME writes fullchain.pem and privatekey.pem under + # IDENTITY_PATH, which the base file binds from ./.portal-certs. nginx + # reads them and never writes. Mounting a separate ./certs would leave it + # looking at an empty directory and failing to start with + # "cannot load certificate". + # + # Only the terminate-the-root-host variant needs this; the default + # pass-through config has no ssl listener of its own. + - ./.portal-certs:/etc/nginx/certs:ro + - acme-webroot:/var/www/acme + restart: unless-stopped + + portal: + # !override replaces the base ports list instead of appending to it. + # Without it Compose merges both entries and still tries to bind :443. + # Requires Docker Compose 2.24.4 or newer; check with `docker compose version`. + # + # This list must carry EVERY mapping the deployment had enabled. The base + # file publishes TCP 443 and the WireGuard port, and comments out three more + # that a deployment may have turned on. Replacing the list drops anything + # left out, silently, and existing tunnels stop working. + ports: !override + - "${WIREGUARD_PORT:-51820}:${WIREGUARD_PORT:-51820}/udp" + # TCP 443 is deliberately absent: nginx owns it, and Portal is reached + # over the Compose network instead. + # + # Uncomment whichever of these the deployment already had enabled. + # - "443:443/udp" + # - "${MIN_PORT:-40000}-${MAX_PORT:-40009}:${MIN_PORT:-40000}-${MAX_PORT:-40009}/udp" + # - "${MIN_PORT:-40000}-${MAX_PORT:-40009}:${MIN_PORT:-40000}-${MAX_PORT:-40009}" + # - "${PPROF_PORT:-6060}:${PPROF_PORT:-6060}" + networks: [edge] + +volumes: + acme-webroot: {} + +networks: + # A named network with its own IPAM, because Compose's implicit default + # network does not accept ipv4_address. Both services join it, so nginx still + # reaches portal:443 and portal:4017 by name. + # + # Check that this subnet does not overlap anything already on the host: + # docker network inspect $(docker network ls -q) --format '{{.Name}} {{range .IPAM.Config}}{{.Subnet}}{{end}}' + edge: + ipam: + config: + - subnet: 172.31.240.0/24 diff --git a/docs/static/examples/reverse-proxy/nginx.conf b/docs/static/examples/reverse-proxy/nginx.conf new file mode 100644 index 00000000..f4513b17 --- /dev/null +++ b/docs/static/examples/reverse-proxy/nginx.conf @@ -0,0 +1,208 @@ +# Portal behind an nginx that already owns :443. +# +# Portal cannot share that port. Its SNI router closes any hostname it has no +# lease for, so putting it on a socket that also serves other sites would drop +# every request meant for them. nginx keeps the port and hands Portal the +# hostnames that belong to it. +# +# TOPOLOGY: nginx runs as a container beside Portal on the same Compose network. +# Only nginx publishes host ports; Portal publishes no TCP port at all and is +# reached as `portal:443` / `portal:4017` over that network. The 127.0.0.1 +# addresses below are inside this container and never appear on the host. See +# compose.override.yaml beside this file. +# +# Substitute portal.example.com with PORTAL_URL's host. Nothing else needs +# editing for the default (pass-through) mode. + +events { + worker_connections 4096; +} + +stream { + # Docker's embedded DNS. Upstreams are reached through variables plus a + # resolver rather than static upstream blocks: a static name is resolved + # once at boot and cached, so nginx would refuse to start while Portal is + # down and would hold a stale address after the container is recreated. + resolver 127.0.0.11 ipv6=off valid=10s; + + map $ssl_preread_server_name $portal_backend { + # REQUIRED. Without it a `map` compares keys as literal strings and + # `*.portal.example.com` never matches anything, so every lease + # hostname silently falls through to `default`. + hostnames; + + # Lease hostnames pass through untouched. Terminating TLS for them + # breaks tunnels: clients started with --ban-mitm probe for exactly + # that and drop the relay when they find it, and it disables keyless + # TLS and Encrypted Client Hello, which need the handshake to reach + # Portal. + *.portal.example.com 127.0.0.1:8444; + + # The relay's own hostname. Pass-through by default -- see "TERMINATING + # THE ROOT HOST" at the bottom before changing this to 127.0.0.1:8443. + portal.example.com 127.0.0.1:8444; + + # Everything else this box already served, terminated by the http block. + default 127.0.0.1:8443; + } + + server { + listen 443; + ssl_preread on; + proxy_pass $portal_backend; + + proxy_socket_keepalive on; + proxy_connect_timeout 5s; + # Tunnels are long-lived. The 10 minute default cuts idle sessions. + proxy_timeout 86400s; + + # Carry the client address across the loopback hop. Both destinations + # below consume this header; Portal never sees it. + proxy_protocol on; + } + + # Portal's SNI listener, reached through a stage that strips the PROXY + # header first. + # + # This stage is not optional. proxy_protocol is a server-level directive, + # so the listener above sends the header to every destination it selects. + # Portal does not parse the PROXY protocol: it would read "PROXY TCP4 ..." + # where it expects a TLS ClientHello and close the connection, breaking + # every wildcard tunnel. `listen ... proxy_protocol` consumes the header + # here, and the absence of `proxy_protocol on` in this server means plain + # TLS goes on to Portal. + server { + listen 127.0.0.1:8444 proxy_protocol; + + # Through a variable, for the same reason as the resolver above: a + # literal `proxy_pass portal:443` is resolved once at boot, so nginx + # refuses to start while Portal is down. + set $portal_sni portal:443; + proxy_pass $portal_sni; + + proxy_socket_keepalive on; + proxy_connect_timeout 5s; + proxy_timeout 86400s; + } +} + +http { + resolver 127.0.0.11 ipv6=off valid=10s; + + # Recover the real client address from the PROXY header. Trust only the + # loopback hop, which is the stream listener in this same container. + set_real_ip_from 127.0.0.1; + real_ip_header proxy_protocol; + + include /etc/nginx/mime.types; + default_type application/octet-stream; + + map $http_upgrade $connection_upgrade { + default upgrade; + '' close; + } + + # Other sites this box serves. proxy_protocol is a listen-socket option, so + # every server block on 8443 receives the header and $remote_addr is the + # real client for all of them. + include /etc/nginx/sites-enabled/*.conf; + + server { + listen 80; + server_name _; + + # ACME HTTP-01 for any domain terminated by this edge. Non-wildcard + # names need no DNS API credentials, so this works with any registrar. + location /.well-known/acme-challenge/ { + root /var/www/acme; + } + + location / { + return 301 https://$host$request_uri; + } + } +} + +# ───────────────────────────────────────────────────────────────────────────── +# TERMINATING THE ROOT HOST +# +# Passing the root host through leaves Portal with no client address: it reads +# X-Forwarded-For and X-Real-IP only, and a pass-through carries no HTTP layer +# to put them in. /api/policy/ips and rate limiting therefore see the tunnel +# ingress rather than the visitor. +# +# Terminating the root host here recovers that, at a cost that has to be checked +# first: +# +# Portal publishes an HTTPS record carrying `ech=` for its own hostname +# whenever a DNS provider is configured, and installs the matching key only on +# its own API listener (portal/server.go prepareAPITLS). nginx has neither. +# ECH-capable clients that read the record can then fail the connection before +# any request arrives. SyncECHConfig is a no-op without a DNS provider, so +# terminating here is only safe when ACME_DNS_PROVIDER is empty and the +# certificate is supplied manually. +# +# If that holds, point the root host at the http listener: +# +# portal.example.com 127.0.0.1:8443; +# +# and add this server block. Note X-Forwarded-For: $remote_addr, not +# $proxy_add_x_forwarded_for. This is a public trust boundary, and the appending +# form keeps whatever the visitor sent. Portal trusts the FIRST entry, so +# "X-Forwarded-For: 10.0.0.9" from the Internet would arrive as +# "10.0.0.9, " and be read as 10.0.0.9 -- an /api/policy/ips bypass. +# $remote_addr has already been restored from the PROXY header above. +# +# server { +# listen 8443 ssl proxy_protocol; +# server_name portal.example.com; +# server_tokens off; +# +# ssl_certificate /etc/nginx/certs/fullchain.pem; +# ssl_certificate_key /etc/nginx/certs/privatekey.pem; +# ssl_protocols TLSv1.2 TLSv1.3; +# +# # Portal's API listener speaks TLS with the relay certificate, which +# # is not the browser-facing one, so it is not verified here. +# set $portal https://portal:4017; +# +# location = /sdk/connect { +# proxy_pass $portal; +# proxy_ssl_verify off; +# proxy_ssl_server_name on; +# proxy_set_header Host $host; +# proxy_set_header X-Real-IP $remote_addr; +# proxy_set_header X-Forwarded-For $remote_addr; +# proxy_set_header X-Forwarded-Proto https; +# +# proxy_http_version 1.1; +# proxy_set_header Upgrade $http_upgrade; +# # $connection_upgrade, not $http_connection: a client may send +# # "keep-alive, Upgrade" and forwarding that verbatim fails the +# # upgrade. +# proxy_set_header Connection $connection_upgrade; +# +# proxy_buffering off; +# proxy_request_buffering off; +# proxy_read_timeout 86400s; +# proxy_send_timeout 86400s; +# } +# +# location / { +# proxy_pass $portal; +# proxy_ssl_verify off; +# proxy_ssl_server_name on; +# proxy_set_header Host $host; +# proxy_set_header X-Real-IP $remote_addr; +# proxy_set_header X-Forwarded-For $remote_addr; +# proxy_set_header X-Forwarded-Proto https; +# } +# } +# +# Then set TRUST_PROXY_HEADERS=true and TRUSTED_PROXY_CIDRS=172.31.240.2/32, +# the fixed address compose.override.yaml pins this container to. Not the +# default private ranges, which would trust every container on the host, and +# not whatever address Compose happens to assign: that changes when the +# container is recreated, and Portal would then quietly attribute every visitor +# to nginx. +# ─────────────────────────────────────────────────────────────────────────────