Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 173 additions & 7 deletions docs/src/routes/deployment/+page.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ The internal API listener defaults to `4017/tcp` and is not published by the
bundled Compose deployment. Root-host traffic reaches it through Portal's own
SNI router.

This topology assumes Portal owns public `443/tcp`. If that port already
belongs to something else on the host, see
[Running Behind an Existing Reverse Proxy](#running-behind-an-existing-reverse-proxy).

## Prerequisites

- A public Linux server with Docker and Docker Compose.
Expand Down Expand Up @@ -95,15 +99,20 @@ also required when wildcard tunnel names terminate TLS at Portal.
```bash
mkdir -p ./.portal-certs
docker compose pull portal
docker compose up -d --force-recreate --remove-orphans portal
docker compose up -d --force-recreate portal
```

For a local source build:

```bash
docker compose up -d --build --force-recreate --remove-orphans portal
docker compose up -d --build --force-recreate portal
```

`--remove-orphans` is deliberately absent. It is useful once, to clear the
containers a superseded topology left behind, and dangerous in a project shared
with unrelated services. Remove those containers by name instead — see
[Migration From the Split Stack](#migration-from-the-split-stack).

The Compose stack publishes:

| Port | Purpose |
Expand Down Expand Up @@ -136,6 +145,141 @@ The `/admin` request must return the SPA entry rather than `404`. Registered
subdomains must continue to reach their tunnel targets through the same public
443 listener.

## Running Behind an Existing Reverse Proxy

Portal expects to own public `443/tcp`. On a host that already serves other
sites from that port, it cannot simply be pointed at: Portal's SNI router
**closes any hostname it has no lease for**, so a shared socket would drop
every request meant for those other sites.

The proxy keeps the port and hands Portal the hostnames that belong to it. How
it hands them over is not uniform, and the two halves are split for opposite
reasons.

### Split by SNI, not by path

```text
:443 -> SNI inspection
portal.example.com -> terminate TLS, proxy to Portal's API listener
*.portal.example.com -> pass through untouched to Portal's SNI listener
anything else -> whatever the host already served
```

**Lease hostnames must pass through.** Terminating TLS for them breaks tunnels:
clients started with `--ban-mitm` probe for termination and drop a relay that
does it. It also disables keyless TLS and Encrypted Client Hello, both of which
need the handshake itself to reach Portal.

**The root host should be terminated.** Portal derives the client address from
`X-Forwarded-For` and `X-Real-IP` only; it does not speak the PROXY protocol.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Account for Portal's root ECH record before terminating this TLS connection

With a managed DNS provider, prepareAPITLS publishes an HTTPS ech= record for PORTAL_URL and installs the corresponding derived ECH key only on Portal's API TLS listener. This nginx terminator has neither that key nor an ECH configuration, so ECH-capable clients that consume the advertised record can reject the connection before any HTTP request reaches Portal. Passing wildcard leases through preserves tenant ECH, but it does not preserve the relay root's ECH. Please either keep root TLS on Portal, add a proxy mode that stops advertising root ECH, or document/provide a front end that can terminate it with the same material.

Passing the root host through raw is simpler, but then every visitor reaches
Portal as the proxy's own address, `TRUST_PROXY_HEADERS` has nothing to read,
and IP policy under `/api/policy/ips` matches everyone or no one.

### Client addresses across the loopback hop

An SNI router that forwards to a local port opens a new connection, so the
terminating listener sees the router rather than the visitor. Carry the
original address explicitly:

```nginx
stream {
server {
listen 443;
ssl_preread on;
proxy_pass $portal_backend;
proxy_protocol on; # adds the PROXY header
}
}

http {
set_real_ip_from 127.0.0.1; # trust only the loopback hop
real_ip_header proxy_protocol; # recover the address from it
}
```

Then set `TRUST_PROXY_HEADERS=true`. Leave `TRUSTED_PROXY_CIDRS` empty to trust
the default private and loopback ranges, which covers container networks.

`proxy_protocol` on a `listen` directive is a socket option, so every server
block on that port receives the header. Other sites keep a plain `listen ... ssl`
and still see real client addresses.

A complete, tested configuration is at
`docs/static/examples/reverse-proxy/nginx.conf`. HAProxy expresses the same
split with `tcp-request inspect-delay` plus `req_ssl_sni` ACLs, and Traefik with
a TCP router using `HostSNI` rules and TLS passthrough.

### Publishing Portal's ports

With a proxy in front, Portal should not publish `443/tcp` on the host. Bind
its SNI listener to loopback or leave it on the container network, and keep the
UDP ports published:

```yaml
services:
portal:
ports: !override
- "127.0.0.1:8443:443"
- "${WIREGUARD_PORT:-51820}:${WIREGUARD_PORT:-51820}/udp"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve every enabled transport mapping in the replacement list

!override replaces the entire base ports list, but this example restores only the remapped TCP SNI listener and WireGuard UDP. Operators using the documented optional 443/udp QUIC backhaul or the MIN_PORTMAX_PORT UDP/raw-TCP lease mappings will silently drop those publications and their existing tunnels will stop working. Please include the optional mappings here or explicitly require copying every enabled mapping into this replacement list.

```

Do not change `SNI_PORT`. It is fixed at 443 because Portal reaches its own API
listener through its SNI router, and because that port is published in the ECH
`HTTPS` record. Only the host-side mapping moves.

`!override` replaces the ports list rather than appending to it; without it
Compose merges both entries and still tries to bind `443`. It requires Docker
Compose 2.24.4 or newer.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

### Sharing a Compose project with unrelated services

A host like this usually runs Portal alongside services that have nothing to do
with it. Compose commands operate on the whole project by default, so name the
service explicitly every time:

```bash
docker compose up -d portal # not: docker compose up -d
docker compose stop portal # not: docker compose down
```

Never pass `--remove-orphans` on a shared project. It deletes every container
in the project that the current file does not define, which includes services
that belong to other stacks. Remove superseded containers by name instead:

```bash
docker stop portal-api portal-frontend
docker rm portal-api portal-frontend
```

### Verifying without a regression hunt

Record how the host answers **before** changing anything, so that an error found
afterwards can be attributed rather than investigated:

```bash
# One line per host, with a path its clients actually use.
cat > probe.txt <<'PROBE'
portal.example.com /api/healthz
other-site.example /
PROBE

probe() {
while read -r host path; do
[ -z "$host" ] && continue
printf '%-32s %-16s %s\n' "$host" "$path" \
"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 10 "https://$host$path")"
done < probe.txt
}

probe > baseline.txt
```

Compare with `probe | diff baseline.txt -` afterwards. Judge by the difference,
not by whether a code looks healthy: `/` is not a valid request for every host,
and a WebSocket-only endpoint answers a plain `GET` with nothing at all, which a
proxy correctly reports as `502`.

## Automated Updates

Production deployments should follow the v2 release track:
Expand All @@ -154,11 +298,21 @@ chmod +x watch_and_deploy.sh

## Migration From the Split Stack

1. Confirm `./.portal-certs` contains the Portal certificate and state.
1. Confirm `./.portal-certs` contains the Portal certificate and state. It holds
the relay identity: losing it makes this a different relay.
2. Pull or build the new single Portal image.
3. Stop the old stack so it releases public port 443.
4. Start `portal` with `--remove-orphans` to remove the old edge and frontend
containers.
4. Remove the superseded edge and frontend containers, then start `portal`.
Name them rather than reaching for `--remove-orphans`, which deletes every
container in the project that the current file does not define — including
services that belong to other stacks when the project is shared:

```bash
docker stop portal-api portal-frontend
docker rm portal-api portal-frontend

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Target the actual Compose containers during migration

The superseded Compose stack did not set container_name, so its containers are normally named <project>-portal-api-1 and <project>-portal-frontend-1; docker stop portal-api portal-frontend therefore fails with No such container. The old stack also had an nginx edge service, which this removal list omits. Please retain/reference the old Compose file and use docker compose -f <old-file> stop/rm for nginx, portal-api, and portal-frontend, or resolve their real names through Compose labels before removing them.

docker compose up -d portal
```

5. Verify the SPA, relay APIs, and at least one wildcard tunnel.

The old edge configuration and its separate browser certificate are no longer
Expand All @@ -169,13 +323,25 @@ for root-host requests.

### Port 443 Is Already Allocated

Stop the previous edge container or host service before starting Portal:
Identify what holds the port first:

```bash
docker compose down --remove-orphans
sudo ss -tlnp | grep ':443\b'
```

If it is a superseded Portal edge container, stop and remove it by name, then
start Portal:

```bash
docker stop <old-edge-container>
docker rm <old-edge-container>
docker compose up -d portal
```

If it is something that has to keep serving — an nginx fronting other sites, for
example — Portal cannot take the port from it and must not try. See
[Running Behind an Existing Reverse Proxy](#running-behind-an-existing-reverse-proxy).

### SPA Routes Return 404

When `PORTAL_FRONTEND_DIR` is empty, use an image built after the embedded
Expand Down
139 changes: 139 additions & 0 deletions docs/static/examples/reverse-proxy/nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Portal behind an nginx that already owns :443.
#
# Portal cannot share that port. Its SNI router closes any hostname it has no
# lease for, so putting it on a socket that also serves other sites would drop
# every request meant for them. nginx keeps the port and hands Portal the
# hostnames that belong to it.
#
# Substitute:
# portal.example.com PORTAL_URL's host
# portal:443 Portal's SNI listener
# portal:4017 Portal's API listener
# 127.0.0.1:8443 the loopback port this file's http block listens on
#
# The two halves are split for opposite reasons. Read the "Running Behind an
# Existing Reverse Proxy" section of the deployment guide before editing either.

events {
worker_connections 4096;
}

stream {
# Docker's embedded DNS. Use whatever resolver the platform provides.
#
# Upstreams are reached through variables plus a resolver rather than static
# upstream blocks: a static name is resolved once at boot and cached, so
# nginx would refuse to start while Portal is down and would hold a stale
# address after the container is recreated.
resolver 127.0.0.11 ipv6=off valid=10s;

map $ssl_preread_server_name $portal_backend {
# The relay's own hostname is terminated by the http block below, which
# is what lets the client address survive. Portal reads that address
# from X-Forwarded-For and does not speak the PROXY protocol, so a raw
# passthrough here would make every visitor look like this proxy.
portal.example.com 127.0.0.1:8443;

# Lease hostnames pass through byte for byte. Terminating TLS here
# breaks tunnels: clients running with --ban-mitm probe for exactly
# that and drop the relay when they find it. It also disables keyless
# TLS and ECH, both of which need the handshake to reach Portal.
*.portal.example.com portal:443;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*.portal.example.com is treated as a literal string here because this map does not enable hostname masks. Add hostnames; at the top of the block (or use an explicit regex such as ~^.+\.portal\.example\.com$). As written, every real lease hostname falls through to the default HTTP terminator instead of reaching Portal.


# Everything else this box already served.
default 127.0.0.1:8443;
}

server {
listen 443;
ssl_preread on;
proxy_pass $portal_backend;

proxy_socket_keepalive on;
proxy_connect_timeout 5s;
# Tunnels are long-lived. The 10 minute default cuts idle sessions.
proxy_timeout 86400s;

# Carries the client address across the loopback hop. Without it every
# request reaches the http block as 127.0.0.1, which makes
# TRUST_PROXY_HEADERS pointless and IP policy match every visitor
# identically.
proxy_protocol on;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This directive applies to every upstream selected by this stream server, including portal:443. Portal does not parse PROXY protocol, so its ClientHello inspection receives PROXY ... before the TLS record and closes wildcard tunnel connections. Please split this into stages: the public listener may send PROXY protocol, but the wildcard path needs an intermediate stream listener with listen ... proxy_protocol that consumes the header and then proxies plain TLS to Portal. The root HTTP listener can continue consuming the header for real-IP recovery.

}
}

http {
resolver 127.0.0.11 ipv6=off valid=10s;

# Recover the real address from the PROXY header. Trust only the loopback
# hop to supply it.
set_real_ip_from 127.0.0.1;
real_ip_header proxy_protocol;

include /etc/nginx/mime.types;
default_type application/octet-stream;

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

server {
# proxy_protocol is a listen-socket option, so it applies to every
# server block on this port. Other sites on this box keep a plain
# `listen 8443 ssl` and still receive real client addresses.
listen 8443 ssl proxy_protocol;
server_name portal.example.com;
server_tokens off;

# Portal's own ACME issues this certificate under IDENTITY_PATH.
ssl_certificate /etc/nginx/certs/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/privatekey.pem;
ssl_protocols TLSv1.2 TLSv1.3;

# Portal's API listener speaks TLS using the relay certificate, which
# is not the browser-facing one, so it is not verified here.
set $portal https://portal:4017;

# The tunnel control plane: long-lived and bidirectional, so buffering
# and the 60s default read timeout both have to go.
location = /sdk/connect {
proxy_pass $portal;

proxy_ssl_verify off;
proxy_ssl_server_name on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Overwrite X-Forwarded-For at this public trust boundary

$proxy_add_x_forwarded_for preserves a header supplied by the Internet client. Portal's ExtractClientIP then trusts the first X-Forwarded-For entry whenever nginx's source address is trusted, so a request with X-Forwarded-For: <allowed-ip> becomes <allowed-ip>, <actual-ip> and can bypass /api/policy/ips. Since $remote_addr has already been restored from the PROXY header here, set X-Forwarded-For $remote_addr (in both locations), or otherwise discard the inbound header before rebuilding it.

proxy_set_header X-Forwarded-Proto https;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
# $connection_upgrade, not $http_connection: a client may send
# "keep-alive, Upgrade", and forwarding that verbatim fails the
# upgrade. That is what the map above exists for.
proxy_set_header Connection $connection_upgrade;

proxy_buffering off;
proxy_request_buffering off;
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
}

# Everything else Portal owns: the embedded dashboard, /api, the rest
# of /sdk, /discovery and /v1/sign.
location / {
proxy_pass $portal;

proxy_ssl_verify off;
proxy_ssl_server_name on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
}

# Other sites this box serves. They keep a plain `listen 8443 ssl`.
include /etc/nginx/sites-enabled/*.conf;
}
Loading