[bug]: StartTunnel 1.2.1 — port forwarding is impossible on a NAT'd VPS: subnet set-wan is uninvokable and upgraded installs are left with wanIp: null

[DritzPS]

Prerequisites

• I have searched for existing issues that already report this problem.

Server Hardware

Lenovo M920q 32gb RAM 2 TB M.2

StartOS Version

0.4.0

Client OS

MacOS

Client OS Version

15.7.7

Browser

Firefox

Browser Version

Version 152.0.5 (64-bit)

Current Behavior

Current Behavior

On a VPS behind 1:1 NAT (where the public IPv4 is assigned at the provider edge and not bound to any local interface—the typical AWS, Oracle, GCP, or Azure model), no new port forwards can be created via the CLI or web UI after upgrading. Both setup paths fail identically:

• CLI:

  $ sudo start-tunnel port-forward add --label "LND" 9735 10.59.203.3:9735
  Network Error: no WAN IP available for device 10.59.203.3

• Web UI (Add published port, IPv4):

  RPC ERROR: Network Error no WAN IP available for device 10.59.203.3
  

Two independent defects combine to make this state entirely unrecoverable for the end user:

Defect A: The subnet wanIp is null after migration

A db dump reveals that wanIp is explicitly null on the gateway, the subnet, and every downstream device. This occurs even though get-available-ips actively returns a perfectly usable local interface IP (172.16.0.12).

While all relevant port-forward schema migrations successfully executed (PortForwardAuto, PortForwardEntry, PortForwardKind, ReclaimPort80Redirect, WgClientKind), none of them backfilled the new wanIp field. Port forwards that existed prior to the upgrade continue to function normally because they carry the external IP embedded directly in their entry keys, allowing their DNAT rules to remain present and pass traffic. As a result, only new additions fail.

Defect B: subnet … set-wan is uninvokable

The subnet ... set-wan CLI administrative command—which is intended to manually correct a missing wanIp—is broken at the argument-parsing level. Its own --help page emits a duplicate <SUBNET> positional argument:

Usage: start-tunnel subnet <SUBNET> set-wan [OPTIONS] <SUBNET>

Because both the parent subnet <SUBNET> argument and the nested set-wan <SUBNET> argument attempt to deserialize into the exact same database key structure, any physical command invocation fails:

• Providing the subnet once:

  error: the following required arguments were not provided: <SUBNET>
  

• Providing the subnet twice:

  General error: error: Serialization Error: duplicate key: subnet
  

Consequently, there is no valid structural syntax to invoke this command.

---

Absence of Workaround

Direct manual patching via db apply is entirely blocked. The utility expects an undocumented, jq-like expression format (where the parser enforces tokens like def / if / ( / [ / .. / { / - / .).

Neither a broad full-document replacement nor a hyper-targeted JSON-pointer operation will validate against the internal structural model, consistently resulting in schema rejection messages:

missing field 'webserver': result does not match database model

As it stands, end users hosting on a NAT'd cloud VPS have zero supported paths to manually or programmatically set wanIp.

Expected Behavior

Proposed Solutions & Fixes

1. Fix CLI Argument Inheritance for set-wan

The set-wan command must correctly inherit the <SUBNET> positional argument from its parent context without requiring a second, duplicate positional argument at the tail.

• Target Syntax: subnet <SUBNET> set-wan --wan-ip <IP>
• Expected Action: Inherit the <SUBNET> parameter and successfully update the subnet's WAN IP.

2. Backfill wanIp via Migration

The database upgrade migration routine must be updated to prevent existing installations from entering an unusable state.

• Detection Logic: Automatically scan and extract the wanIp from the active network interface.
• Fallback Logic: If interface detection is blocked, extract and backfill the wanIp using the embedded IP addresses found inside legacy port-forward entries.

3. Implement Fallback Resolution for Port Forwarding

Both the CLI port-forward add command and the Web UI Add published port dialog must support NAT'd VPS setups.

• Dynamic Lookup: If the wanIp state field evaluates to null, the application must query get-available-ips to dynamically resolve it.
• Validation Rule: Prevent hard-failing with a network error when the necessary IP configuration data is already discoverable on the system.

Steps to Reproduce

Steps to Reproduce

1. Confirm Environment and Available Interfaces

Run StartTunnel 1.2.1 on a Debian 13 VPS whose public IPv4 is NAT'd and not present on the NIC (only a private IP on the interface).

$ ip -4 addr show
2: ens3: ... inet 172.16.0.12/20 ... scope global dynamic ens3

$ sudo start-tunnel web get-available-ips
[ "172.16.0.12", "<global-IPv6>", "<link-local-IPv6>", "127.0.0.1", "::1" ]

2. Attempt to Add Port Forward via CLI

Try to add a port forward using the command-line interface.

$ sudo start-tunnel port-forward add --label "LND" 9735 10.59.203.3:9735

Verbatim Error:

Network Error: no WAN IP available for device 10.59.203.3

3. Attempt to Add Port Forward via Web UI

Navigate to Port Forwards → Add published port and enter the details:

• Protocol: IPv4
• External Port: 9735
• Internal Port: 9735
• Target: Server device

Verbatim Error:

RPC ERROR: Network Error no WAN IP available for device 10.59.203.3

4. Attempt to Set Subnet WAN IP (Both Forms Fail)

Try to manually set the subnet WAN IP so the forward can resolve. Both syntax forms fail to invoke properly.

Form A (Missing argument error):

$ sudo start-tunnel subnet 10.59.203.1/24 set-wan --wan-ip 172.16.0.12

Verbatim Error:

error: the following required arguments were not provided:
  <SUBNET>

Form B (Duplicate key error):

$ sudo start-tunnel subnet 10.59.203.1/24 set-wan --wan-ip 172.16.0.12 10.59.203.1/24

Verbatim Error:

General error: error: Serialization Error: duplicate key: subnet

Anything else?

System Diagnostics & Impact Assessment

📋 Full Environment Profile

• Component: StartTunnel 1.2.1
• Update Verification:

  {"status":"up-to-date","installed":"1.2.1","candidate":"1.2.1"}

• Installation Protocol: Upgraded from legacy 0.x/1.0.x line via:

  curl -sSL https://start9.com/start-tunnel/install.sh | sh

• Network Topology: 1:1 NAT Cloud VPS (Public IPv4 at provider edge; local interface ens3 bound to private IP 172.16.0.12/20). Matches standard documentation models for AWS EC2, Oracle Cloud, GCP, and Azure.
• Subnet Layout: One active subnet (10.59.203.1/24) containing one StartOS server device (10.59.203.3) and three downstream clients.

---

🗄️ Database Evidence (db dump Analysis)

A redacted look at the post-migration database state shows exactly where state validation collapses. Existing entries remain functional due to inline legacy keys, while the new wanIp parent tracking keys remain explicitly empty:

{
  "gateways": { 
    "ens3": { 
      "ipInfo": { "wanIp": null } 
    } 
  },
  "wg": { 
    "subnets": { 
      "10.59.203.1/24": {
        "clients": {
          "10.59.203.3": { "kind": "server", "allowAutoPortForward": true, "wanIp": null }
          // All other sub-connected clients are similarly set to "wanIp": null
        },
        "wanIp": null  // <-- CRITICAL BLOCK: Gating condition for all new port-forward adds
      } 
    } 
  },
  "portForwards": {
    // Grandfathered entries carry the external IP inline; DNAT mechanics process normally
    "172.16.0.12:443":  { "target": "10.59.203.2:443",  "kind": "dnat", "enabled": true  },  
    "172.16.0.12:8333": { "target": "10.59.203.3:8333", "kind": "dnat", "enabled": false }
  }
}

---

💥 Severity & Scope of Impact

Port forwarding configuration is fundamentally broken across all virtual private servers operating behind an edge NAT layout (including AWS, Azure, GCP, and Oracle Cloud).

Because pre-existing DNAT rules continue to pass traffic perfectly, the underlying translation layer remains healthy. The failure is strictly administrative: the software blocks new additions when wanIp is null, yet provides no valid command sequence to populate it.

---

🛠️ Recommended Code Fixes

1. Fix CLI Argument Inheritance: Remove the redundant trailing <SUBNET> positional argument requirement from the set-wan command structure. Allow it to natively inherit context from the parent subnet command space.
2. CLI Sub-Command Audit: Audit sister sub-commands (specifically set-dns and set-ipv6) to ensure this duplicate positional parameter defect is not replicated across other network configuration modules.
3. Enhance Migration Routine: Update database migration logic to automatically backfill the wanIp data property using either the active interface IP or values parsed from legacy pre-upgrade port-forward records.
4. Implement Port Forwarding Fallbacks: Modify the validation check within the port-forward add sequence. If wanIp resolves to null, fallback to query get-available-ips dynamically instead of executing an immediate hard failure.

[helix-nine]

Root-caused both defects. The CLI half is fixed in #3570. The wanIp half is not a migration problem, and the proposed backfill would make this host worse rather than better — details below, since the diagnosis in the report is close but points at the wrong mechanism.

Defect B (the CLI) — confirmed, and it is two commands

subnet_api is a ParentHandler<C, SubnetParams> (tunnel/api.rs:188), so <SUBNET> belongs to the parent subnet command. add, remove and set-dns register with .with_inherited(...); set-wan and set-ipv6 did not and re-declared subnet themselves, so both positionals land in one usage line. One subnet fails clap's required-arg check; two fail combine() with duplicate key: subnet. There is no valid syntax, exactly as reported. set-dns is fine, set-ipv6 is not — the audit you asked for found one more, not zero.

device set-wan is unaffected: device_api has no inherited params.

Fixed in #3570 (start-tunnel 1.2.2). No RPC change, so the web UI is unaffected.

Defect A — wanIp: null is not upgrade damage

gateways.*.ipInfo.wanIp is null on every StartTunnel install, fresh or upgraded, on every topology. TunnelContext::init fills gateways from crate::net::utils::load_ip_info() (tunnel/context.rs:242), which parses ip -o addr show and sets only name/scope_id/subnets/device_type. The code that discovers a WAN IP — UPnP GetExternalIPAddress, then an echoip fetch (net/gateway.rs:2318-2421) — lives in StartOS's NetworkInterfaceController, which start-tunneld never constructs. So no migration failed to backfill it; nothing has ever written it here.

What actually gates the add is default_wan (tunnel/forward/igd.rs:414-437), which falls back to the first local address passing is_wan_candidate (net/port_map/upnp.rs:86-94) — and that rejects RFC1918, so 172.16.0.12 is skipped. On an ordinary VPS the public IPv4 is configured on the NIC, so it is simultaneously local, bindable and globally routable, and the single-field design works by coincidence.

This is a 1.1.0 regression, not 1.2.1. Before #3306, port-forward add took the external socket from the operator (AddPortForwardParams { source: SocketAddrV4, ... }) — which is precisely how the surviving 172.16.0.12:443 entries were created. #3306 replaced source with a server-side derivation that cannot express "the address packets actually arrive on".

Two related failures on this topology, both unreported: PCP MAP from an attached StartOS box returns CANNOT_PROVIDE_EXTERNAL and UPnP GetExternalIPAddress faults, so automatic port forwarding is dead too; and 1.2.1's ReclaimPort80Redirect migration deletes every forward covering port 80 unconditionally, while the replacement redirect only binds addresses passing the same public-address test — so any 172.16.0.12:80 forward was removed on upgrade with nothing to replace it and no way to re-add it.

Why backfilling wanIp does not fix this

wanIp is one field serving three consumers that need different addresses on a 1:1-NAT host:

consumer	code	needs
DNAT match, forward db key, conntrack flush selector	build/lib/scripts/forward-port:85,88,92-93 via api.rs:1253 → net/forward.rs:1082	the address that arrives (172.16.0.12)
SNI demux listener bind	tunnel/forward/sni.rs:285	a bindable address
egress SNAT source	tunnel/context.rs:571-573	a local address
WireGuard client-config Endpoint	api.rs:1167-1198 → client.conf.template	the public address
PCP/UPnP external address, UI "External IP"	net/port_map/server/{mod,igd}.rs, published-ports table	the public address

Note show_config reads client.wanIp then subnet.wanIp first, above every other rung. So the two correct answers are mutually exclusive and live in the same field.

These were measured, not reasoned — four ip netns with a 1:1-NAT edge, running the verbatim rule text forward-port emits with counters attached:

• Public IP as wanIp: the packet arrives with daddr=172.16.0.12; the DNAT rule keyed on the public address records 0 matches. Not an error, not a drop — a non-match, with no diagnostic anywhere.
• Public IP as wanIp, egress: nft ... snat to <public> inserts with rc=0 on a host that does not hold the address. With a stateless provider edge the packet leaves, the server replies, the reply reaches the NIC — and conntrack recorded the reply tuple after SNAT (expecting dst=public) while the edge delivers dst=private, so it never matches and is discarded. With a stateful edge the reply never comes back at all. Today wanIp is null everywhere, so egress is masquerade and works; writing a value is what creates the SNAT rule. A backfill would take a working masqueraded subnet and blackhole it.
• Binding: nothing fails with EADDRNOTAVAIL. bind_tokio_listener sets IP_FREEBIND unconditionally (net/utils.rs:46-50), so the SNI listener binds a wholly non-local address, logs SNI demux listening on ..., and accepts nothing forever.
• Private IP as wanIp: DNAT matches, rules are byte-identical to the grandfathered ones, SNI binds for real. Cost: every client config generated afterwards gets Endpoint = 172.16.0.12.

A backfill migration also cannot work mechanically: run_migrations is context.rs:240 and the wholesale ipInfo overwrite is context.rs:244 — same init, seconds later — and default_wan re-filters whatever it finds through is_wan_candidate anyway. And set_subnet_wan/set_device_wan call resync_forward_keys, so pinning the public IP would re-key the working 172.16.0.12:443 forwards onto an address that never matches, breaking what still works.

Every failure mode here is silent. That is the strongest argument for validating the value where it is set: set_subnet_wan and set_device_wan currently accept any Ipv4Addr with no checks at all.

Proposed fix

1. CLI — fix(start-tunnel): let subnet set-wan/set-ipv6 inherit <SUBNET> #3570, ready.
2. Split the two roles. external_ipv4 becomes two resolvers:
   • an anchor for the inbound/kernel path (DNAT key, SNI bind, SNAT, forward re-keying): pinned value → first local address passing is_wan_candidate → first local non-loopback, non-wg0 IPv4 even if RFC1918. Two-pass, so a public address always wins where one exists; the private fallback only ever engages where the add fails outright today. This alone unblocks 1:1-NAT VPSes and home-router installs, and needs no db change.
   • an advertised address for the outward-facing path (client-config endpoint, PCP/UPnP external address, UI columns), which keeps today's stricter resolution rather than silently advertising an RFC1918 address.
3. Give the advertised address a home — an operator-settable public IP, optionally seeded by an echoip probe. get_wan_ipv4 (net/gateway.rs:1390) is a context-free free function over (iface, url, local_ipv4), so the tunnel can call it directly; the UPnP half is not reusable, since the tunnel is the IGD. With that field present, a NAT'd VPS is fully supported: rules attach to the private address, users are told the public one.
4. Validate at set-time — reject a wanIp that is not on a local interface, since every wrong value fails silently.
5. UI — the WAN dropdown is built from the same public-address filter (components/wan.ts:29-35), so it is empty on this host and offers only "System default". It has to follow whatever the model becomes.

Separately worth deciding: ctx.net_iface is a snapshot taken once at daemon start and never refreshed, so an address added to the NIC needs a restart — the IPv4 docs say otherwise.

Workaround on 1.2.1 today, no upgrade needed

device set-wan parses correctly (it is not affected by the parse bug), and the pinned value is read without any candidate filter:

start-tunnel db dump --pointer /wg/subnets     # confirm the subnet key and device IP
start-tunnel device set-wan 10.59.203.1/24 10.59.203.3 --wan-ip 172.16.0.12
start-tunnel port-forward add --label "LND" 9735 10.59.203.3:9735
start-tunnel db dump --pointer /portForwards   # expect the key 172.16.0.12:9735

This produces exactly the rules your working 172.16.0.12:443 forward already has. It also pins egress for that device's /32 to 172.16.0.12 (identical to the masquerade it replaces on a single-NIC host) and sets that device's client-config Endpoint, which on your host already resolves to the webserver listen address.

db apply is also viable, and explains the error you hit: it evaluates the expression over the whole document and re-parses the result as a complete TunnelDatabase, so any expression that narrows the document fails on the first field with no serde default. A root-returning assignment is fine:

start-tunnel db apply '.wg.subnets["10.59.203.1/24"].wanIp = "172.16.0.12"'

[DritzPS]

Workaround Confirmation & Diagnostic Insights

1. Workaround Verification: device set-wan

The device set-wan command successfully bypasses the broken subnet set-wan CLI interface on a production 1:1-NAT VPS (single NIC, private IP 172.16.0.12/20, public IPv4 at the provider edge) running StartTunnel 1.2.1 — a real host that was stuck exactly as described, not netns.

Because device_api does not use inherited parameters, it avoids the <SUBNET> parsing collision entirely:

sudo start-tunnel device set-wan 10.59.203.1/24 10.59.203.3 --wan-ip 172.16.0.12

Results:

• Unblocks configuration: restores the ability to execute port-forward add immediately — no upgrade needed.
• Schema integrity: generated portForwards entries are structurally identical to legacy pre-feat(net): automatic gateway configuration — port forwarding (PCP/NAT-PMP/UPnP) + private-domain DNS (RFC 2136) #3306 records.
• External reachability: inbound TCP traffic hitting the public IPv4 on the forwarded port maps straight through the DNAT chain.

---

2. Silent Outage Risk: ReclaimPort80Redirect

One production data point on ReclaimPort80Redirect, since silent failure was the concern you flagged — this routine is a critical, delayed failure vector for hosts relying on HTTP-01 ACME challenges.

• The problem: the upgrade deleted the existing :80 forward without binding a replacement mechanism.
• The impact: on hosts using certbot's nginx authenticator (which serves the HTTP-01 challenge on :80 in-place), this deletion silently pre-arms a Let's Encrypt renewal failure. Because the host's existing TLS certificate is still valid at upgrade time, the system appears healthy — the outage then triggers unannounced during the next renewal cycle (~40 days later).
• Remediation: re-adding the :80 forward via the same device set-wan path fully restores functionality, and a subsequent certbot renew --dry-run passes end-to-end through the forward.

⚠️ Warning: For any NAT'd host using an HTTP-01 authenticator bound to port 80, this migration behaves as a delayed, silent production outage rather than a benign cleanup of a missing redirect.

---

3. Review of the Proposed Architecture Fix

Observations from a single-NIC environment confirm that egress and endpoint side effects match your predictions and introduce no negative network behavior.

• Dual-resolver model: decoupling into a kernel-path anchor (bound to the arrival address) and a separately-advertised public address is the correct approach.
• Set-time validation: rejecting a non-local wanIp at input time is essential — because misconfigured values fail silently in this topology, validation is what prevents unlogged routing blackholes.

Appreciate the depth on the root cause; the #3306 attribution and netns measurements made this straightforward to act on.

[helix-nine]

Adversarial pass over my own proposal above. Three of its points were wrong or under-specified, and the sharpened version changes what should ship.

Correction to my table. The egress SNAT rule does not go through external_ipv4 at all — resync_egress reads cfg.wan_ip straight from the DB (tunnel/context.rs:571, :585). So SNAT is a separate consumer that a resolver split does not touch, and it wants the arrival address in both worlds: on 1:1 NAT the correct source is the local address the provider's outbound map expects, and on a multi-public-IP VPS the chosen public address is also local. That leaves the advertised address with no dataplane effect at all — its only consumers are the client-config Endpoint, the PCP/UPnP external address, and UI columns. The change is smaller than "split wanIp in two".

Do not naively relax is_wan_candidate. default_wan skips only loopback and wg0 (tunnel/forward/igd.rs:417-423) and iterates a name-sorted map, so admitting RFC1918 makes docker0 (172.17.0.1) or virbr0 (192.168.122.1) the resolved "WAN" ahead of the real uplink on any host running Docker or libvirt. net/gateway.rs:2314-2318 already excludes Bridge from WAN probing; default_wan is the outlier. Any relaxation has to be two-pass — public across all interfaces first, private only as a last resort — and exclude bridges. A single-pass relaxation silently regresses every dual-homed host that works today.

And it must not reach the PCP/IGD reply path. The tunnel is the port-control gateway for downstream StartOS boxes, and the same resolver answers them (net/port_map/server/igd.rs:199, server/mod.rs:338). A downstream UPnP client discards a private answer (net/port_map/upnp.rs:104-105), but the PCP path returns it unfiltered (net/port_map/client.rs:328-337, :361) and net/gateway.rs:652-675 short-circuits the reachability probe once auto-forwarding has succeeded — so the box would report open_externally: true at an RFC1918 address. That is a worse failure than today's honest refusal.

Revised shape for the model change. Not two new fields — that would revert #3415, which deliberately removed the caller-supplied wan_addr from ShowConfigParams and made the endpoint follow client.wanIp/subnet.wanIp. Instead: keep wanIp with arrival/bind semantics, which is what the dataplane already treats it as, and add one additive #[serde(default)] advertised field that falls back to wanIp when unset. That is key-preserving on every existing install — a NAT'd host's key is already the private address, a normal VPS's is already the public one — so resync_forward_keys produces an empty diff and nobody's forwards move. No migration is needed, but the #[serde(default)] is load-bearing: run_migrations re-deserializes the whole document every boot (tunnel/migrations/mod.rs:38-42), so a field without one stops the daemon on an existing db.

Prerequisite, filed separately as #3579. resync_forward_keys re-keys DNAT forwards with a bare BTreeMap::insert and discards the evicted entry (context.rs:625-638), while the SNI branch beside it warns; it also builds the new map with no overlap check, which breaks the "forwards on one external IP never overlap" invariant that overlapping()/occupied() depend on (tunnel/db.rs:347-383). It is reachable today, and every candidate fix here increases the number of forwards that can land on one address, so it should go first.

Revised recommendation. 1.2.2 carries only the CLI fix (#3570, merged) plus documenting that set-wan/device set-wan accept an address that is not globally routable — which is exactly what a 1:1-NAT or behind-a-router install needs, and is the escape hatch that makes the CLI fix a real unblock rather than a formality. The two-pass default_wan, the advertised field, and the UI work belong together in 1.3.0, after #3579.

Docs that are wrong today independent of any of this: docs/src/subnets.md:24-31 describes WAN IP as only the egress/SNAT address, when it is also the DNAT key, the SNI bind address, the Endpoint source and the PCP/IGD external address; and docs/src/ipv4.md:30-31 says public addresses are picked up automatically with no restart, but ctx.net_iface is a snapshot taken once at daemon start (tunnel/context.rs:242-259) and never refreshed.

One more gap worth naming: there is no free-text WAN entry anywhere in the UI — both selectors are closed dropdowns over the same public-address filter (web/src/app/routes/home/components/wan.ts:29-35), so on this topology they offer only "System default". The CLI already has the pattern to copy: init_web falls back to a prompt when its filtered suggestion list comes back empty (tunnel/web.rs:546-556).