Merge pull request #250 from anywherelan/windows-vpn-gateway

vpn gateway: add Windows support (client + exit node) and refactor into vpn/netstate

Author: pymq

.github/workflows/test.yml

@@ -44,8 +44,24 @@ jobs:
         # `if: always()` cleanup step
         if: matrix.os == 'ubuntu-latest'
         run: |
-          go test -c -tags vpn_hostnet -o gw-hostnet.test ./vpn/routes/
+          go test -c -tags vpn_hostnet -o gw-hostnet.test ./vpn/netstate/
           sudo ./gw-hostnet.test -test.run '^TestGatewayHostNet' -test.v
+      - name: VPN gateway host-network integration test (Windows, admin)
+        # Windows counterpart of the step above: exercises WinNAT + WFP +
+        # per-interface forwarding (SetupNAT/TeardownNAT) against a real NIC
+        # and the /1 client routes against a real Wintun adapter. GitHub
+        # Windows runners execute as Administrator, so no sudo equivalent is
+        # needed. The diagnostic Get-NetNat shows what the runner image holds
+        # in WinNAT (a Docker/HNS instance would conflict — decide about
+        # pre-cleaning based on what this prints).
+        if: matrix.os == 'windows-latest'
+        shell: pwsh
+        run: |
+          Get-NetNat | Format-List
+          go test -c -tags vpn_hostnet -o gw-hostnet.test.exe ./vpn/netstate/
+          # Flags are quoted: pwsh splits an unquoted `-test.run` at the dot
+          # into `-test` + `.run`, which the test binary rejects.
+          ./gw-hostnet.test.exe '-test.run' '^TestGatewayHostNet' '-test.v'
       - name: Build cmd/awl
         run: go build github.com/anywherelan/awl/cmd/awl
       - name: Upload cmd/awl build
@@ -143,8 +159,10 @@ jobs:
 
           # Probe the egress IP via https://ifconfig.me
           curl4() { curl -4 -s --max-time 20 --retry 3 --retry-delay 2 https://ifconfig.me/; }
-
           IP_DIRECT=$(curl4); echo "egress IP (direct): $IP_DIRECT"
+          # Source IP of the physical uplink, used below to force a leak probe
+          # out of the NIC (bypassing the fwmark policy route).
+          NIC_IP=$(ip -4 route get 1.1.1.1 | grep -oP 'src \K\S+'); echo "uplink src IP: $NIC_IP"
 
           ./awl cli gateway client use --name awl-tester
           ./awl cli gateway status
@@ -168,6 +186,26 @@ jobs:
           ! echo "$RULE_OFF"  | grep -q fwmark               # policy route removed on disable
           ! echo "$ROUTE_OFF" | grep -q 'default dev awl0'   # TUN default removed on disable
 
+          # Second enable/disable cycle: a runtime re-enable after a disable is
+          # the exact flow that looked broken in manual Windows testing (it was
+          # not — browser keep-alive pools were lying). Assert the egress flips
+          # again and reverts again. Also probe for a leak: a socket forced out
+          # of the physical NIC (--interface) bypasses the fwmark policy route,
+          # but on Linux those packets are still routed by destination into the
+          # TUN and die there (their NIC source is not NATed by the exit node),
+          # so a NIC-bound curl must fail while the gateway is on. This is the
+          # weak-host-model counterpart of the Windows WFP leak fence.
+          ./awl cli gateway client use --name awl-tester
+          IP_GW2=$(curl4 || true); echo "egress IP (via gateway, cycle 2): $IP_GW2"
+          LEAK=$(curl -4 -s --max-time 8 --interface "$NIC_IP" https://ifconfig.me/ || true)
+          echo "NIC-bound egress while gateway on (must be empty): '$LEAK'"
+          ./awl cli gateway client stop
+          IP_REVERTED2=$(curl4); echo "egress IP (reverted, cycle 2): $IP_REVERTED2"
+          test -n "$IP_GW2"                                  # re-enable tunnelled again
+          test "$IP_GW2" != "$IP_DIRECT"                     # egress changed on 2nd enable too
+          test -z "$LEAK"                                     # NIC-bound egress did NOT bypass the tunnel
+          test "$IP_REVERTED2" = "$IP_DIRECT"                # 2nd disable reverted egress
+
           # Re-enable so the SIGINT shutdown path (teardownGatewayAtShutdown) is exercised.
           ./awl cli gateway client use --name awl-tester
 
@@ -209,11 +247,124 @@ jobs:
           sleep 10
           awl_pid=`jobs -l | grep './awl' | awk '{print $2}'`
           ./awl.exe cli peers status
+
           ./librespeed-cli.exe --local-json config_librespeed.json --server 2 --json --share --telemetry-level disabled | python3 -m json.tool
 
           ping -w 20000 -n 10 10.66.0.2
           ping -w 20000 -n 10 -a awl-tester.awl
 
+          # ---- VPN gateway server (exit-node) mode: runtime enable/disable round-trips OS state ----
+          # Diagnostic first: what the runner already holds in WinNAT (a
+          # Docker/HNS instance here would explain a New-NetNat conflict).
+          powershell -NoProfile -Command 'Get-NetNat | Format-List'
+          ./awl.exe cli gateway server enable
+          powershell -NoProfile -Command 'if (-not (Get-NetNat -Name awl-gateway -ErrorAction SilentlyContinue)) { exit 1 }'
+          # WinNAT must not break p2p traffic inside the TUN
+          ping -w 20000 -n 4 10.66.0.2
+          ./awl.exe cli gateway server disable
+          powershell -NoProfile -Command 'if (Get-NetNat -Name awl-gateway -ErrorAction SilentlyContinue) { exit 1 }'
+
+          # ---- VPN gateway full-tunnel e2e: client = this runner, exit node = awl-tester ----
+          # Requires awl-tester to permit this CI peer as an exit node
+          # (WeAllowUsingAsExitNode) — the Windows CI peer id differs from the
+          # Linux one.
+          for i in $(seq 1 15); do
+            if ./awl.exe cli gateway list | grep -E 'awl-tester.*\[connected\]'; then break; fi
+            sleep 2
+          done
+          ./awl.exe cli gateway list | grep -E 'awl-tester.*\[connected\]'   # fail if never became available
+
+          curl4() { curl -4 -s --max-time 20 --retry 3 --retry-delay 2 https://ifconfig.me/; }
+          IP_DIRECT=$(curl4); echo "egress IP (direct): $IP_DIRECT"
+          # Source IP of the physical uplink (captured before enable, so the
+          # best route still points at the NIC, not the TUN). Used to force a
+          # leak probe out of the NIC past the /1 routes.
+          NIC_IP=$(powershell -NoProfile -Command '(Find-NetRoute -RemoteIPAddress 1.1.1.1 | Select-Object -First 1).IPAddress' | tr -d '\r'); echo "uplink src IP: $NIC_IP"
+
+          # NRPT diagnostics before enable (split-DNS rules for the awl zone)
+          powershell -NoProfile -Command 'Get-DnsClientNrptPolicy | Format-List Namespace,NameServers'
+
+          ./awl.exe cli gateway client use --name awl-tester
+          ./awl.exe cli gateway status
+          ROUTES_ON=$(powershell -NoProfile -Command '(Get-NetRoute -DestinationPrefix 0.0.0.0/1,128.0.0.0/1 -ErrorAction SilentlyContinue | Measure-Object).Count' | tr -d '\r')
+          NRPT_ON=$(powershell -NoProfile -Command '(Get-DnsClientNrptPolicy | Where-Object { $_.Namespace -eq "." } | Measure-Object).Count' | tr -d '\r')
+          powershell -NoProfile -Command 'Get-DnsClientNrptPolicy | Format-List Namespace,NameServers'
+          # Route diagnostics incl. lifetimes/protocol: a route created without
+          # InitializeIpForwardEntry shows zero lifetimes and is ignored by the
+          # forwarding path while still being listed here.
+          powershell -NoProfile -Command 'Get-NetRoute -DestinationPrefix 0.0.0.0/1,128.0.0.0/1,::/1,8000::/1 -ErrorAction SilentlyContinue | Format-List DestinationPrefix,InterfaceAlias,RouteMetric,Protocol,ValidLifetime,PreferredLifetime'
+          # Tolerate a failed curl here so we always reach `client stop` below
+          # (a broken tunnel must not leave the runner black-holed); the
+          # `test -n "$IP_GW"` assertion after teardown turns it into a clean failure.
+          IP_GW=$(curl4 || true); echo "egress IP (via gateway): $IP_GW"
+          # Leak probe: a socket forced out of the physical NIC (--interface)
+          # bypasses the /1 routes via the strong host model — exactly the leak
+          # the WFP fence exists to close. Must fail while the gateway is on.
+          LEAK=$(curl.exe --interface "$NIC_IP" -4 -s --max-time 8 https://ifconfig.me/ || true)
+          echo "NIC-bound egress while gateway on (must be empty): '$LEAK'"
+
+          ./awl.exe cli gateway client stop
+          ROUTES_OFF=$(powershell -NoProfile -Command '(Get-NetRoute -DestinationPrefix 0.0.0.0/1,128.0.0.0/1 -ErrorAction SilentlyContinue | Measure-Object).Count' | tr -d '\r')
+          NRPT_OFF=$(powershell -NoProfile -Command '(Get-DnsClientNrptPolicy | Where-Object { $_.Namespace -eq "." } | Measure-Object).Count' | tr -d '\r')
+          IP_REVERTED=$(curl4); echo "egress IP (reverted): $IP_REVERTED"
+
+          # Assertions run with the gateway already disabled, so a failure here
+          # can't leave the runner's egress black-holed. Values are echoed
+          # first so a failed assertion is attributable from the log alone.
+          echo "ROUTES_ON=$ROUTES_ON NRPT_ON=$NRPT_ON ROUTES_OFF=$ROUTES_OFF NRPT_OFF=$NRPT_OFF LEAK='$LEAK'"
+          test "$ROUTES_ON" -eq 2                            # /1 pair installed while gateway is on
+          test "$NRPT_ON" -ge 1                              # full-capture NRPT rule while gateway is on
+          test -n "$IP_GW"                                   # traffic really flowed through the exit node
+          test "$IP_GW" != "$IP_DIRECT"                      # egress changed => full-tunnel works
+          test "$IP_REVERTED" = "$IP_DIRECT"                 # runtime disable reverted egress
+          test "$ROUTES_OFF" -eq 0                           # /1 routes removed on disable
+          test "$NRPT_OFF" -eq 0                             # full-capture NRPT rule removed on disable
+          test -z "$LEAK"                                     # WFP fence blocked NIC-bound egress bypass
+
+          # Second enable/disable cycle: a runtime re-enable after a disable is
+          # the exact flow that looked broken in manual testing (it was not —
+          # browser keep-alive pools were lying). Assert egress flips again and
+          # reverts again; the gateway is already off at each assertion so a
+          # failure can't black-hole the runner.
+          ./awl.exe cli gateway client use --name awl-tester
+          IP_GW2=$(curl4 || true); echo "egress IP (via gateway, cycle 2): $IP_GW2"
+          ./awl.exe cli gateway client stop
+          IP_REVERTED2=$(curl4); echo "egress IP (reverted, cycle 2): $IP_REVERTED2"
+          test -n "$IP_GW2"                                  # re-enable tunnelled again
+          test "$IP_GW2" != "$IP_DIRECT"                     # egress changed on 2nd enable too
+          test "$IP_REVERTED2" = "$IP_DIRECT"                # 2nd disable reverted egress
+
+          # Re-enable the gateway and hard-kill awl. A graceful-shutdown test is
+          # not possible here: Git-Bash `kill -SIGINT` cannot deliver a console
+          # ctrl event to a native Windows process on a console-less CI runner
+          # (verified: awl kept logging after the kill). Graceful teardown is
+          # already covered by the `client stop` assertions above and by the
+          # Linux branch; what a hard kill lets us assert is the documented
+          # CRASH semantics instead.
+          ./awl.exe cli gateway client use --name awl-tester
           sleep 1
-          kill -SIGINT $awl_pid
-          sleep 1
+          taskkill //F //IM awl.exe
+
+          # Crash fail-open: the /1 routes (v4 + v6 fence) are bound to the
+          # Wintun LUID and must disappear together with the adapter. Driver
+          # cleanup after a hard kill is asynchronous — poll.
+          DANGLING=-1
+          for i in $(seq 1 30); do
+            DANGLING=$(powershell -NoProfile -Command '(Get-NetRoute -DestinationPrefix 0.0.0.0/1,128.0.0.0/1,::/1,8000::/1 -ErrorAction SilentlyContinue | Measure-Object).Count' | tr -d '\r')
+            if [ "$DANGLING" -eq 0 ]; then break; fi
+            sleep 1
+          done
+          echo "DANGLING=$DANGLING"
+          test "$DANGLING" -eq 0
+
+          # The NRPT catch-all rule lives in the registry and must SURVIVE a
+          # hard kill — that's the documented crash behaviour (the next awl
+          # start reconfigures DNS and cleans it up).
+          NRPT_CRASH=$(powershell -NoProfile -Command '(Get-DnsClientNrptPolicy | Where-Object { $_.Namespace -eq "." } | Measure-Object).Count' | tr -d '\r')
+          echo "NRPT_CRASH=$NRPT_CRASH"
+          test "$NRPT_CRASH" -ge 1
+
+          # Runner hygiene: with awl dead, the leftover catch-all rule points
+          # all DNS at a resolver that no longer exists — remove it so the
+          # runner's post-job steps keep working DNS.
+          powershell -NoProfile -Command 'Get-DnsClientNrptRule | Where-Object { $_.Namespace -eq "." } | Remove-DnsClientNrptRule -Force'

README.md

@@ -250,18 +250,18 @@ In short: reach for SOCKS5 to send a single app through a peer, and for the VPN
 | --- | --- | --- | --- |
 | Linux | ✅ | ✅ | fully supported |
 | Android | ✅ | ❌ | exit-node role needs root — not planned |
-| Windows | ⏳ | ⏳ | coming next |
+| Windows | ✅ | ✅ | fully supported |
 | macOS | ❌ | ❌ | needs volunteers for testing |
 
-On macOS and Windows awl currently refuses to start with VPN gateway enabled.
+On macOS awl currently refuses to start with VPN gateway enabled.
 
 > ⚠️ **IPv6 is not tunnelled.** The gateway only carries IPv4. While it's on, all IPv6 traffic is dropped so that your real IPv6 address is never exposed past the exit node:
 > - **Dual-stack (IPv4 + IPv6):** everything automatically uses IPv4 through the tunnel.
 > - **IPv6-only network:** you'll have no internet connectivity until you turn the gateway off.
 
 ### Serve as an exit node
 
-This lets your other devices route their internet traffic out through this one. It is **off by default** — see [Why serving as an exit node is opt-in](#why-serving-as-an-exit-node-is-opt-in) below. Two things need to be set: turn the gateway service on, then allow each specific device to use it. Serving as an exit node is Linux-only (see the status table above).
+This lets your other devices route their internet traffic out through this one. It is **off by default** — see [Why serving as an exit node is opt-in](#why-serving-as-an-exit-node-is-opt-in) below. Two things need to be set: turn the gateway service on, then allow each specific device to use it.
 
 **Desktop (web UI):** open http://admin.awl, go to **Settings** (the gear icon, top-right) and turn on **Serve as VPN Gateway**. Then, for each device you want to permit, open its card on the Overview page, click **Settings**, and set **Allow as exit node** to *Allowed*. On `awl-tray` you can also toggle the service from the tray menu under **VPN Gateway → Serve as VPN Gateway**.
 
@@ -305,7 +305,7 @@ awl cli gateway client stop
 
 ### Why serving as an exit node is opt-in
 
-Unlike the SOCKS5 proxy, serving as a VPN gateway changes global system state on the host: awl turns on `net.ipv4.ip_forward` and installs iptables rules. That can interfere with the host's existing networking or firewall setup, and it isn't something awl can sandbox — so we don't enable it on a routine install. You opt in explicitly, the same way every mainstream VPN (ZeroTier, WireGuard, OpenVPN, ...) keeps exit-node mode opt-in.
+Unlike the SOCKS5 proxy, serving as a VPN gateway changes global system state on the host: on Linux awl turns on `net.ipv4.ip_forward` and installs iptables rules; on Windows it creates a WinNAT instance, enables per-interface IP forwarding and installs a WFP firewall filter. That can interfere with the host's existing networking or firewall setup, and it isn't something awl can sandbox — so we don't enable it on a routine install. You opt in explicitly, the same way every mainstream VPN (ZeroTier, WireGuard, OpenVPN, ...) keeps exit-node mode opt-in.
 
 The privacy exposure — your IP appearing as the source of another device's traffic — is *not* what this toggle gates: it is the same for SOCKS5 and the VPN gateway, and it's controlled by the per-device **Use as exit** permission (see [Security and privacy notes](#security-and-privacy-notes) below). This toggle only governs the host-level networking changes above.
 
@@ -314,12 +314,16 @@ The privacy exposure — your IP appearing as the source of another device's tra
 - **A device isn't available as an exit node.** It hasn't turned on **Serve as VPN Gateway**, or hasn't set **Allow as exit node** to *Allowed* for you, or the status exchange hasn't propagated yet — wait up to ~5 minutes or until the next reconnect.
 - **A "what's my IP" site still shows your own IP after enabling.** Check the gateway status (the **VPN Gateway** card, or `awl cli gateway status`): if it's not connected, awl can't reach the exit node, so nothing is being tunnelled.
 - **A site works over IPv6 but not through the gateway.** Expected — IPv6 isn't tunnelled (see the note above). Dual-stack hosts fall back to IPv4 automatically; anything IPv6-only won't work while the gateway is on.
+- **Turning on *Serve as VPN Gateway* fails on Windows.** Windows effectively allows one NAT instance per host, and it may already be taken by Docker (Windows containers), WSL2 or Internet Connection Sharing — the error message lists the current holders. Free it up, or share this device over SOCKS5 instead: the SOCKS5 exit node doesn't need NAT.
+- **No IPv6 connectivity after awl crashed (Linux).** If awl is killed (not shut down) with the gateway client on, its IPv6 block stays behind. It is removed automatically on the next awl start (and stop).
+- **Devices are reachable only via relay from a Windows machine with multiple network interfaces.** awl on Windows pins its peer-to-peer traffic to the interface that holds the default route, so peers reachable only through a secondary network card may fall back to relayed connections.
 
 ### Security and privacy notes
 
 - **Your IP is exposed.** Once you serve as an exit node, the public IPs that those devices reach see your IP, not theirs.
-- **Your LAN is not.** awl drops forwarded traffic to RFC 1918 / RFC 6598 / RFC 3927 ranges (`10/8`, `172.16/12`, `192.168/16`, `100.64/10`, `169.254/16`) so a gateway client cannot reach the exit node's home network.
+- **Your LAN is not.** awl drops forwarded traffic to RFC 1918 / RFC 6598 / RFC 3927 ranges (`10/8`, `172.16/12`, `192.168/16`, `100.64/10`, `169.254/16`) so a gateway client cannot reach the exit node's home network. On Linux this is an iptables chain; on Windows, a WFP forward-layer BLOCK filter with the same subnet list.
 - **DNS:** in client gateway mode awl forces upstream DNS to a public resolver (`1.1.1.1` by default) so the LAN resolver can't leak queries past the tunnel. If you'd rather use a different resolver, you can change it by hand in the config file (`dns.upstreamDNSAddress`) while awl is stopped.
+- **Connections open before you enable the gateway are cut, not leaked.** When you turn client gateway mode on, existing outbound connections (a browser's keep-alive pools are the common case) are forced to break so applications re-establish them through the tunnel — otherwise, on Windows, they would keep flowing directly through your physical network card and expose your real IP. On Linux this happens naturally (the tunnel routing kills them); on Windows awl installs a WFP firewall to force it. One deliberate exception on Windows: if this device is *hosting* a service (e.g. an RDP or SMB server) and someone is already connected to it, that inbound connection keeps answering directly, so that enabling the gateway over a remote session does not disconnect you. Fresh inbound connections while the gateway is on are still limited to the tunnel path.
 
 ## Configuration