WIP 9 vpn: add gateway support

Author: pymq

README.md

@@ -22,6 +22,7 @@
   - [Linux server (`awl`)](#linux-server-awl)
 - [Connecting devices](#connecting-devices)
 - [Using devices as SOCKS5 proxy](#using-devices-as-socks5-proxy)
+- [VPN gateway (full-tunnel exit node)](#vpn-gateway-full-tunnel-exit-node)
 - [Configuration](#configuration)
   - [Config file location](#config-file-location)
   - [Example config](#example-config)
@@ -45,6 +46,7 @@ Some things people use it for:
 - SSH / RDP / VNC into your home or work laptop from anywhere, without port forwarding or exposing anything to the internet
 - reach selfhosted services (Nextcloud, Home Assistant, Bitwarden, ...) privately
 - route traffic through a remote device as a SOCKS5 proxy — useful for bypassing regional blocks
+- route *all* your traffic through a remote device at the IP layer — a full-tunnel VPN gateway / exit node
 - LAN-style multiplayer gaming across the internet
 - keep an old Android phone accessible for apps that only run there (e.g. with [scrcpy](https://github.com/Genymobile/scrcpy))
 
@@ -65,6 +67,7 @@ Tradeoffs worth knowing about:
 
 - fully peer-to-peer, no coordination server — see [Why Anywherelan](#why-anywherelan) above
 - route traffic through a device as a SOCKS5 proxy
+- route **all** your traffic through a device — full-tunnel VPN gateway / exit node
 - automatic NAT traversal via libp2p; falls back to community relays when a direct path isn't possible
 - TLS 1.3 encryption (QUIC or TCP+TLS)
 - built-in DNS: reach devices at `work-laptop.awl` instead of typing IPs
@@ -114,7 +117,7 @@ awl's transport security comes from [libp2p](https://docs.libp2p.io/).
 
 awl ships in two desktop flavors:
 
-- **`awl-tray`** — desktop build with a system-tray indicator: status at a glance, start/stop/restart, peer list. Use this for regular desktop usage.
+- **`awl-tray`** — desktop build with a system-tray indicator: status at a glance, start/stop/restart, peer list, and quick exit-node selection (SOCKS5 proxy / VPN gateway). Use this for regular desktop usage.
 - **`awl`** — headless server build, no GUI. Use this for servers and embedded devices.
 
 Both share the same web UI and the same [CLI](#terminal-based-client).
@@ -242,6 +245,7 @@ In addition to the per-application SOCKS5 proxy, awl can route **all** of your I
 - **macOS / Windows / other:** **not supported yet** — awl will refuse to start with VPN gateway enabled. Windows-side code exists but is unfinished; see `vpn/sockmark/sockmark_windows.go` and `vpn/routes/nat_windows.go` for the open work items.
 - IPv6 traffic is not tunnelled through the gateway in either direction (it is forwarded as a regular awl peer packet instead).
 - Both client and exit-node sides require **CAP_NET_ADMIN** on Linux. AWL needs that already to bring up the TUN, so there is no extra capability to grant.
+- The exit node sets up NAT through the host's `iptables` binary, which must be installed (on modern distros it resolves to `iptables-nft`). AWL only sees rules on whatever backend that binary uses; rules created against the other backend by unrelated software are invisible to it.
 
 ### Use a friend as your exit node (client side)
 
@@ -254,9 +258,8 @@ In addition to the per-application SOCKS5 proxy, awl can route **all** of your I
    awl cli gateway client use --name="peer-name"
    # or:  awl cli gateway client use --pid=<peer-id>
    ```
-   Calling `client use` again with a different peer atomically switches to the new gateway.
-3. **Restart `awl`** for the change to take effect. The OS-level routes and the libp2p socket marking that prevent the gateway from looping back through itself are wired up only at startup.
-4. Verify:
+   This applies immediately — no restart needed. The OS-level routes and the libp2p socket marking that keep the gateway from looping back through itself are installed at runtime. Calling `client use` again with a different peer atomically switches to the new gateway. (On the Android app the change is applied by re-establishing the `VpnService` in place, also without restarting awl.)
+3. Verify:
    ```bash
    awl cli gateway status
    curl ifconfig.me   # should show the exit node's public IP
@@ -265,23 +268,22 @@ In addition to the per-application SOCKS5 proxy, awl can route **all** of your I
 To turn the gateway off:
 
 ```bash
-awl cli gateway client stop
-# then restart awl to remove the OS-level routes
+awl cli gateway client stop   # OS-level routes are removed immediately
 ```
 
-You can also flip these settings from the web UI / system-tray menu, or by editing `config_awl.json` directly while awl is stopped (`gateway.enabled`, `gateway.exitNodePeerID`).
+You can also flip these settings from the web UI or the system-tray menu — both expose the exit-node picker and the "Serve as VPN gateway" toggle — or by editing `config_awl.json` directly while awl is stopped (`gateway.enabled`, `gateway.exitNodePeerID`).
 
 ### Serve as an exit node
 
 To allow others to route their internet traffic through this device:
 
-1. Set `gateway.serveAsVPNGateway: true` in `config_awl.json` (it is **off by default** — see "Why off by default" below) and restart awl. Equivalent: `awl cli gateway server enable` (and `awl cli gateway server disable` to turn it back off).
+1. Enable it with `awl cli gateway server enable` (and `awl cli gateway server disable` to turn it back off), or set `gateway.serveAsVPNGateway: true` in `config_awl.json` while awl is stopped. It is **off by default** — see "Why off by default" below. You can also use the "Serve as VPN gateway" toggle in the web UI or system tray.
 2. For each friend you want to permit, also tick *Allow as exit node* in their per-peer settings (or `awl cli peers allow_exit_node --name=… --allow=true`). The same flag governs SOCKS5; if you want different per-peer policies for SOCKS5 vs VPN gateway, file a request and we'll split it.
-3. Restart awl. On startup AWL will:
+3. Enabling exit-node mode applies immediately (no restart). AWL will:
    - flip `net.ipv4.ip_forward` on,
    - install a dedicated `AWL-FORWARD` iptables chain that DROPs traffic to your LAN/CGNAT/link-local subnets and ACCEPTs the rest,
    - add a `MASQUERADE` rule for the awl subnet,
-   - reverse all of the above on a clean shutdown.
+   - reverse all of the above when you disable it or on a clean shutdown.
 4. Connected friends can now pick you in their `gateway list` output once the next status exchange propagates the new state (≤ 5 minutes, or on next reconnect).
 
 ### Why off by default

application.go

@@ -142,7 +142,7 @@ func (a *Application) Init(ctx context.Context, tunDevice tun.Device) error {
 
 	a.Dns = NewDNSService(a.Conf, a.Eventbus, a.ctx, a.logger)
 	a.AuthStatus = service.NewAuthStatus(a.P2p, a.Conf, a.Eventbus)
-	a.SOCKS5, err = service.NewSOCKS5(a.P2p, a.Conf)
+	a.SOCKS5, err = service.NewSOCKS5(a.P2p, a.Conf, a.SockMarker)
 	if err != nil {
 		return fmt.Errorf("failed to init socks5: %v", err)
 	}

cmd/gomobile-lib/main.go

@@ -1,15 +1,14 @@
 //go:build linux && android
-// +build linux,android
 
 package anywherelan
 
 import (
 	"context"
 	"fmt"
 	"os"
-	"strconv"
 
 	"github.com/libp2p/go-libp2p/p2p/host/eventbus"
+	"golang.zx2c4.com/wireguard/tun"
 
 	"github.com/anywherelan/awl"
 	"github.com/anywherelan/awl/config"
@@ -20,6 +19,10 @@ import (
 var (
 	globalApp     *awl.Application
 	globalDataDir string
+	// globalSwapTUN is the swappable TUN wrapper handed to the Application when
+	// a VPN interface is active. UpdateTunDevice swaps a fresh fd into it after
+	// the host app re-establishes VpnService with different routes.
+	globalSwapTUN *vpn.SwappableTUN
 )
 
 // All public functions are part of the library
@@ -73,22 +76,60 @@ func StartServer(tunFD int32, protector SocketProtector) (err error) {
 		}
 	}()
 
-	_ = os.Setenv(vpn.TunFDEnvKey, strconv.Itoa(int(tunFD)))
-
 	globalApp = awl.New()
 	globalApp.SetupLoggerAndConfig()
 	globalApp.SockMarker = sockmark.NewAndroid(protectorToFunc(protector))
 
-	err = globalApp.Init(context.Background(), nil)
+	// A tunFD of 0 means the host did not establish a VPN interface (VPN
+	// disabled in config); Init then skips the VPN device entirely. Otherwise
+	// wrap the fd in a SwappableTUN so the interface can be replaced at runtime
+	// via UpdateTunDevice without restarting P2P.
+	var tunDevice tun.Device
+	if tunFD > 0 {
+		inner, tunErr := vpn.NewAndroidTUNFromFD(int(tunFD))
+		if tunErr != nil {
+			globalApp = nil
+			return tunErr
+		}
+		globalSwapTUN = vpn.NewSwappableTUN(inner)
+		tunDevice = globalSwapTUN
+	}
+
+	err = globalApp.Init(context.Background(), tunDevice)
 	if err != nil {
 		globalApp.Close()
 		globalApp = nil
+		if globalSwapTUN != nil {
+			_ = globalSwapTUN.Close()
+			globalSwapTUN = nil
+		}
 		return err
 	}
 
 	return nil
 }
 
+// UpdateTunDevice swaps the live TUN interface for the one backed by tunFD,
+// which the host app obtained from a fresh VpnService.establish() (e.g. after
+// changing routes to toggle VPN gateway mode). P2P connections are unaffected:
+// they run on sockets already protected via VpnService.protect, which bypass
+// the VPN regardless of the interface, and the swap replaces only the tun fd
+// inside the running Application. The previous fd is owned and closed by Go.
+func UpdateTunDevice(tunFD int32) error {
+	if globalApp == nil {
+		return fmt.Errorf("server is not running")
+	}
+	if globalSwapTUN == nil {
+		return fmt.Errorf("vpn interface is not active")
+	}
+
+	inner, err := vpn.NewAndroidTUNFromFD(int(tunFD))
+	if err != nil {
+		return err
+	}
+	return globalSwapTUN.Swap(inner)
+}
+
 func protectorToFunc(p SocketProtector) sockmark.ProtectFunc {
 	if p == nil {
 		return nil
@@ -100,9 +141,11 @@ func protectorToFunc(p SocketProtector) sockmark.ProtectFunc {
 
 func StopServer() {
 	if globalApp != nil {
+		// Application.Close closes the VPN device, which closes globalSwapTUN.
 		globalApp.Close()
 		globalApp = nil
 	}
+	globalSwapTUN = nil
 }
 
 func ImportConfig(data string) error {

service/socks5.go

@@ -7,6 +7,7 @@ import (
 	"net"
 	"slices"
 	"strings"
+	"syscall"
 	"time"
 
 	socks5Proxy "github.com/haxii/socks5"
@@ -18,6 +19,7 @@ import (
 	"github.com/anywherelan/awl/metrics"
 	"github.com/anywherelan/awl/protocol"
 	"github.com/anywherelan/awl/socks5"
+	"github.com/anywherelan/awl/vpn/sockmark"
 )
 
 type SOCKS5 struct {
@@ -29,7 +31,7 @@ type SOCKS5 struct {
 	server *socks5.Server
 }
 
-func NewSOCKS5(p2pService P2p, conf *config.Config) (*SOCKS5, error) {
+func NewSOCKS5(p2pService P2p, conf *config.Config, sockMarker sockmark.Marker) (*SOCKS5, error) {
 	logger := log.Logger("awl/service/socks5")
 
 	var client *socks5.Client
@@ -43,7 +45,15 @@ func NewSOCKS5(p2pService P2p, conf *config.Config) (*SOCKS5, error) {
 		logger.Infof("started socks5 proxy on socks5://%s", conf.SOCKS5.ListenAddress)
 	}
 
-	server := socks5.NewServer()
+	// Mark the exit-node's outbound sockets so they bypass our own VPN gateway
+	// default route (if any), mirroring libp2p socket marking. sockMarker is
+	// always set by Application.Init; ControlFunc may still be nil on platforms
+	// without socket marking, which NewServer handles.
+	var dialControl func(network, address string, c syscall.RawConn) error
+	if sockMarker != nil {
+		dialControl = sockMarker.ControlFunc()
+	}
+	server := socks5.NewServer(dialControl)
 	socks := &SOCKS5{
 		logger: logger,
 		p2p:    p2pService,

service/vpn_gateway.go

@@ -147,11 +147,11 @@ func (g *VPNGateway) ListAvailableVPNGateways() []entity.AvailableVPNGateway {
 // tunnel binding on apply failure.
 //
 // On android the OS-level apply (routes.SetupGatewayRoutes / sockmark) is a
-// no-op — routing is owned by the host's VpnService.Builder, which is only
-// rebuilt at daemon (re)start. Runtime calls still flip the in-memory tunnel
-// binding and persist config, but a restart is required for the change to
-// actually affect OS traffic; the UI is responsible for prompting that
-// restart.
+// no-op — routing is owned by the host's VpnService.Builder. This call flips
+// the in-memory tunnel binding and persists config; the host then re-establishes
+// the VpnService with the new routes and hot-swaps the fresh tun fd into the
+// running app (see cmd/gomobile-lib UpdateTunDevice and vpn.SwappableTUN), so the
+// change takes effect without restarting the daemon.
 func (g *VPNGateway) EnableClient(gatewayPeerID peer.ID) error {
 	if err := VPNGatewayClientSupported(); err != nil {
 		return err

socks5/server.go

@@ -6,6 +6,7 @@ import (
 	"io"
 	"net"
 	"sync/atomic"
+	"syscall"
 
 	"github.com/haxii/socks5"
 	"github.com/ipfs/go-log/v2"
@@ -22,15 +23,29 @@ type Server struct {
 	rule  *UpdatableRule
 }
 
-func NewServer() *Server {
+// NewServer constructs the SOCKS5 exit-node server. dialControl, when non-nil,
+// is applied to every outbound connection the server opens to the real
+// destination — it is the same socket-marking callback used for libp2p
+// sockets (sockmark). This keeps SOCKS5 exit-node traffic going out the host's
+// physical NIC, independent of whether this node is itself a VPN gateway
+// client (whose 0.0.0.0/0-via-TUN default route would otherwise capture these
+// sockets). Marking unconditionally is safe: when gateway mode is off there is
+// no matching fwmark ip-rule, so the kernel ignores the mark — mirroring how
+// libp2p sockets are marked at all times (see application.go / GATEWAY_FEATURE
+// §3.1). A nil dialControl yields a plain context-aware dialer.
+func NewServer(dialControl func(network, address string, c syscall.RawConn) error) *Server {
 	rule := NewUpdatableRule(NewRuleDenyLocalhost())
+	dialer := &net.Dialer{Control: dialControl}
 	conf := &socks5.Config{
 		// fake addr, we don't bind address for server
 		BindIP:   net.IPv4(127, 0, 0, 1),
 		BindPort: 8000,
 		Rules:    rule,
 		Logger:   NewLogger(),
 		Resolver: nil,
+		Dial: func(ctx context.Context, network, addr string) (net.Conn, error) {
+			return dialer.DialContext(ctx, network, addr)
+		},
 	}
 	server, err := socks5.New(conf)
 	if err != nil {

socks5/server_test.go

@@ -18,7 +18,7 @@ import (
 
 func TestProxy(t *testing.T) {
 	listenAddr := pickFreeAddr(t)
-	socksServer := NewServer()
+	socksServer := NewServer(nil)
 	socksServer.SetRules(NewRulePermitAll())
 	socksClient, err := NewClient(listenAddr, "", "")
 	require.NoError(t, err)
@@ -49,7 +49,7 @@ func TestProxy(t *testing.T) {
 
 func TestProxyWithAuth(t *testing.T) {
 	listenAddr := pickFreeAddr(t)
-	socksServer := NewServer()
+	socksServer := NewServer(nil)
 	socksServer.SetRules(NewRulePermitAll())
 	socksClient, err := NewClient(listenAddr, "testuser", "testpass")
 	require.NoError(t, err)

vpn/iface_android.go

@@ -1,37 +1,41 @@
 //go:build linux && android
-// +build linux,android
 
 package vpn
 
 import (
 	"fmt"
 	"net"
-	"os"
-	"strconv"
 
 	"golang.org/x/sys/unix"
 	"golang.zx2c4.com/wireguard/tun"
 )
 
-// TODO: refactor and remove this hack
-const TunFDEnvKey = "AWL_TUN_FD"
-
-func newTUN(ifname string, mtu int, localIP net.IP, ipMask net.IPMask) (tun.Device, error) {
-	fdStr := os.Getenv(TunFDEnvKey)
-	tunFD, err := strconv.ParseInt(fdStr, 10, 32)
-	if err != nil || tunFD == 0 {
-		return nil, fmt.Errorf("invalid tun FD %s: %v", fdStr, err)
-	}
-
-	tunDevice, _, err := tun.CreateUnmonitoredTUNFromFD(int(tunFD))
+// NewAndroidTUNFromFD wraps a file descriptor handed in by the host app's
+// VpnService (from establish().detachFd()) into a tun.Device. The returned
+// device takes ownership of fd and closes it on Close.
+//
+// On Android the awl process never creates the TUN itself — routing is owned by
+// VpnService.Builder, and changing routes (e.g. toggling VPN gateway mode)
+// requires a fresh establish() and therefore a new fd. The host passes that fd
+// down to gomobile, which wraps it here and swaps it into the running
+// SwappableTUN without restarting the rest of awl.
+func NewAndroidTUNFromFD(fd int) (tun.Device, error) {
+	tunDevice, _, err := tun.CreateUnmonitoredTUNFromFD(fd)
 	if err != nil {
-		unix.Close(int(tunFD))
+		unix.Close(fd)
 		return nil, fmt.Errorf("CreateUnmonitoredTUNFromFD: %v", err)
 	}
 
 	return tunDevice, nil
 }
 
+// newTUN is the nil-device path of NewDevice. On Android the TUN device must be
+// supplied externally via NewAndroidTUNFromFD (the host owns the fd), so being
+// asked to create one here is a programming error.
+func newTUN(_ string, _ int, _ net.IP, _ net.IPMask) (tun.Device, error) {
+	return nil, fmt.Errorf("android requires an externally-supplied tun device (use NewAndroidTUNFromFD)")
+}
+
 func (d *Device) InterfaceName() (string, error) {
 	interfaceName, err := d.tun.Name()
 	if err != nil {

vpn/routes/routes_linux.go

@@ -224,12 +224,20 @@ func buildFwmarkRule(fwmark uint32) *netlink.Rule {
 // SCOPE_LINK because the TUN is a point-to-point device with no gateway —
 // this matches what `ip route add default dev awl0` would produce, and using
 // the identical shape on RouteDel is required for the kernel to match it.
+//
+// Dst must be an explicit 0.0.0.0/0 *net.IPNet rather than nil: netlink's
+// RouteAdd rejects a route with no Dst.IP, Src and Gw ("either Dst.IP, Src.IP
+// or Gw must be set"). A TUN default route has no gateway (point-to-point), so
+// the destination is the only field we can populate to satisfy that check.
 func buildTunDefaultRoute(tunLinkIndex int) *netlink.Route {
 	return &netlink.Route{
 		LinkIndex: tunLinkIndex,
-		Dst:       nil, // 0.0.0.0/0
-		Scope:     netlink.SCOPE_LINK,
-		Priority:  tunRouteMetric,
+		Dst: &net.IPNet{
+			IP:   net.IPv4zero,
+			Mask: net.CIDRMask(0, 32),
+		},
+		Scope:    netlink.SCOPE_LINK,
+		Priority: tunRouteMetric,
 	}
 }