feat(relay): NetworkOnly ACL — gate reservations on cluster membership (#1032)

* feat(relay): NetworkOnly ACL — gate reservations on cluster membership

Follow-up to #1031. With the relay service offered by every edgevpn
node, anyone on the public DHT who finds us can reserve a slot and
get us to carry their traffic. In a private cluster only network
members should be able to use us as a relay.

Adds a relayv2.ACLFilter (NetworkOnlyACL) that consults the local
ledger's alive bucket. A NetworkService periodically snapshots the
bucket into the ACL via an atomic.Pointer swap; AllowReserve checks
reduce to a constant-time map lookup.

Bootstrap window:
  A peer joining for the first time is not yet in our alive bucket
  (it needs to join gossipsub to write to it). If we strict-gated
  from t=0 a new peer could deadlock trying to reserve its way in.
  The ACL therefore allows ALL reservations until the first
  successful alive-bucket snapshot, then switches to strict mode.

AllowConnect is left permissive (return true): we gate the
reservation step, not in-flight relayed sessions, so a peer's
existing tunnel doesn't get yanked if the alive bucket briefly
flickers.

Knobs:
  --relay-service-network-only / EDGEVPN_RELAY_SERVICE_NETWORK_ONLY
    bool, default TRUE — secure by default. Pass =false to open
    the relay to all peers.
  --relay-service-acl-refresh / EDGEVPN_RELAY_SERVICE_ACL_REFRESH
    duration, default 30s — should be <= the alive-service announce
    interval so churn is reflected within a couple of ticks.

Tests (Ginkgo, package config_test):
- bootstrap window admits any peer until the first Members call
- strict mode admits members listed in the set
- strict mode rejects non-members
- AllowConnect stays permissive regardless of membership
- Members defensively copies the caller's map

The ACL lives in pkg/config alongside the existing relay-service
plumbing so pkg/node doesn't grow a relayv2 dependency. Wired via
the existing node.WithNetworkService pattern.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(relay-acl): e2e reservation handshake against a real libp2p relay

The state-machine specs cover the ACL in isolation. These put it
inside a real circuit-v2 relay and exercise the actual reservation
handshake — the contract the NetworkService relies on in production.

Four scenarios:
- bootstrap window: fresh ACL accepts any peer (client.Reserve
  returns a valid voucher)
- strict + member: voucher binds to the right peer ID
- strict + stranger: libp2p surfaces the relay's refusal as
  "reservation error: status: PERMISSION_DENIED reason:
  reservation failed" — proves the ACL ran on the real handshake
- membership flip: pre-membership denied, then Members(set)
  including the joiner is called, then the very next reservation
  attempt succeeds (mimics the alive-bucket watcher's behaviour)

Uses libp2p.ForceReachabilityPublic() in the relay host so the
relay service actually advertises itself — without it AutoNAT may
refuse to register /hop and the e2e test can't reach the ACL code
path.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Author: mudler

cmd/util.go

@@ -254,6 +254,18 @@ var CommonFlags []cli.Flag = []cli.Flag{
 		EnvVars: []string{"EDGEVPN_RELAY_SERVICE"},
 		Value:   true,
 	},
+	&cli.BoolFlag{
+		Name:    "relay-service-network-only",
+		Usage:   "Restrict incoming relay reservations to peers observed in the local ledger's alive bucket (cluster members). Strangers that found us via the public DHT or another relay discovery path are rejected. Requires the alive service to be running. During a short bootstrap window — before the alive bucket is first observed — every reservation is allowed so the node itself can finish joining the cluster. Default ON: secure by default; pass --relay-service-network-only=false to open the relay to all peers.",
+		EnvVars: []string{"EDGEVPN_RELAY_SERVICE_NETWORK_ONLY"},
+		Value:   true,
+	},
+	&cli.StringFlag{
+		Name:    "relay-service-acl-refresh",
+		Usage:   "Cadence at which the NetworkOnly relay-service ACL re-snapshots the alive bucket (Go duration). Should be <= the alive-service announce interval so peer churn is reflected within a couple of ticks.",
+		EnvVars: []string{"EDGEVPN_RELAY_SERVICE_ACL_REFRESH"},
+		Value:   config.DefaultRelayServiceACLRefresh.String(),
+	},
 	&cli.Int64Flag{
 		Name:    "relay-service-max-data",
 		Usage:   "Bytes (per direction) a relayed connection may carry before reset. Higher values let cluster peers carry larger relayed transfers (e.g. model files for distributed inference) at the cost of a larger memory footprint per relay client. Set lower for resource-constrained deployments.",
@@ -453,6 +465,10 @@ func ConfigFromContext(c *cli.Context) *config.Config {
 	if err != nil {
 		relayReservationTTL = 0
 	}
+	relayACLRefresh, err := time.ParseDuration(c.String("relay-service-acl-refresh"))
+	if err != nil {
+		relayACLRefresh = 0 // zero → ToOpts falls back to DefaultRelayServiceACLRefresh
+	}
 
 	// Authproviders are supposed to be passed as a json object
 	pa := c.String("peergate-auth")
@@ -507,12 +523,14 @@ func ConfigFromContext(c *cli.Context) *config.Config {
 			HighWater:                  c.Int("connection-high-water"),
 			LowWater:                   c.Int("connection-low-water"),
 			RelayService: config.RelayService{
-				Disabled:       !c.Bool("relay-service"),
-				MaxData:        c.Int64("relay-service-max-data"),
-				MaxDuration:    relayMaxDuration,
-				MaxCircuits:    c.Int("relay-service-max-circuits"),
-				ReservationTTL: relayReservationTTL,
-				BufferSize:     c.Int("relay-service-buffer-size"),
+				Disabled:           !c.Bool("relay-service"),
+				NetworkOnly:        c.Bool("relay-service-network-only"),
+				NetworkOnlyRefresh: relayACLRefresh,
+				MaxData:            c.Int64("relay-service-max-data"),
+				MaxDuration:        relayMaxDuration,
+				MaxCircuits:        c.Int("relay-service-max-circuits"),
+				ReservationTTL:     relayReservationTTL,
+				BufferSize:         c.Int("relay-service-buffer-size"),
 			},
 		},
 		Limit: config.ResourceLimit{

pkg/config/config.go

@@ -145,6 +145,27 @@ type RelayService struct {
 	// its own NAT. Default (zero value) is false → service is offered,
 	// preserving back-compat for programmatic callers.
 	Disabled bool
+	// NetworkOnly, when true, gates incoming relay reservations on
+	// cluster membership: only peers whose libp2p peer ID appears in
+	// our local ledger's alive bucket (peers that completed a
+	// gossipsub handshake against our network token) may reserve a
+	// slot on this node. Strangers that found us via the public DHT
+	// or another relay discovery path are rejected. During a short
+	// bootstrap window — before the alive bucket has been observed —
+	// every reservation is allowed so the node itself can finish
+	// joining its cluster. Requires the alive service to be running.
+	//
+	// Note on the default: the CLI flag --relay-service-network-only
+	// defaults to TRUE (secure by default). The Go zero value of this
+	// field is false purely because Go doesn't let us pick a different
+	// zero. Programmatic callers constructing &config.Config{} who
+	// want to match the CLI default should set NetworkOnly: true
+	// explicitly.
+	NetworkOnly bool
+	// NetworkOnlyRefresh is the cadence at which the NetworkOnly ACL
+	// re-snapshots the alive bucket. Zero means the package default
+	// (30s).
+	NetworkOnlyRefresh time.Duration
 	// MaxData is the byte limit (per direction) for a single relayed
 	// connection before it is reset. libp2p default is 128 KiB.
 	MaxData int64
@@ -326,9 +347,25 @@ func (c Config) ToOpts(l log.StandardLogger) ([]node.Option, []vpn.Option, error
 	// ability to reserve slots on OTHER relays via AutoRelay) stays on
 	// regardless — disabling the service does not prevent this node
 	// from using third-party relays to traverse its own NAT.
+	//
+	// When NetworkOnly is set, an ACL gates incoming reservations on
+	// cluster membership (peers observed in our alive bucket). The
+	// ACL is constructed here and shared with a NetworkService that
+	// keeps it up to date from the ledger; see relay_acl.go.
 	if !c.Connection.RelayService.Disabled {
-		libp2pOpts = append(libp2pOpts,
-			libp2p.EnableRelayService(relayv2.WithResources(RelayServiceResources(c.Connection.RelayService))))
+		relayOpts := []relayv2.Option{
+			relayv2.WithResources(RelayServiceResources(c.Connection.RelayService)),
+		}
+		if c.Connection.RelayService.NetworkOnly {
+			acl := &NetworkOnlyACL{}
+			relayOpts = append(relayOpts, relayv2.WithACL(acl))
+			refresh := c.Connection.RelayService.NetworkOnlyRefresh
+			if refresh <= 0 {
+				refresh = DefaultRelayServiceACLRefresh
+			}
+			opts = append(opts, node.WithNetworkService(NetworkOnlyACLService(acl, refresh)))
+		}
+		libp2pOpts = append(libp2pOpts, libp2p.EnableRelayService(relayOpts...))
 	}
 
 	if c.NAT.RateLimit {
@@ -581,6 +618,11 @@ const (
 	DefaultRelayServiceMaxCircuits    int           = 64
 	DefaultRelayServiceReservationTTL time.Duration = time.Hour
 	DefaultRelayServiceBufferSize     int           = 64 << 10 // 64 KiB
+	// DefaultRelayServiceACLRefresh is how often the NetworkOnly ACL
+	// re-snapshots the alive bucket. Should be ≤ the alive-service
+	// announce interval (default 120s) so peers leaving/joining are
+	// reflected within a couple of ticks.
+	DefaultRelayServiceACLRefresh time.Duration = 30 * time.Second
 )
 
 // RelayServiceResources builds a relayv2.Resources struct from the

pkg/config/config_suite_test.go

@@ -0,0 +1,26 @@
+/*
+Copyright © 2021-2026 Ettore Di Giacinto <mudler@mocaccino.org>
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestConfig(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Config Suite")
+}

pkg/config/relay_acl.go

@@ -0,0 +1,145 @@
+/*
+Copyright © 2021-2026 Ettore Di Giacinto <mudler@mocaccino.org>
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config
+
+import (
+	"context"
+	"sync/atomic"
+	"time"
+
+	"github.com/ipfs/go-log"
+	"github.com/libp2p/go-libp2p/core/peer"
+	ma "github.com/multiformats/go-multiaddr"
+
+	"github.com/mudler/edgevpn/pkg/blockchain"
+	"github.com/mudler/edgevpn/pkg/node"
+	"github.com/mudler/edgevpn/pkg/protocol"
+)
+
+// NetworkOnlyACL is a relayv2.ACLFilter that gates incoming circuit-v2
+// relay reservations on cluster membership. A peer is a "cluster
+// member" when its libp2p peer ID is present in the local ledger's
+// alive bucket (i.e. it has successfully gossiped a healthcheck
+// timestamp against our network token).
+//
+// During a short bootstrap window — before the ACL has been told
+// "you can start strict-mode now" — every reservation is allowed.
+// Without this window a new peer joining a network where we are
+// reachable would be unable to reserve a slot on us before having
+// proved itself, even though it can't prove itself until it has
+// joined gossipsub, which on many NAT'd setups requires going
+// through a relay first.
+//
+// The zero value is usable: AllowReserve returns true (open) until
+// the first call to Members(...) flips the gate via Start.
+type NetworkOnlyACL struct {
+	members atomic.Pointer[map[peer.ID]struct{}]
+	started atomic.Bool
+}
+
+// AllowReserve implements relayv2.ACLFilter. Returns true if the
+// peer is a recognised cluster member, or if strict-mode hasn't been
+// entered yet (bootstrap window).
+func (a *NetworkOnlyACL) AllowReserve(p peer.ID, _ ma.Multiaddr) bool {
+	if !a.started.Load() {
+		return true
+	}
+	m := a.members.Load()
+	if m == nil {
+		return true
+	}
+	_, ok := (*m)[p]
+	return ok
+}
+
+// AllowConnect implements relayv2.ACLFilter. We only gate the
+// reservation step — once a peer has a reservation it is by
+// definition a cluster member, so any connect through it stays
+// permitted. This keeps in-flight relayed sessions stable even if
+// the alive bucket flickers.
+func (a *NetworkOnlyACL) AllowConnect(_ peer.ID, _ ma.Multiaddr, _ peer.ID) bool {
+	return true
+}
+
+// Members replaces the authorised peer set. Callers can drop or add
+// peers atomically; readers see a consistent snapshot. Calling
+// Members for the first time also flips the gate out of bootstrap
+// mode: subsequent AllowReserve calls enforce the set strictly.
+func (a *NetworkOnlyACL) Members(set map[peer.ID]struct{}) {
+	// Defensive copy so callers can keep mutating their map without
+	// racing with the goroutines reading via AllowReserve.
+	cp := make(map[peer.ID]struct{}, len(set))
+	for k, v := range set {
+		cp[k] = v
+	}
+	a.members.Store(&cp)
+	a.started.Store(true)
+}
+
+// NetworkOnlyACLService is a node.NetworkService that periodically
+// snapshots the ledger's alive bucket into the supplied ACL.
+//
+// The first non-empty snapshot ends the bootstrap window. If the
+// bucket is empty (e.g. the alive service is disabled), the ACL
+// stays in open-mode and a debug log line is emitted on each tick
+// so operators can spot the misconfiguration.
+//
+// The refresh cadence should be ≤ the alive-service announce
+// interval; once per 30 s is a reasonable default.
+func NetworkOnlyACLService(acl *NetworkOnlyACL, refresh time.Duration) node.NetworkService {
+	return func(ctx context.Context, c node.Config, n *node.Node, b *blockchain.Ledger) error {
+		if acl == nil {
+			return nil
+		}
+		// Refresh once immediately so the very first reservation after
+		// the ledger is up does not have to wait a full tick.
+		refreshACL(c.Logger, acl, b)
+
+		go func() {
+			t := time.NewTicker(refresh)
+			defer t.Stop()
+			for {
+				select {
+				case <-ctx.Done():
+					return
+				case <-t.C:
+					refreshACL(c.Logger, acl, b)
+				}
+			}
+		}()
+		return nil
+	}
+}
+
+func refreshACL(logger log.StandardLogger, acl *NetworkOnlyACL, b *blockchain.Ledger) {
+	bucket := b.LastBlock().Storage[protocol.HealthCheckKey]
+	if len(bucket) == 0 {
+		// Ledger has no alive entries yet — could mean alive-service is
+		// disabled, or we are very early in startup. Stay in open-mode
+		// until something appears.
+		if logger != nil {
+			logger.Debugf("relay-service NetworkOnly ACL: alive bucket empty, keeping ACL open until next refresh")
+		}
+		return
+	}
+	set := make(map[peer.ID]struct{}, len(bucket))
+	for uuid := range bucket {
+		pid, err := peer.Decode(uuid)
+		if err != nil {
+			continue
+		}
+		set[pid] = struct{}{}
+	}
+	acl.Members(set)
+}