feat: host-level data path performance tuning

.agents/ARCHITECTURE.md

@@ -71,20 +71,26 @@ Validation is split: CEL on the CRD enforces the spec shape (`tls` only on TCP b
 
 ### 4.1 EdgeNodeReconciler (sole writer to the machine and uplink)
 1. **IPAM and keys**: Assigns tunnel IP by ordinal. Generates one WireGuard keypair per replica (stored in the `<node>-uplink-keys` Secret). The VPS keypair is generated locally in-cluster with `wgctrl`/`wgtypes` and cached in the `<node>-vps-key` Secret, so no command runs on the VPS to create it.
-2. **SSH Enrollment**: Idempotent. Pushes the matching static `tunnelctl` binary to the VPS (read from the local `--tunnelctl-dir`: baked into the operator image, or a local build dir under `make run`; the right arch chosen from `uname -m`), plus the static Envoy binary; writes the relay `tunnelctl` desired-state document and applies it with `tunnelctl apply` (WireGuard via netlink, no `wireguard-tools`, no `wg-quick`, no download on the VPS). The only systemd units installed are the Envoy service and a `wg-relay` boot oneshot (`Type=oneshot`, `RemainAfterExit=yes`) that reapplies the relay document on boot so WireGuard survives a VPS reboot; the Envoy unit `Requires`/`After` it. Neither is a resident WireGuard daemon. It also writes the immutable Envoy bootstrap plus LDS/CDS files, any TLS material for edge-terminating bindings, and `state.json` with artifact hashes (relay document, tunnelctl binary hash, LDS, CDS, TLS).
+2. **SSH Enrollment**: Idempotent. Pushes the matching static `tunnelctl` binary to the VPS (read from the local `--tunnelctl-dir`: baked into the operator image, or a local build dir under `make run`; the right arch chosen from `uname -m`), plus the static Envoy binary; writes the relay `tunnelctl` desired-state document and applies it with `tunnelctl apply` (WireGuard via netlink, no `wireguard-tools`, no `wg-quick`, no download on the VPS). The only systemd units installed are the Envoy service and the `tunnel-boot.service` boot oneshot (`Type=oneshot`, `RemainAfterExit=yes`) that reapplies the relay desired-state document on boot (WireGuard plus any `netdev` tuning) so the relay survives a VPS reboot; the Envoy unit `Requires`/`After` it. Neither is a resident daemon. The unit is named `tunnel-boot` because it reapplies the whole node document, not just WireGuard; the WireGuard *interface* it brings up is named `wg-relay`. It also writes the immutable Envoy bootstrap plus LDS/CDS files, any TLS material for edge-terminating bindings, and `state.json` with artifact hashes (relay document, tunnelctl binary hash, LDS, CDS, TLS).
 3. **Uplink**: Creates/updates a StatefulSet with the uplink image, a ConfigMap holding the shared uplink `tunnelctl` desired-state document (WireGuard + nftables; each replica injects its own identity at runtime), and the headless Service the StatefulSet's `spec.serviceName` references (ClusterIP None, selector-only).
 4. **Aggregate Render**: Lists all PortBindings that reference this EdgeNode, resolves Service targets to ClusterIPs, validates cross-binding port conflicts (and conflicts with the tunnel ListenPort), and renders the relay and uplink `tunnelctl` desired-state documents plus the Envoy LDS/CDS via `internal/planner.BuildPlan`. For bindings with `tls` in `offload`/`mutual`, the plan also declares the TLS material to install (the per-binding SDS document path on the VPS); the reconciler reads the referenced `kubernetes.io/tls` Secret, validates the required keys, renders a per-binding SDS document (server cert/key inline, plus `ca.crt` for `mutual`) and pushes it to the edge.
 5. **Drift**: Re-render is event-driven plus a periodic safety net. The happy path requeues after `RequeueInterval` (default 3 min) for drift detection and status refresh, and also reacts to watches (the EdgeNode itself, the PortBindings that reference it, and the TLS Secrets referenced by its bindings). It compares `status.appliedConfigHash` (the plan hash folded with the TLS material hash, so cert rotations are visible) and the per-artifact hashes against the desired plan and the remote `state.json`; only re-pushes/restarts services when hashes differ, and reports health back into status. Envoy health is read via `systemctl is-active`; relay WireGuard health and the peer handshakes are read via `tunnelctl status` (JSON + exit code). Optimistic-concurrency conflicts on writes/status updates are turned into clean requeues, never swallowed.
 6. **Binary delivery**: which Envoy release is installed is an operator-wide setting, the manager flag `--envoy-version` (default `controller.DefaultEnvoyVersion`); `provision.installEnvoyBinary` builds the download URL from version + architecture and reinstalls when Envoy is absent or a different version. `tunnelctl` is NOT downloaded on the VPS: the operator reads the static binary from `--tunnelctl-dir` (default `controller.DefaultTunnelctlDir`, baked into the operator image; overridden to a local build dir under `make run`) and pushes the arch-matching one over SSH, re-pushing only when its SHA-256 changes (tracked in `state.json`). The controller injects the Envoy version and the tunnelctl dir into the plan after `BuildPlan` (the planner stays pure). The operator stamps its own version into the binary via `internal/version.Version` (`-ldflags -X`), logged at startup. A running Envoy does not pick up a new binary until it is restarted (see the restart-envoy annotation).
-7. **Teardown** (on deletion, guarded by a finalizer): runs `provision.Teardown` over SSH (deletes the `wg-relay` interface with `ip link del`, removes the `tunnelctl` binary, stops/disables both the Envoy service and the `wg-relay` boot oneshot and removes both systemd units, removes the Envoy config, the sysctl drop-in and `/etc/tunnel`) and then explicitly deletes the in-cluster uplink StatefulSet, ConfigMap, keys Secret and the cached `<node>-vps-key` Secret. They are deleted explicitly (not via owner references) because they live in `spec.uplink.namespace`, which can differ from the EdgeNode namespace, and cross-namespace owner-reference GC is not allowed. Set the `tunnel.achetronic.com/skip-deprovision: "true"` annotation to skip the SSH teardown.
-
+7. **Teardown** (on deletion, guarded by a finalizer): runs `provision.Teardown` over SSH (deletes the `wg-relay` interface with `ip link del`, removes the `tunnelctl` binary, stops/disables both the Envoy service and the `tunnel-boot` boot oneshot and removes both systemd units, removes the Envoy config, the sysctl drop-in and `/etc/tunnel`) and then explicitly deletes the in-cluster uplink StatefulSet, ConfigMap, keys Secret and the cached `<node>-vps-key` Secret. They are deleted explicitly (not via owner references) because they live in `spec.uplink.namespace`, which can differ from the EdgeNode namespace, and cross-namespace owner-reference GC is not allowed. Set the `tunnel.achetronic.com/skip-deprovision: "true"` annotation to skip the SSH teardown.
 #### 4.1.0 Operational annotations
 - `tunnel.achetronic.com/skip-deprovision: "true"` skips the SSH teardown on deletion (the in-cluster uplink resources are still removed).
 - `tunnel.achetronic.com/restart-envoy: "true"` makes the next reconcile restart Envoy on the VPS (via `provision.RestartEnvoy`, a systemctl restart that waits for the unit to become active) after enrollment, the way to apply a new `--envoy-version`. It is a one-shot: the reconciler consumes and removes the annotation in the same metadata update as the finalizer, before any status mutation, so it never clobbers the status write and never loops. A failed restart surfaces as `Ready=False`/`EnvoyRestartFailed`.
 
 #### 4.1.1 TLS certificate rotation
 The EdgeNodeReconciler `Watches` Secrets. When a Secret referenced by some binding's `tls.secretRef` changes (e.g. cert-manager renews it), a map function (`mapSecretToEdgeNodes`) resolves which EdgeNodes consume it and enqueues them, so the renewed certificate is re-pushed to the VPS. Delivery is via file-based **SDS**: the operator writes `/etc/envoy/tls/<binding>.sds.yaml` (cert/key inline, plus the CA for `mutual`) with a single atomic `mv` into the `watched_directory` (`/etc/envoy/tls`) that Envoy monitors; Envoy hot-reloads just that secret gracefully, with no listener rebuild, no dropped connections and no restart. Only Secrets actually referenced by a TLS binding produce a requeue. The `PrivateKeyOnEdge` warning Event is emitted only when key material is genuinely (re)written this reconcile, not on every loop.
 
+#### 4.1.2 Host-level performance tuning
+The optional `spec.host` block tunes the VPS as a machine (kernel and NIC), independent of the proxy. The planner defaults and threads it; nothing here is per-listener and there is no generic escape-hatch.
+- **`kernelMaxSocketBufferBytes` (socket-buffer ceiling, default 25MB).** One knob wired to two layers, so the user sets a single number. (1) The operator writes the kernel ceiling `net.core.rmem_max`/`wmem_max` into the sysctl drop-in `/etc/sysctl.d/99-tunnel.conf` over SSH during enrollment (alongside `ip_forward`), persisted natively across reboots like any sysctl drop-in. (2) The Envoy render sets `SO_RCVBUF`/`SO_SNDBUF` (SOL_SOCKET, `STATE_PREBIND`) on the UDP listeners only, up to that ceiling. TCP autotunes below the ceiling on its own, so it gets no explicit socket options; UDP does not autotune, so the explicit reclaim is what makes the ceiling effective for it.
+- **`disableNicOffloads` (GRO/GSO toggle, default false).** Turns GRO and GSO off on the underlay NIC carrying the WireGuard transport, auto-detected via a netlink route lookup to a public IP (never the `wg-relay` tunnel device, never a hardcoded name). It is applied natively over ethtool-netlink by `tunnelctl` as the `netdev` section of the relay desired-state document (no `ethtool` CLI on the host), so the `tunnel-boot` oneshot reapplies it on reboot. Default off because disabling offloads degrades throughput for ordinary TCP/QUIC traffic by raising per-packet CPU; it is for encapsulated, non-uniform UDP (tunnel-in-tunnel) where GRO coalescing corrupts datagram boundaries and breaks Envoy's L4 datagram semantics.
+- **Envoy render (no knob).** GRO is delegated to the OS (controlled at the NIC via `disableNicOffloads`), so the UDP listeners carry no `prefer_gro`. Every listener sets `enable_reuse_port: true` explicitly (honored for both TCP and UDP on Linux; the `reuse_port` field is deprecated).
+- **Teardown does not restore live values.** Like `ip_forward`, teardown removes the persistence (the sysctl drop-in file and the `tunnel-boot` unit that reapplies the `netdev` section) and lets the live kernel/NIC values revert on the next reboot, rather than capturing and restoring prior state.
+
 ### 4.2 PortBindingReconciler (independent of the EdgeNodeReconciler)
 PortBindings have their own reconciler with its own lifecycle; the two controllers share no code or memory and communicate exclusively through the API server (`status.appliedBindings` + an EdgeNode watch upstream, see the NOTE below). This reconciler does not speak SSH, does not touch the uplink, does not resolve targets or validate conflicts, and never writes to the EdgeNode; its sole job is publishing the PortBinding's conditions. Plan rebuilding is driven by the EdgeNodeReconciler itself: it `Watches` PortBindings with a map function (`mapPortBindingToEdgeNode`) that enqueues the referenced EdgeNode on every create, spec change and deletion (a generation predicate filters PortBinding status writes, which carry no plan input), so the aggregate plan always reflects the live set of bindings, including dropped listeners after a deletion. PortBindings carry no finalizer; if an object still has `tunnel.achetronic.com/portbinding-finalizer`, the reconciler removes it on deletion so the object is never stuck. Target resolution and cross-binding conflict validation (including binding-name uniqueness across all PortBindings of a node, since the name keys the Envoy listener and the SDS document path) happen inside `BuildPlan` during EdgeNode reconciliation.
 

.agents/DESIGN_AND_RULES.md

@@ -41,3 +41,15 @@ A TCP PortBinding may terminate or route TLS at the VPS via an optional `tls: {
 
 ## 6. Agents
 This `.agents/` folder contains the knowledge material the LLM must read to absorb context before altering the data path.
+
+## 7. Host-Level Tuning (Closed Decisions)
+
+- **Single knob for socket buffers:** `spec.host.kernelMaxSocketBufferBytes` provides a unified control for kernel `net.core.rmem_max`/`wmem_max` sysctls and Envoy UDP listener `SO_RCVBUF`/`SO_SNDBUF` socket options. This simplifies user configuration: one value effectively raises the socket buffer ceiling across both kernel and userspace for UDP, which does not autotune. TCP socket buffers autotune up to this ceiling, so no explicit Envoy TCP listener socket options are set.
+- **Dedicated NIC offload toggle:** `spec.host.disableNicOffloads` offers a precise switch for GRO/GSO on the underlay physical NIC carrying the WireGuard transport.
+    - **Auto-detection:** The underlay NIC is auto-detected via `netlink route get` to the VPS's public IP.
+    - **Natively applied:** Offloads are managed using `ethtool-netlink` by `tunnelctl` as a `netdev` section within the relay desired-state document. This method ensures changes persist across reboots via the boot oneshot.
+    - **Default off:** The default `false` (offloads enabled) is chosen because disabling GRO/GSO degrades throughput for general TCP/QUIC traffic. It is intended for specific encapsulated UDP or tunnel-in-tunnel workloads where GRO can corrupt datagram boundaries and break L4 datagram semantics.
+    - **No per-listener granularity:** The offload setting applies to the entire underlay NIC, not per Envoy listener, reflecting its kernel-level nature.
+    - **No generic escape-hatch:** This specific toggle for GRO/GSO addresses a known issue in certain high-performance tunnel scenarios; a generic `ethtool` escape-hatch is not provided to prevent arbitrary kernel network changes.
+- **Envoy `prefer_gro` absent, `enable_reuse_port:true` explicit:** GRO is delegated to the OS (and disabled at the NIC via `disableNicOffloads` when needed), so the operator-managed Envoy config carries no `prefer_gro`. Every listener sets `enable_reuse_port:true` (honored for both TCP and UDP on Linux; the `reuse_port` field is deprecated).
+- **Teardown does not restore live values:** Consistent with `ip_forward` and other host-level settings, teardown removes the persistence mechanisms (sysctl drop-in, `netdev` section from the boot oneshot) for `kernelMaxSocketBufferBytes` and `disableNicOffloads`. Live kernel values are not restored; the host reverts to its defaults on the next reboot.

api/v1alpha1/edgenode_types.go

@@ -163,6 +163,32 @@ type HealthCheckSpec struct {
 	UnhealthyThreshold int32 `json:"unhealthyThreshold,omitempty"`
 }
 
+// HostSpec tunes the VPS as a machine (kernel and NIC), independent of the
+// proxy (edge). These knobs shape the data path at the operating-system level:
+// the kernel socket-buffer ceiling and the underlay NIC offloads. They are
+// applied to the host during enrollment and reconciliation and are orthogonal
+// to spec.edge, which configures the Envoy proxy itself.
+type HostSpec struct {
+	// KernelMaxSocketBufferBytes is the socket-buffer ceiling. It is threaded to
+	// both the kernel sysctls (net.core.rmem_max/wmem_max) and the
+	// SO_RCVBUF/SO_SNDBUF of Envoy's UDP listeners. UDP does not autotune, so
+	// this single knob raises the effective buffer for high-throughput UDP while
+	// TCP autotunes up to the same ceiling. Defaults to 26214400 (25MB).
+	// +kubebuilder:default=26214400
+	// +kubebuilder:validation:Minimum=0
+	// +optional
+	KernelMaxSocketBufferBytes int64 `json:"kernelMaxSocketBufferBytes,omitempty"`
+
+	// DisableNicOffloads turns GRO/GSO off on the underlay NIC carrying the
+	// WireGuard transport. It defaults false (offloads enabled) because disabling
+	// them degrades throughput for ordinary TCP/QUIC traffic; enable it only for
+	// encapsulated/non-uniform UDP payloads where GRO coalescing can corrupt
+	// datagram boundaries.
+	// +kubebuilder:default=false
+	// +optional
+	DisableNicOffloads bool `json:"disableNicOffloads,omitempty"`
+}
+
 // EdgeNodeSpec defines the desired state of a EdgeNode: a single VPS acting
 // as a public relay plus the in-cluster uplink that terminates its tunnels.
 type EdgeNodeSpec struct {
@@ -174,6 +200,12 @@ type EdgeNodeSpec struct {
 	// +required
 	SSH SSHSpec `json:"ssh"`
 
+	// Host tunes the VPS as a machine (kernel and NIC), independent of the proxy.
+	// Optional; sane defaults apply. Placed between SSH and Edge by convention:
+	// host-level OS tuning sits below the proxy configuration.
+	// +optional
+	Host HostSpec `json:"host,omitempty"`
+
 	// Edge configures the VPS-side edge proxy: how it health checks the uplink
 	// replicas and fails over between them. Optional; sane defaults apply.
 	// +optional

cmd/tunnelctl/main.go

@@ -116,7 +116,10 @@ func cmdApply(args []string) error {
 	if err := agentrun.Apply(doc); err != nil {
 		return err
 	}
-	slog.Info("applied desired state", "interface", doc.WireGuard.Interface.Name, "nftables", doc.Nftables != nil)
+	slog.Info("applied desired state",
+		"interface", doc.WireGuard.Interface.Name,
+		"nftables", doc.Nftables != nil,
+		"netdev", doc.Netdev != nil)
 	return nil
 }
 

config/crd/bases/tunnel.achetronic.com_edgenodes.yaml

@@ -102,6 +102,33 @@ spec:
                         type: integer
                     type: object
                 type: object
+              host:
+                description: |-
+                  Host tunes the VPS as a machine (kernel and NIC), independent of the proxy.
+                  Optional; sane defaults apply. Placed between SSH and Edge by convention:
+                  host-level OS tuning sits below the proxy configuration.
+                properties:
+                  disableNicOffloads:
+                    default: false
+                    description: |-
+                      DisableNicOffloads turns GRO/GSO off on the underlay NIC carrying the
+                      WireGuard transport. It defaults false (offloads enabled) because disabling
+                      them degrades throughput for ordinary TCP/QUIC traffic; enable it only for
+                      encapsulated/non-uniform UDP payloads where GRO coalescing can corrupt
+                      datagram boundaries.
+                    type: boolean
+                  kernelMaxSocketBufferBytes:
+                    default: 26214400
+                    description: |-
+                      KernelMaxSocketBufferBytes is the socket-buffer ceiling. It is threaded to
+                      both the kernel sysctls (net.core.rmem_max/wmem_max) and the
+                      SO_RCVBUF/SO_SNDBUF of Envoy's UDP listeners. UDP does not autotune, so
+                      this single knob raises the effective buffer for high-throughput UDP while
+                      TCP autotunes up to the same ceiling. Defaults to 26214400 (25MB).
+                    format: int64
+                    minimum: 0
+                    type: integer
+                type: object
               ssh:
                 description: SSH holds the connection details used to enroll and manage
                   the VPS.