Make Hetzner private network readiness explicit

Author: dmorn

.scratch/transport-layer-foundation/issues/12-add-reference-quic-benchmark-scripts.md

@@ -82,3 +82,8 @@ from that smoke-test experience.
   infrastructure decision before reference-comparison runs is #25: either make
   Hetzner private-network readiness deterministic or scope #12 explicitly to
   public IPv4 paths for the first implementation.
+- 2026-05-21: Follow-up #25 is closed: Hetzner private-network readiness is now
+  an explicit operator step with static guest netplan config and
+  `just bench-transport-private-check`, validated by smoke
+  `20260521T093427Z-private-smoke`. #12 can now design reference-comparison
+  runtime command contracts against both public IPv4 and private-network paths.

.scratch/transport-layer-foundation/issues/25-fix-hetzner-private-network-readiness.md

@@ -1,6 +1,6 @@
 # Fix Hetzner private-network readiness
 
-Status: needs-triage
+Status: closed
 Type: AFK
 
 ## Parent
@@ -25,22 +25,52 @@ The public IPv4 path worked and produced valid benchmark JSONL.
 
 ## Acceptance criteria
 
-- [ ] After `terraform apply`, both nodes expose their configured private IPs
+- [x] After `terraform apply`, both nodes expose their configured private IPs
       on an UP interface.
-- [ ] A smoke check proves client-to-server ICMP and TCP connectivity over the
+- [x] A smoke check proves client-to-server ICMP and TCP connectivity over the
       private IPs before benchmark traffic starts.
-- [ ] The Hetzner README documents the private-path readiness check.
-- [ ] If explicit OS network configuration is needed, it is handled by
+- [x] The Hetzner README documents the private-path readiness check.
+- [x] If explicit OS network configuration is needed, it is handled by
       Terraform/cloud-init without making cloud-init large or fragile.
-- [ ] If private paths are intentionally deferred, the README and outputs make
-      that limitation clear.
+- [x] If private paths are intentionally deferred, the README and outputs make
+      that limitation clear. Not applicable: private paths are supported after
+      the readiness check passes.
 
 ## Blocked by
 
 None.
 
+## Resolution
+
+Implemented private-network readiness as a first-class Hetzner operator step:
+
+- Cloud-init writes a small static netplan file for the first Hetzner private
+  NIC when private networking is enabled.
+- The private NIC defaults to `enp7s0`, uses the Terraform-assigned private IP
+  as `/32`, MTU 1450, and routes the private network CIDR via the subnet
+  gateway.
+- Cloud-init stops and masks `hc-net-ifup@enp7s0.service`, flushes any early
+  global address on the private NIC, then applies netplan.
+- `just bench-transport-private-check` waits for cloud-init on both nodes,
+  confirms peer routes, pings the server private IP from the client, and runs a
+  one-second TCP `iperf3` probe over the private IP.
+- Terraform outputs now include `private_network_check_command` so operators
+  can discover the required readiness probe from the applied run.
+
 ## Comments
 
 - 2026-05-20: Created from Hetzner smoke `20260520T134420Z-smoke`. Public IPv4
   benchmark traffic is usable; private-network benchmark traffic is not yet a
   reliable operator path.
+- 2026-05-21: Implementation started. Direction: configure the first Hetzner
+  private NIC explicitly in cloud-init using static netplan and add a
+  `just bench-transport-private-check` readiness probe that proves peer route,
+  ICMP, and TCP connectivity before private-path benchmarks.
+- 2026-05-21: Closed after Hetzner ARM smoke
+  `20260521T093427Z-private-smoke`. The client and server both exposed
+  `enp7s0` as UP with `10.88.0.11/32` and `10.88.0.12/32`, respectively, and
+  routes to the peer private IPs via `10.88.0.1`. The readiness check proved
+  client-to-server ICMP with 3/3 packets delivered and TCP with
+  `iperf3 --client 10.88.0.12 --port 55209 --time 1`. The Terraform pair was
+  destroyed afterward, and `just bench-transport-verify-clean` confirmed no
+  Terraform state entries or labelled Hetzner resources remained.

bench/transport/README.md

@@ -703,6 +703,17 @@ Terraform outputs include path metadata for public IPv4 and private-network
 runs. Benchmark tools should merge those outputs with live host inventory and
 run-specific metrics.
 
+Before using private-network path metadata, run:
+
+```bash
+just bench-transport-private-check
+```
+
+The check proves that both nodes have their configured private IPs ready and
+that the client can reach the server private IP over ICMP and TCP. Treat private
+path benchmark results as invalid if this readiness check has not passed for
+the same Terraform run.
+
 ## Implementation Order
 
 The intended issue order is:

bench/transport/infra/hetzner/README.md

@@ -40,6 +40,18 @@ Check cloud-init and the installed tools before running benchmarks:
 terraform output -json toolchain_check_commands
 ```
 
+When the private network is enabled, prove private-path readiness before using
+private IPs for benchmarks:
+
+```bash
+just bench-transport-private-check
+```
+
+The check waits for cloud-init on both nodes, verifies that each node has a
+route to its peer private IP, pings the server private IP from the client, and
+runs a one-second `iperf3` TCP probe bound to the server private IP. Do not use
+`path_metadata_private` for benchmark results until this check passes.
+
 Destroy the pair when the run is finished:
 
 ```bash
@@ -84,11 +96,19 @@ allowed.
 
 Cloud-init intentionally does little:
 
+- writes a static netplan file for the first Hetzner private NIC when private
+  networking is enabled;
 - installs build tools, `iperf3`, and small shell utilities;
 - installs Go from the official Linux archive at `go.dev/dl`;
 - installs Erlang/OTP and Elixir with the official Elixir install script;
 - writes a short note under `/opt/moqx-bench/`.
 
+For the current CAX/CCX profiles, the first Hetzner private interface is
+configured as `enp7s0` with the assigned private IP as `/32`, MTU 1450, and a
+route to the private network CIDR via the network gateway. The Terraform
+variable `private_network_interface` exists so a future profile can override
+the guest interface name if needed.
+
 The benchmark repo is not cloned automatically, and no benchmark process is
 started automatically. Deploy a `moqx-transport-bench` release artifact or use
 the installed Elixir/Mix toolchain for development-only checks after cloud-init

bench/transport/infra/hetzner/cloud-init.yaml.tftpl

@@ -18,12 +18,49 @@ write_files:
       Provisioning installs only the base build toolchain, iperf3, Go, Erlang,
       and Elixir. Deploy a moqx-transport-bench release artifact when you are
       ready to run a benchmark.
+%{ if private_network_enabled }
+  - path: /etc/netplan/60-moqx-bench-private-network.yaml
+    permissions: "0644"
+    content: |
+      network:
+        version: 2
+        renderer: networkd
+        ethernets:
+          ${private_network_interface}:
+            dhcp4: false
+            mtu: 1450
+            addresses:
+              - ${private_ip}/32
+            routes:
+              - to: ${private_network_cidr}
+                via: ${private_gateway_ip}
+                on-link: true
+%{ endif }
 
 runcmd:
   - |
       set -eu
       export HOME=/root
       export DEBIAN_FRONTEND=noninteractive
+%{ if private_network_enabled }
+      private_iface="${private_network_interface}"
+
+      for _ in $(seq 1 30); do
+        if ip link show dev "$private_iface" >/dev/null 2>&1; then
+          break
+        fi
+        sleep 1
+      done
+
+      ip link show dev "$private_iface"
+      systemctl stop "hc-net-ifup@$private_iface.service" || true
+      systemctl mask "hc-net-ifup@$private_iface.service" || true
+      ip address flush dev "$private_iface" scope global || true
+      netplan generate
+      netplan apply
+      ip -4 address show dev "$private_iface"
+      ip route get "${private_gateway_ip}"
+%{ endif }
       apt-get update
       apt-get install -y \
         autoconf \

bench/transport/infra/hetzner/main.tf

@@ -139,6 +139,12 @@ resource "hcloud_server" "node" {
     otp_version    = var.otp_version
     elixir_version = var.elixir_version
     go_version     = var.go_version
+
+    private_network_enabled   = var.enable_private_network
+    private_network_interface = var.private_network_interface
+    private_network_cidr      = var.private_network_cidr
+    private_gateway_ip        = cidrhost(var.private_subnet_cidr, 1)
+    private_ip                = each.value.private_ip
   })
 
   firewall_ids = [hcloud_firewall.operator.id]

bench/transport/infra/hetzner/outputs.tf

@@ -81,3 +81,8 @@ output "toolchain_check_commands" {
     "ssh root@${server.ipv4_address} 'cloud-init status --wait && go version && elixir --version && iperf3 --version | head -1'"
   }
 }
+
+output "private_network_check_command" {
+  description = "Read-only recipe that proves private ICMP and TCP connectivity between the benchmark endpoints."
+  value       = var.enable_private_network ? "just bench-transport-private-check ${var.run_id}" : null
+}

bench/transport/infra/hetzner/variables.tf

@@ -94,6 +94,12 @@ variable "private_network_zone" {
   default     = "eu-central"
 }
 
+variable "private_network_interface" {
+  description = "Guest OS interface name for the first attached Hetzner private network on the selected server families."
+  type        = string
+  default     = "enp7s0"
+}
+
 variable "client_private_ip" {
   description = "Private IP for the client endpoint when enable_private_network is true."
   type        = string

justfile

@@ -129,6 +129,58 @@ bench-transport-destroy run_id=current_run profile="arm-smoke":
 bench-transport-outputs:
     cd "{{ infra_dir }}" && terraform output
 
+# Prove private-network readiness with ICMP and TCP before benchmark traffic.
+bench-transport-private-check run_id=current_run port="55209":
+    #!/usr/bin/env bash
+    set -euo pipefail
+
+    if [ -z "{{ run_id }}" ]; then
+      printf '%s\n' 'Missing run_id. Run `just bench-transport-new-run` first or pass run_id explicitly.' >&2
+      exit 2
+    fi
+
+    key="{{ bench_dir }}/.keys/{{ run_id }}/id_ed25519"
+    test -f "$key" || {
+      printf 'Missing SSH key for run %s\n' "{{ run_id }}" >&2
+      exit 2
+    }
+
+    servers_json="$(cd "{{ infra_dir }}" && terraform output -json servers)"
+    client_public="$(printf '%s' "$servers_json" | jq -r '.client.public_ipv4')"
+    server_public="$(printf '%s' "$servers_json" | jq -r '.server.public_ipv4')"
+    client_private="$(printf '%s' "$servers_json" | jq -r '.client.private_ip // empty')"
+    server_private="$(printf '%s' "$servers_json" | jq -r '.server.private_ip // empty')"
+
+    if [ -z "$client_private" ] || [ -z "$server_private" ]; then
+      printf '%s\n' 'Terraform outputs do not include private IPs. Private network is disabled or not applied.' >&2
+      exit 2
+    fi
+
+    ssh_opts=(
+      -i "$key"
+      -o IdentitiesOnly=yes
+      -o StrictHostKeyChecking=accept-new
+      -o UserKnownHostsFile="{{ bench_dir }}/.keys/{{ run_id }}/known_hosts"
+    )
+
+    client="root@$client_public"
+    server="root@$server_public"
+    remote_log="/tmp/moqx-private-iperf3-{{ port }}.log"
+
+    cleanup() {
+      ssh "${ssh_opts[@]}" "$server" "pkill -f 'iperf3 .*--port {{ port }}' >/dev/null 2>&1 || true" >/dev/null 2>&1 || true
+    }
+    trap cleanup EXIT
+
+    ssh "${ssh_opts[@]}" "$client" "cloud-init status --wait >/dev/null && ip -4 address show && ip route get '$server_private'"
+    ssh "${ssh_opts[@]}" "$server" "cloud-init status --wait >/dev/null && ip -4 address show && ip route get '$client_private'"
+    ssh "${ssh_opts[@]}" "$server" "nohup iperf3 --server --bind '$server_private' --port '{{ port }}' --one-off > '$remote_log' 2>&1 &"
+    sleep 1
+    ssh "${ssh_opts[@]}" "$client" "ping -c 3 -W 2 '$server_private'"
+    ssh "${ssh_opts[@]}" "$client" "iperf3 --client '$server_private' --port '{{ port }}' --time 1 --json >/tmp/moqx-private-check.json"
+
+    printf 'Private network ready: %s -> %s over ICMP and TCP port %s\n' "$client_private" "$server_private" "{{ port }}"
+
 # Verify Terraform state and Hetzner labelled resources are clean after destroy.
 bench-transport-verify-clean:
     #!/usr/bin/env bash