|
| 1 | +# ADR-016: Canary-Gated Proactive Relays |
| 2 | + |
| 3 | +## Status |
| 4 | + |
| 5 | +Accepted |
| 6 | + |
| 7 | +Supersedes [ADR-014](./ADR-014-proactive-relay-first-nat-traversal.md). |
| 8 | + |
| 9 | +## Context |
| 10 | + |
| 11 | +ADR-014 described an earlier relay-first design. The implementation evolved in |
| 12 | +several important ways: |
| 13 | + |
| 14 | +- a relay allocation must not be published merely because the requesting node |
| 15 | + can establish it; |
| 16 | +- the canary protocol is necessarily a bounded public dial service, so its |
| 17 | + abuse controls must not depend on a requester-supplied proof that the |
| 18 | + requester can mint for itself; |
| 19 | +- a canary dial must not reuse or disconnect a live application connection; |
| 20 | +- relay state changes and network teardown must not block one another behind a |
| 21 | + lifecycle mutex; and |
| 22 | +- close-group churn is a replication concern, not evidence that a healthy |
| 23 | + relay should be replaced. |
| 24 | + |
| 25 | +This ADR records the implemented model and replaces the contradictory |
| 26 | +thresholds, capacity limits, and maintenance behavior in ADR-014. |
| 27 | + |
| 28 | +## Decision |
| 29 | + |
| 30 | +### Relay acquisition and publication |
| 31 | + |
| 32 | +Every non-client node may walk suitable routing-table peers and prepare one |
| 33 | +proactive MASQUE allocation. Preparation creates a dedicated relay control |
| 34 | +connection and a separate Quinn endpoint, but the allocation remains |
| 35 | +provisional and absent from the node's published address set. |
| 36 | + |
| 37 | +The target asks three randomized, non-close witnesses to probe the provisional |
| 38 | +address using the unreleased `relay-canary-v1` request/response protocol. The |
| 39 | +request contains the target peer ID, public relay socket, and an hourly witness |
| 40 | +eligibility epoch. Its ordinary signed transport envelope must authenticate as |
| 41 | +the same target peer ID, so a node can request a probe only for its own |
| 42 | +identity. |
| 43 | + |
| 44 | +Witness eligibility is deterministic and independent of the requested |
| 45 | +address. A domain-separated BLAKE3 hash of the target peer ID, witness peer ID, |
| 46 | +and eligibility epoch must have its first two bits clear. This assigns roughly |
| 47 | +one quarter of witnesses to a target for an hour and prevents a requester from |
| 48 | +recruiting the whole routing table for one identity. A witness accepts the |
| 49 | +current or immediately previous epoch to tolerate an hour boundary; requesters |
| 50 | +use the current epoch and filter candidates before selecting three randomized, |
| 51 | +non-close witnesses. |
| 52 | + |
| 53 | +After validating the request, an eligible witness opens a fresh one-shot |
| 54 | +authenticated QUIC connection which never enters ordinary peer, address, or |
| 55 | +dial-deduplication maps. The witness closes only that owned probe connection. |
| 56 | +The wire response is deliberately coarse: success, failure, or rate limited. |
| 57 | +Detailed dial and identity failures remain local debug information rather than |
| 58 | +turning the protocol into a richer port-scanning oracle. |
| 59 | + |
| 60 | +Admission requires three positive witness results. One explicit |
| 61 | +canary-capable failure rejects the provisional allocation. |
| 62 | + |
| 63 | +During the mixed-version rollout, a request that was successfully sent to a |
| 64 | +selected witness but receives no canary-protocol response before the response |
| 65 | +deadline counts as an assumed positive result. This preserves the pre-canary |
| 66 | +behavior until that witness upgrades. This compatibility rule is deliberately |
| 67 | +limited to the response stage: failure to connect to a selected witness and an |
| 68 | +explicit rate-limit response remain ineligible; neither is promoted to |
| 69 | +success. An assumed result is logged separately from a confirmed probe. |
| 70 | + |
| 71 | +The implementation still requires three selectable non-close witnesses and |
| 72 | +intentionally has no sparse-network threshold or replacement sampling. |
| 73 | + |
| 74 | +Canary work has its own four-permit concurrency semaphore and hourly limits. |
| 75 | +Before starting a dial, each witness consumes all of these budgets: |
| 76 | + |
| 77 | +- at most 4 probes per authenticated target peer ID; |
| 78 | +- at most 20 probes per transport source IPv4 address or IPv6 `/64` prefix; |
| 79 | +- at most 4 probes per destination socket; |
| 80 | +- at most 20 probes per destination IP address; and |
| 81 | +- at most 60 probes in total on that witness. |
| 82 | + |
| 83 | +The limits are intentionally redundant. Ephemeral identities cannot bypass the |
| 84 | +source-network or witness-wide limits, while rotating destination ports cannot |
| 85 | +bypass the destination-IP limit. The source IP is taken from the authenticated |
| 86 | +transport connection, never from request data. Validation and budgets happen |
| 87 | +before any canary-triggered network acquisition. Canary work does not consume |
| 88 | +the general DHT handler budget and does not retry a failed cold dial. Replayed |
| 89 | +requests consume the same hourly budgets as new requests. |
| 90 | + |
| 91 | +### Established-relay maintenance |
| 92 | + |
| 93 | +The node polls local tunnel health every five seconds and repeats independent |
| 94 | +third-party canary verification every two hours, with deterministic initial |
| 95 | +jitter spread across a full interval. The slower external cadence is |
| 96 | +intentional: admission already proved reachability, tunnel loss is detected by |
| 97 | +the cheap local health path, and every canary round creates three witness |
| 98 | +requests plus three fresh PQC relay handshakes. The two-hour interval avoids |
| 99 | +continuous fleet-wide dial pressure and remains comfortably inside the hourly |
| 100 | +witness budgets. |
| 101 | + |
| 102 | +Maintenance accepts two positive witness results, including temporary |
| 103 | +assumed-positive legacy results, and rejects on two explicit canary-capable |
| 104 | +failures. An inconclusive maintenance round retains the relay and waits for the |
| 105 | +ordinary two-hour interval; immediately retrying unavailable witnesses would |
| 106 | +amplify a partial outage. A rejected round withdraws the relay immediately; it |
| 107 | +is not confirmed by a second round. |
| 108 | + |
| 109 | +Tunnel death, explicit canary rejection, or an explicit trust/quality decision |
| 110 | +may replace a relay. A healthy established relay remains in place when the |
| 111 | +K-closest set changes. Close-group changes only publish the current |
| 112 | +authoritative address set to peers newly entering the replication set. |
| 113 | + |
| 114 | +### Publication and teardown ordering |
| 115 | + |
| 116 | +On relay loss, local published-relay state is cleared first. DHT withdrawal and |
| 117 | +transport teardown then run concurrently, so neither waits for the other. |
| 118 | +Relay allocation resources are owned by a small lifecycle actor. The actor |
| 119 | +serializes short state transitions; relay acquisition and teardown awaits run |
| 120 | +outside it. Generation numbers prevent a late acquisition or canary verdict |
| 121 | +from acting on a superseding allocation. Every owned allocation carries a |
| 122 | +synchronous cleanup guard: if a lifecycle reply or graceful teardown future is |
| 123 | +cancelled, dropping the owner closes the endpoint, aborts the tunnel tasks, and |
| 124 | +removes the matching relay session. |
| 125 | + |
| 126 | +Candidate `ADD_ADDRESS` advertisements are allowed while an allocation is |
| 127 | +absent or provisional and suppressed only after the relay reaches the |
| 128 | +`Published` state. Relay publication itself is owned by the authenticated, |
| 129 | +sequenced DHT address-set path. Saorsa-core therefore does not forward or drain |
| 130 | +transport `PeerAddressUpdated` events. |
| 131 | + |
| 132 | +### Capacity and address-family ownership |
| 133 | + |
| 134 | +Public relay servers accept at most four active relay clients. A prepared |
| 135 | +allocation must preserve the address family of the selected relay path. A |
| 136 | +mismatch is aborted through the same transport-stack owner that created it and |
| 137 | +is returned as an error; later publication and teardown never redispatch an |
| 138 | +allocation to a different stack. |
| 139 | + |
| 140 | +### Packaging |
| 141 | + |
| 142 | +Dependency versioning and release packaging are managed separately by the |
| 143 | +release process and are not decided here. |
| 144 | + |
| 145 | +## Consequences |
| 146 | + |
| 147 | +- Published relay addresses have independent external reachability evidence |
| 148 | + when selected witnesses support canaries; during mixed-version rollout an |
| 149 | + unsupported selected witness temporarily contributes assumed-positive |
| 150 | + compatibility credit. |
| 151 | +- Canary traffic cannot tear down shared application/DHT connections. |
| 152 | +- A malicious node can ask eligible witnesses to attempt a connection to an |
| 153 | + unrelated public address, but the authenticated-self rule, deterministic |
| 154 | + witness assignment, hourly peer/source/destination/global limits, and |
| 155 | + isolated concurrency budget strictly bound that service. Canary work cannot |
| 156 | + exhaust the general handler pool. |
| 157 | +- Healthy relay sessions avoid churn when routing-table responsibility moves. |
| 158 | +- DHT withdrawal begins without waiting for local transport shutdown. |
| 159 | +- Mixed-version witnesses do not block admission merely because they lack the |
| 160 | + canary protocol; their missing protocol response is temporarily counted as |
| 161 | + positive. |
| 162 | +- Routing tables with fewer than three selectable non-close witnesses can |
| 163 | + still produce inconclusive admission. |
| 164 | +- Canary requests carry no allocation receipt. This removes untrusted |
| 165 | + self-signed proof material and several kilobytes of redundant ML-DSA key and |
| 166 | + signature data from every request. |
0 commit comments