Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ once_cell = "1.21"
dashmap = "6"

# Networking
saorsa-transport = "0.35.3"
saorsa-transport = { git = "https://github.com/WithAutonomi/saorsa-transport.git", branch = "fix/pr136-provisional-relay" }

# Core-specific dependencies
dirs = "6.0"
Expand Down
4 changes: 2 additions & 2 deletions docs/ROUTING_TABLE_DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,10 +396,10 @@ The routing table MUST emit events on membership changes to allow consumers to r
|---|---|
| `PeerAdded(PeerId)` | New peer inserted into routing table |
| `PeerRemoved(PeerId)` | Peer evicted, blocked, or departed |
| `KClosestPeersChanged { old, new }` | Composition of the `K_BUCKET_SIZE`-closest peers to self changed |
| `KClosestPeersChanged { old, new, added, removed }` | Composition of the `K_BUCKET_SIZE`-closest peers to self changed |
| `BootstrapComplete { num_peers }` | Bootstrap process finished (routing table stabilized or timeout reached) |

`KClosestPeersChanged` is emitted when a routing table admission attempt causes the set of `K_BUCKET_SIZE` nearest peers to self to differ from the pre-admission set. The routing table snapshots the K-closest set before each admission attempt and compares after; the event carries both the old and new sets. This fires at most once per admission attempt — the entire admission (including sub-mutations like swaps and stale evictions) is treated as one logical operation.
`KClosestPeersChanged` is emitted when a routing table admission attempt causes the set of `K_BUCKET_SIZE` nearest peers to self to differ from the pre-admission set. The routing table snapshots the K-closest set before each admission attempt and compares after; the network event carries the old and new sets plus their precomputed `added` and `removed` differences. This fires at most once per admission attempt — the entire admission (including sub-mutations like swaps and stale evictions) is treated as one logical operation.

`BootstrapComplete` is emitted once per bootstrap cycle — both at initial startup and on each auto re-bootstrap (Section 10.3). It fires when the bootstrap lookups for that cycle complete — specifically, after the self-lookup and bucket refresh operations (Section 11) have all terminated. The event carries the total number of peers in the routing table at the time of emission. Consumers (e.g., replication, application-layer services) SHOULD wait for this event before initiating operations that depend on a populated routing table.

Expand Down
2 changes: 1 addition & 1 deletion docs/adr/ADR-014-proactive-relay-first-nat-traversal.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Status

Proposed
Superseded by [ADR-016](./ADR-016-canary-gated-proactive-relays.md)

## Context

Expand Down
166 changes: 166 additions & 0 deletions docs/adr/ADR-016-canary-gated-proactive-relays.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# ADR-016: Canary-Gated Proactive Relays

## Status

Accepted

Supersedes [ADR-014](./ADR-014-proactive-relay-first-nat-traversal.md).

## Context

ADR-014 described an earlier relay-first design. The implementation evolved in
several important ways:

- a relay allocation must not be published merely because the requesting node
can establish it;
- the canary protocol is necessarily a bounded public dial service, so its
abuse controls must not depend on a requester-supplied proof that the
requester can mint for itself;
- a canary dial must not reuse or disconnect a live application connection;
- relay state changes and network teardown must not block one another behind a
lifecycle mutex; and
- close-group churn is a replication concern, not evidence that a healthy
relay should be replaced.

This ADR records the implemented model and replaces the contradictory
thresholds, capacity limits, and maintenance behavior in ADR-014.

## Decision

### Relay acquisition and publication

Every non-client node may walk suitable routing-table peers and prepare one
proactive MASQUE allocation. Preparation creates a dedicated relay control
connection and a separate Quinn endpoint, but the allocation remains
provisional and absent from the node's published address set.

The target asks three randomized, non-close witnesses to probe the provisional
address using the unreleased `relay-canary-v1` request/response protocol. The
request contains the target peer ID, public relay socket, and an hourly witness
eligibility epoch. Its ordinary signed transport envelope must authenticate as
the same target peer ID, so a node can request a probe only for its own
identity.

Witness eligibility is deterministic and independent of the requested
address. A domain-separated BLAKE3 hash of the target peer ID, witness peer ID,
and eligibility epoch must have its first two bits clear. This assigns roughly
one quarter of witnesses to a target for an hour and prevents a requester from
recruiting the whole routing table for one identity. A witness accepts the
current or immediately previous epoch to tolerate an hour boundary; requesters
use the current epoch and filter candidates before selecting three randomized,
non-close witnesses.

After validating the request, an eligible witness opens a fresh one-shot
authenticated QUIC connection which never enters ordinary peer, address, or
dial-deduplication maps. The witness closes only that owned probe connection.
The wire response is deliberately coarse: success, failure, or rate limited.
Detailed dial and identity failures remain local debug information rather than
turning the protocol into a richer port-scanning oracle.

Admission requires three positive witness results. One explicit
canary-capable failure rejects the provisional allocation.

During the mixed-version rollout, a request that was successfully sent to a
selected witness but receives no canary-protocol response before the response
deadline counts as an assumed positive result. This preserves the pre-canary
behavior until that witness upgrades. This compatibility rule is deliberately
limited to the response stage: failure to connect to a selected witness and an
explicit rate-limit response remain ineligible; neither is promoted to
success. An assumed result is logged separately from a confirmed probe.

The implementation still requires three selectable non-close witnesses and
intentionally has no sparse-network threshold or replacement sampling.

Canary work has its own four-permit concurrency semaphore and hourly limits.
Before starting a dial, each witness consumes all of these budgets:

- at most 4 probes per authenticated target peer ID;
- at most 20 probes per transport source IPv4 address or IPv6 `/64` prefix;
- at most 4 probes per destination socket;
- at most 20 probes per destination IP address; and
- at most 60 probes in total on that witness.

The limits are intentionally redundant. Ephemeral identities cannot bypass the
source-network or witness-wide limits, while rotating destination ports cannot
bypass the destination-IP limit. The source IP is taken from the authenticated
transport connection, never from request data. Validation and budgets happen
before any canary-triggered network acquisition. Canary work does not consume
the general DHT handler budget and does not retry a failed cold dial. Replayed
requests consume the same hourly budgets as new requests.

### Established-relay maintenance

The node polls local tunnel health every five seconds and repeats independent
third-party canary verification every two hours, with deterministic initial
jitter spread across a full interval. The slower external cadence is
intentional: admission already proved reachability, tunnel loss is detected by
the cheap local health path, and every canary round creates three witness
requests plus three fresh PQC relay handshakes. The two-hour interval avoids
continuous fleet-wide dial pressure and remains comfortably inside the hourly
witness budgets.

Maintenance accepts two positive witness results, including temporary
assumed-positive legacy results, and rejects on two explicit canary-capable
failures. An inconclusive maintenance round retains the relay and waits for the
ordinary two-hour interval; immediately retrying unavailable witnesses would
amplify a partial outage. A rejected round withdraws the relay immediately; it
is not confirmed by a second round.

Tunnel death, explicit canary rejection, or an explicit trust/quality decision
may replace a relay. A healthy established relay remains in place when the
K-closest set changes. Close-group changes only publish the current
authoritative address set to peers newly entering the replication set.

### Publication and teardown ordering

On relay loss, local published-relay state is cleared first. DHT withdrawal and
transport teardown then run concurrently, so neither waits for the other.
Relay allocation resources are owned by a small lifecycle actor. The actor
serializes short state transitions; relay acquisition and teardown awaits run
outside it. Generation numbers prevent a late acquisition or canary verdict
from acting on a superseding allocation. Every owned allocation carries a
synchronous cleanup guard: if a lifecycle reply or graceful teardown future is
cancelled, dropping the owner closes the endpoint, aborts the tunnel tasks, and
removes the matching relay session.

Candidate `ADD_ADDRESS` advertisements are allowed while an allocation is
absent or provisional and suppressed only after the relay reaches the
`Published` state. Relay publication itself is owned by the authenticated,
sequenced DHT address-set path. Saorsa-core therefore does not forward or drain
transport `PeerAddressUpdated` events.

### Capacity and address-family ownership

Public relay servers accept at most four active relay clients. A prepared
allocation must preserve the address family of the selected relay path. A
mismatch is aborted through the same transport-stack owner that created it and
is returned as an error; later publication and teardown never redispatch an
allocation to a different stack.

### Packaging

Dependency versioning and release packaging are managed separately by the
release process and are not decided here.

## Consequences

- Published relay addresses have independent external reachability evidence
when selected witnesses support canaries; during mixed-version rollout an
unsupported selected witness temporarily contributes assumed-positive
compatibility credit.
- Canary traffic cannot tear down shared application/DHT connections.
- A malicious node can ask eligible witnesses to attempt a connection to an
unrelated public address, but the authenticated-self rule, deterministic
witness assignment, hourly peer/source/destination/global limits, and
isolated concurrency budget strictly bound that service. Canary work cannot
exhaust the general handler pool.
- Healthy relay sessions avoid churn when routing-table responsibility moves.
- DHT withdrawal begins without waiting for local transport shutdown.
- Mixed-version witnesses do not block admission merely because they lack the
canary protocol; their missing protocol response is temporarily counted as
positive.
- Routing tables with fewer than three selectable non-close witnesses can
still produce inconclusive admission.
- Canary requests carry no allocation receipt. This removes untrusted
self-signed proof material and several kilobytes of redundant ML-DSA key and
signature data from every request.
7 changes: 7 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@ An Architecture Decision Record (ADR) is a document that captures an important a
|-----|-------|--------|---------|
| [ADR-013](./ADR-013-no-offline-delivery-v1.md) | No Offline Message Delivery (v1) | Accepted | 1-hour TTL limit without extended offline delivery (future reconsideration) |

### Reachability

| ADR | Title | Status | Summary |
|-----|-------|--------|---------|
| [ADR-014](./ADR-014-proactive-relay-first-nat-traversal.md) | Proactive Relay-First NAT Traversal | Superseded | Initial proactive-relay design replaced by canary-gated publication |
| [ADR-016](./ADR-016-canary-gated-proactive-relays.md) | Canary-Gated Proactive Relays | Accepted | Signed allocation receipts, isolated witnesses, stable relay lifecycle, and sequenced publication |

### Operations

| ADR | Title | Status | Summary |
Expand Down
15 changes: 15 additions & 0 deletions src/adaptive/dht.rs
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,21 @@ impl AdaptiveDHT {
.peer_addresses_for_dial_typed(peer_id)
.await
}

/// Ensure the shared DHT dial coordinator has an authenticated channel.
///
/// Keeping application reconnects on the same path as iterative lookups
/// means both callers share address-failure suppression and never create
/// independent retry loops against a known-bad relay.
pub(crate) async fn ensure_peer_channel(
&self,
peer_id: &PeerId,
candidates: &[(MultiAddr, AddressType)],
) -> Result<()> {
self.dht_manager
.ensure_peer_channel(peer_id, candidates)
.await
}
}

#[cfg(test)]
Expand Down
Loading
Loading