Skip to content

Commit 9e6fb36

Browse files
mickvandijkeclaude
andcommitted
docs(adr): add ADR-0002 client-side fallback for full-node shunning
Client-side role in the full-node-shunning plan: classify node rejections (full vs price-floor vs transport) instead of flattening to RemotePut, and extend put fallback beyond the quoted close group to next-closest peers within the K=20 issuer-closeness window, reusing the same ProofOfPayment (no re-quote, no re-pay). Quorum stays at 4; reads rely on close-group convergence (see saorsa-node ADR-0003). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 3ab50c4 commit 9e6fb36

1 file changed

Lines changed: 134 additions & 0 deletions

File tree

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# ADR-0002: Client-side fallback and diagnostics for full-node shunning
2+
3+
- **Status:** Proposed
4+
- **Date:** 2026-06-25
5+
- **Decision owners:** Mick
6+
- **Reviewers:** <pending>
7+
- **Supersedes:** none
8+
- **Superseded by:** none
9+
- **Related:** saorsa-node ADR-0003 (node-side full-node detection, penalisation, and eviction); ADR-0001 (adopt ADRs)
10+
11+
## Context
12+
13+
Full storage nodes are rejecting client uploads. When a chunk's close group
14+
contains a full node, the client's put to that node fails and the upload can fall
15+
short of write quorum, even though the network as a whole still has capacity. This
16+
ADR covers the **client's** role in the wider full-node-shunning plan; the node and
17+
membership roles are covered by saorsa-node ADR-0003.
18+
19+
The decision rests on behaviour verified directly in the node and client code, not
20+
on assumption:
21+
22+
- The client selects peers via a DHT lookup of the `CLOSE_GROUP_SIZE = 7` closest,
23+
and write quorum is `CLOSE_GROUP_MAJORITY = 4` (`ant-core/src/data/client/chunk.rs:308`).
24+
Both constants come from `ant-protocol` and are not client-configurable.
25+
- Today the client only falls back **among the 7 quoted peers**
26+
(`ant-core/src/data/client/chunk.rs:273-358`).
27+
- A node accepts a client put from **any** peer whose own local 20-closest view of
28+
the address includes one of the proof's quote issuers; it does **not** require the
29+
receiving node to have been quoted (`saorsa-node` `src/payment/verifier.rs:811-837,
30+
942-1003`; `src/storage/handler.rs:283-285`). The issuer-closeness width is
31+
`PAID_QUOTE_ISSUER_CLOSENESS_WIDTH = K_BUCKET_SIZE = 20`, nearly 3× the close
32+
group. **Consequence: the same `ProofOfPayment` is reusable by further peers
33+
within that 20-wide window — fallback needs no re-quote and no re-pay.**
34+
- A full node returns a **distinct** `ProtocolError::StorageFailed` *before* payment
35+
verification (`saorsa-node` `src/storage/handler.rs:274-281`), whereas a price-floor
36+
shortfall returns a `Payment` error. The client currently **flattens both** into
37+
`Error::RemotePut` (`ant-core/src/data/client/chunk.rs:417-443`), losing the
38+
distinction it needs to respond correctly.
39+
- GET returns on **first success** and is read-safe while at least one queried close
40+
peer holds the chunk; it does **not** expand its walk beyond K
41+
(`ant-core/src/data/client/chunk.rs:483-623`).
42+
43+
## Decision Drivers
44+
45+
- Unblock uploads when a *minority* of the close group is full, with no protocol or
46+
payment changes.
47+
- Make the cause of a put failure legible so the client picks the right response
48+
(fall back, skip, or retry) instead of one opaque error.
49+
- Stay strictly within the node's verified acceptance rules; never assume behaviour
50+
the node does not implement.
51+
- Stay read-safe in the common case and flag the near-capacity boundary explicitly
52+
rather than silently relying on it.
53+
54+
## Considered Options
55+
56+
1. **Do nothing on the client; rely solely on node-side eviction to reshape close
57+
groups.** Rejected: leaves an immediate failure during the eviction convergence
58+
window and wastes the already-available 20-wide acceptance headroom.
59+
2. **Re-quote and re-pay a fresh close group whenever a put fails.** Rejected
60+
outright, and a deliberate non-goal: it is unnecessary now that the proof is known
61+
to be reusable, and **we do not re-quote or top-up payment** under any failure. A
62+
peer whose local floor the existing payment does not clear is simply skipped.
63+
3. **Extend client fallback to next-closest peers within the issuer-closeness
64+
window, reusing the existing proof, with error-class-aware handling (chosen).**
65+
66+
## Decision
67+
68+
- **Classify node rejections** instead of collapsing them into one `RemotePut`:
69+
`StorageFailed` (full) → fall back to a further peer; `Payment`/price-floor → this
70+
peer wants more than was paid, so **skip it and advance fallback** (we do **not**
71+
re-quote or top-up payment, and do not retry the same peer); transport/timeout →
72+
bounded retry.
73+
- **Extend `chunk_put_to_close_group` fallback** from the 7 quoted peers to the
74+
next-closest peers from the same DHT walk, **reusing the same `ProofOfPayment`**,
75+
bounded by the `K_BUCKET_SIZE = 20` issuer-closeness window — beyond it the node
76+
provably rejects ("issuer not among this node's local K=20 closest"). Fallback is
77+
**best-effort**: whether a given far peer accepts depends on its own routing view
78+
and local price floor, which the client cannot see.
79+
- **Keep write quorum at `CLOSE_GROUP_MAJORITY = 4`.** Fallback changes *which* peers
80+
satisfy quorum, not the threshold.
81+
- **Read availability:** rely on close-group convergence driven by node/membership
82+
eviction (saorsa-node ADR-0003) as the primary guarantee; keep GET-on-first-success
83+
plus the existing single retry. Do **not** widen the GET walk now. Explicitly flag
84+
the near-capacity read-gap — a chunk that lands *entirely* on fallback peers outside
85+
the queried 7 is unreadable until convergence shifts those peers into the close
86+
group — and gate any future GET-walk widening behind observed GET shortfalls in the
87+
near-capacity regime.
88+
89+
## Consequences
90+
91+
### Positive
92+
93+
- Uploads blocked by a single (or minority) full close peer now reach quorum via
94+
fallback, with no protocol, payment, or node change required.
95+
- Failure causes are legible: full, under-paid, and transport failures get distinct,
96+
correct client responses.
97+
98+
### Negative / Trade-offs
99+
100+
- Fallback is best-effort and **bounded at the 20-wide window**; if more than
101+
`20 − 4` of the nearest peers refuse, quorum still cannot be met — the same
102+
near-capacity boundary the whole plan is bounded by.
103+
- A **per-node price floor** can reject a fully-paid put even with free disk; because
104+
we do not re-quote or top-up, such a peer is simply skipped — so a neighbourhood
105+
priced above the paid median can still cause a quorum shortfall this ADR does not
106+
resolve.
107+
- The near-capacity read-gap remains until convergence; we accept it for now rather
108+
than widen GET prematurely.
109+
- More client-side branching and a retry/fallback budget to tune.
110+
111+
### Neutral / Operational
112+
113+
- Quorum and close-group constants remain owned by `ant-protocol`; this ADR does not
114+
change them.
115+
- Adds one tunable: the fallback peer budget (how far down the next-closest list the
116+
client will try before giving up).
117+
118+
## Validation
119+
120+
- In a minority-full testnet, uploads that previously failed on a single full close
121+
peer reach quorum via fallback **without re-quoting or re-paying**.
122+
- Tests required before this ADR is Accepted: error classification maps
123+
`StorageFailed` / `Payment` / transport correctly; fallback stops at the 20-window;
124+
quorum still requires 4 successes; a price-floor rejection skips the peer and
125+
advances fallback without re-quoting or retrying the same peer; GET stays read-safe
126+
while ≥1 queried close peer holds the chunk.
127+
- Re-open trigger: if observed GET shortfalls reveal a persistent read-gap in the
128+
near-capacity regime, revisit widening the GET walk.
129+
130+
## Notes for AI-assisted work
131+
132+
AI tools may help draft this ADR, but **must not mark it Accepted without human
133+
review**. Accepted ADRs are immutable: create a new superseding ADR rather than
134+
editing this one.

0 commit comments

Comments
 (0)