|
| 1 | +# ADR-0002: Client-side fallback and diagnostics for full-node shunning |
| 2 | + |
| 3 | +- **Status:** Proposed |
| 4 | +- **Date:** 2026-06-25 |
| 5 | +- **Decision owners:** Mick |
| 6 | +- **Reviewers:** <pending> |
| 7 | +- **Supersedes:** none |
| 8 | +- **Superseded by:** none |
| 9 | +- **Related:** saorsa-node ADR-0003 (node-side full-node detection, penalisation, and eviction); ADR-0001 (adopt ADRs) |
| 10 | + |
| 11 | +## Context |
| 12 | + |
| 13 | +Full storage nodes are rejecting client uploads. When a chunk's close group |
| 14 | +contains a full node, the client's put to that node fails and the upload can fall |
| 15 | +short of write quorum, even though the network as a whole still has capacity. This |
| 16 | +ADR covers the **client's** role in the wider full-node-shunning plan; the node and |
| 17 | +membership roles are covered by saorsa-node ADR-0003. |
| 18 | + |
| 19 | +The decision rests on behaviour verified directly in the node and client code, not |
| 20 | +on assumption: |
| 21 | + |
| 22 | +- The client selects peers via a DHT lookup of the `CLOSE_GROUP_SIZE = 7` closest, |
| 23 | + and write quorum is `CLOSE_GROUP_MAJORITY = 4` (`ant-core/src/data/client/chunk.rs:308`). |
| 24 | + Both constants come from `ant-protocol` and are not client-configurable. |
| 25 | +- Today the client only falls back **among the 7 quoted peers** |
| 26 | + (`ant-core/src/data/client/chunk.rs:273-358`). |
| 27 | +- A node accepts a client put from **any** peer whose own local 20-closest view of |
| 28 | + the address includes one of the proof's quote issuers; it does **not** require the |
| 29 | + receiving node to have been quoted (`saorsa-node` `src/payment/verifier.rs:811-837, |
| 30 | + 942-1003`; `src/storage/handler.rs:283-285`). The issuer-closeness width is |
| 31 | + `PAID_QUOTE_ISSUER_CLOSENESS_WIDTH = K_BUCKET_SIZE = 20`, nearly 3× the close |
| 32 | + group. **Consequence: the same `ProofOfPayment` is reusable by further peers |
| 33 | + within that 20-wide window — fallback needs no re-quote and no re-pay.** |
| 34 | +- A full node returns a **distinct** `ProtocolError::StorageFailed` *before* payment |
| 35 | + verification (`saorsa-node` `src/storage/handler.rs:274-281`), whereas a price-floor |
| 36 | + shortfall returns a `Payment` error. The client currently **flattens both** into |
| 37 | + `Error::RemotePut` (`ant-core/src/data/client/chunk.rs:417-443`), losing the |
| 38 | + distinction it needs to respond correctly. |
| 39 | +- GET returns on **first success** and is read-safe while at least one queried close |
| 40 | + peer holds the chunk; it does **not** expand its walk beyond K |
| 41 | + (`ant-core/src/data/client/chunk.rs:483-623`). |
| 42 | + |
| 43 | +## Decision Drivers |
| 44 | + |
| 45 | +- Unblock uploads when a *minority* of the close group is full, with no protocol or |
| 46 | + payment changes. |
| 47 | +- Make the cause of a put failure legible so the client picks the right response |
| 48 | + (fall back, skip, or retry) instead of one opaque error. |
| 49 | +- Stay strictly within the node's verified acceptance rules; never assume behaviour |
| 50 | + the node does not implement. |
| 51 | +- Stay read-safe in the common case and flag the near-capacity boundary explicitly |
| 52 | + rather than silently relying on it. |
| 53 | + |
| 54 | +## Considered Options |
| 55 | + |
| 56 | +1. **Do nothing on the client; rely solely on node-side eviction to reshape close |
| 57 | + groups.** Rejected: leaves an immediate failure during the eviction convergence |
| 58 | + window and wastes the already-available 20-wide acceptance headroom. |
| 59 | +2. **Re-quote and re-pay a fresh close group whenever a put fails.** Rejected |
| 60 | + outright, and a deliberate non-goal: it is unnecessary now that the proof is known |
| 61 | + to be reusable, and **we do not re-quote or top-up payment** under any failure. A |
| 62 | + peer whose local floor the existing payment does not clear is simply skipped. |
| 63 | +3. **Extend client fallback to next-closest peers within the issuer-closeness |
| 64 | + window, reusing the existing proof, with error-class-aware handling (chosen).** |
| 65 | + |
| 66 | +## Decision |
| 67 | + |
| 68 | +- **Classify node rejections** instead of collapsing them into one `RemotePut`: |
| 69 | + `StorageFailed` (full) → fall back to a further peer; `Payment`/price-floor → this |
| 70 | + peer wants more than was paid, so **skip it and advance fallback** (we do **not** |
| 71 | + re-quote or top-up payment, and do not retry the same peer); transport/timeout → |
| 72 | + bounded retry. |
| 73 | +- **Extend `chunk_put_to_close_group` fallback** from the 7 quoted peers to the |
| 74 | + next-closest peers from the same DHT walk, **reusing the same `ProofOfPayment`**, |
| 75 | + bounded by the `K_BUCKET_SIZE = 20` issuer-closeness window — beyond it the node |
| 76 | + provably rejects ("issuer not among this node's local K=20 closest"). Fallback is |
| 77 | + **best-effort**: whether a given far peer accepts depends on its own routing view |
| 78 | + and local price floor, which the client cannot see. |
| 79 | +- **Keep write quorum at `CLOSE_GROUP_MAJORITY = 4`.** Fallback changes *which* peers |
| 80 | + satisfy quorum, not the threshold. |
| 81 | +- **Read availability:** rely on close-group convergence driven by node/membership |
| 82 | + eviction (saorsa-node ADR-0003) as the primary guarantee; keep GET-on-first-success |
| 83 | + plus the existing single retry. Do **not** widen the GET walk now. Explicitly flag |
| 84 | + the near-capacity read-gap — a chunk that lands *entirely* on fallback peers outside |
| 85 | + the queried 7 is unreadable until convergence shifts those peers into the close |
| 86 | + group — and gate any future GET-walk widening behind observed GET shortfalls in the |
| 87 | + near-capacity regime. |
| 88 | + |
| 89 | +## Consequences |
| 90 | + |
| 91 | +### Positive |
| 92 | + |
| 93 | +- Uploads blocked by a single (or minority) full close peer now reach quorum via |
| 94 | + fallback, with no protocol, payment, or node change required. |
| 95 | +- Failure causes are legible: full, under-paid, and transport failures get distinct, |
| 96 | + correct client responses. |
| 97 | + |
| 98 | +### Negative / Trade-offs |
| 99 | + |
| 100 | +- Fallback is best-effort and **bounded at the 20-wide window**; if more than |
| 101 | + `20 − 4` of the nearest peers refuse, quorum still cannot be met — the same |
| 102 | + near-capacity boundary the whole plan is bounded by. |
| 103 | +- A **per-node price floor** can reject a fully-paid put even with free disk; because |
| 104 | + we do not re-quote or top-up, such a peer is simply skipped — so a neighbourhood |
| 105 | + priced above the paid median can still cause a quorum shortfall this ADR does not |
| 106 | + resolve. |
| 107 | +- The near-capacity read-gap remains until convergence; we accept it for now rather |
| 108 | + than widen GET prematurely. |
| 109 | +- More client-side branching and a retry/fallback budget to tune. |
| 110 | + |
| 111 | +### Neutral / Operational |
| 112 | + |
| 113 | +- Quorum and close-group constants remain owned by `ant-protocol`; this ADR does not |
| 114 | + change them. |
| 115 | +- Adds one tunable: the fallback peer budget (how far down the next-closest list the |
| 116 | + client will try before giving up). |
| 117 | + |
| 118 | +## Validation |
| 119 | + |
| 120 | +- In a minority-full testnet, uploads that previously failed on a single full close |
| 121 | + peer reach quorum via fallback **without re-quoting or re-paying**. |
| 122 | +- Tests required before this ADR is Accepted: error classification maps |
| 123 | + `StorageFailed` / `Payment` / transport correctly; fallback stops at the 20-window; |
| 124 | + quorum still requires 4 successes; a price-floor rejection skips the peer and |
| 125 | + advances fallback without re-quoting or retrying the same peer; GET stays read-safe |
| 126 | + while ≥1 queried close peer holds the chunk. |
| 127 | +- Re-open trigger: if observed GET shortfalls reveal a persistent read-gap in the |
| 128 | + near-capacity regime, revisit widening the GET walk. |
| 129 | + |
| 130 | +## Notes for AI-assisted work |
| 131 | + |
| 132 | +AI tools may help draft this ADR, but **must not mark it Accepted without human |
| 133 | +review**. Accepted ADRs are immutable: create a new superseding ADR rather than |
| 134 | +editing this one. |
0 commit comments