diff --git a/docs.json b/docs.json index 132e21c8b..e40d28b84 100644 --- a/docs.json +++ b/docs.json @@ -719,6 +719,13 @@ "foundations/shards", "foundations/limits", "foundations/config", + { + "group": "Network protocols", + "pages": [ + "foundations/network/rldp", + "foundations/network/dht" + ] + }, { "group": "Web3 services", "pages": [ @@ -1410,17 +1417,17 @@ }, { "source": "/v3/documentation/network/protocols/dht/overview", - "destination": "https://old-docs.ton.org/v3/documentation/network/protocols/dht/overview", + "destination": "/foundations/network/dht", "permanent": true }, { "source": "/v3/documentation/network/protocols/dht/deep-dive", - "destination": "https://old-docs.ton.org/v3/documentation/network/protocols/dht/deep-dive", + "destination": "/foundations/network/dht", "permanent": true }, { "source": "/v3/documentation/network/protocols/rldp", - "destination": "https://old-docs.ton.org/v3/documentation/network/protocols/rldp", + "destination": "/foundations/network/rldp", "permanent": true }, { diff --git a/foundations/network/dht.mdx b/foundations/network/dht.mdx new file mode 100644 index 000000000..63da44b49 --- /dev/null +++ b/foundations/network/dht.mdx @@ -0,0 +1,179 @@ +--- +title: "Distributed Hash Table" +description: "Kademlia-like DHT for node discovery, address resolution, and overlay network management in TON" +sidebarTitle: "DHT" +--- + +import { Aside } from "/snippets/aside.jsx"; + +The TON Distributed Hash Table (DHT) is a Kademlia-like distributed key-value database used for discovering nodes in the network. Any network participant can operate a DHT node, generate keys, and store data. + +The DHT is used to store IP addresses of [ADNL](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/overview) nodes, torrent holder lists for [TON Storage](/foundations/web3/ton-storage), overlay subnetwork node addresses, and ADNL addresses of TON services and blockchain accounts. + +## Keys and values + +Keys are 256-bit integers, typically derived from SHA-256 of a TL-serialized object. Values are arbitrary byte strings of limited length, with meaning determined by the key's pre-image. + +In the simplest case, the key represents an ADNL address and the value is an IP address and port. + +## DHT nodes + +Each DHT node has a 256-bit address that should remain relatively stable. If an address changes too frequently, other nodes cannot locate the keys it stores. + +The value of key `K` is stored on the `S` nearest nodes to `K` (typically `S = 7`). This redundancy ensures that the value survives individual nodes going offline. + +## Kademlia distance + +Distance between a key and a node is computed via 256-bit XOR: + +```text +distance(key, node) = key XOR node_address +``` + +This distance is purely computational and unrelated to geographic location. A smaller XOR result means the node is "closer" to the key. + +## Routing table + +Each DHT node maintains a Kademlia routing table of 256 buckets (numbered 0 to 255). Bucket `i` contains nodes at Kademlia distance `2^i` to `2^(i+1) - 1` from the node's own address. + +Each bucket stores a fixed number of "best" nodes plus additional candidates. Stored information includes DHT addresses, IP addresses, UDP ports, and availability data (last ping time, latency). + +When a query discovers a new node, it is placed in the appropriate bucket as a candidate. Unresponsive "best" nodes are eventually replaced by candidates, keeping the routing table populated. + +## Key-value update rules + +Update rules vary by key type: + +- **Signature-based**: The new value must be signed by the key owner. The signature is retained for later verification. +- **Sequence-number-based**: The new value must contain a higher sequence number than the old value, preventing replay attacks. +- **Unrestricted**: Anyone can update the value. + +## Finding a value by key + +To look up a value, [connect to any DHT node via ADNL UDP](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/udp) and send a `dht.findValue` query. + +### Constructing the key ID + +First, build the DHT key using: + +```tlb +dht.key id:int256 name:bytes idx:int = dht.Key +``` + +- `id` — the ADNL address or other identifier being searched. +- `name` — the key type as a byte string (for example, `"address"` for ADNL address lookups, `"nodes"` for shard node lookups). +- `idx` — 0 when there is only one key. + +Serialize this structure and compute its SHA-256 hash — this is the key ID. + +### Sending the query + +```tlb +dht.findValue key:int256 k:int = dht.ValueResult +``` + +- `key` — the key ID from the previous step. +- `k` — search width. Smaller values produce more precise results but fewer candidate nodes. + +The response is either [`dht.valueFound`](#dhtvaluefound) or [`dht.valueNotFound`](#dhtvaluenotfound). + +### `dht.valueNotFound` + +The response includes a list of nodes that are close to the requested key: + +```tlb +dht.node id:PublicKey addr_list:adnl.addressList version:int signature:bytes = dht.Node; +dht.nodes nodes:(vector dht.node) = dht.Nodes; +dht.valueNotFound nodes:dht.nodes = dht.ValueResult; +``` + +For each node: + +1. Verify the signature: read it, set the field to empty bytes, serialize the `dht.node`, and verify the signature against the `id` public key. +1. Connect via [ADNL UDP](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/udp) using the address from `addr_list` and the `id` as the server key. +1. Compute the XOR distance between the node's [key ID](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/tcp) and the target key. If the distance is small enough, repeat the query. + +Continue until a value is found or no new nodes remain. + +### `dht.valueFound` + +```tlb +dht.keyDescription key:dht.key id:PublicKey update_rule:dht.UpdateRule signature:bytes = dht.KeyDescription; +dht.value key:dht.keyDescription value:bytes ttl:int signature:bytes = dht.Value; +dht.valueFound value:dht.Value = dht.ValueResult; +``` + +The `key:dht.keyDescription` provides full key metadata: + +- `key:dht.key` — must match the searched key. +- `id:PublicKey` — the record owner's public key. +- `update_rule:dht.UpdateRule` — determines who can modify this record: + +| Rule | Description | +| ----------------------------- | --------------------------------------------------------------------------------- | +| `dht.updateRule.signature` | Only the private key owner can update; both key and value signatures are verified | +| `dht.updateRule.anybody` | Anyone can update; signatures are not checked | +| `dht.updateRule.overlayNodes` | Nodes from the same overlay can update | + +### Verifying signature-based values + +For `dht.updateRule.signature` (used in ADNL address lookups): + +1. Set the `signature` field of `dht.keyDescription` to empty bytes, serialize, and verify against the `id` public key. +1. Restore the key signature, then set the `signature` field of `dht.value` to empty bytes, serialize, and verify. + +[Implementation example](https://github.com/xssnick/tonutils-go/blob/46dbf5f820af066ab10c5639a508b4295e5aa0fb/adnl/dht/client.go#L331). + +### Verifying overlay node values + +For `dht.updateRule.overlayNodes`, the value contains: + +```tlb +overlay.node id:PublicKey overlay:int256 version:int signature:bytes = overlay.Node; +overlay.nodes nodes:(vector overlay.node) = overlay.Nodes; +``` + +For each node, verify the `signature` by serializing: + +```tlb +overlay.node.toSign id:adnl.id.short overlay:int256 version:int = overlay.node.ToSign; +``` + +Replace `id` with `adnl.id.short` (the key ID hash of the original `id` field) and verify the signature. + +### Using the value + +Once verified and the `ttl` has not expired, extract `value:bytes`. For ADNL address lookups, this contains an `adnl.addressList` with IP addresses and ports. Use the `id:PublicKey` from the key description as the server key when connecting. + +## Finding blockchain nodes + +The DHT is also used to discover nodes storing blockchain data for specific workchains and shards. + +### Building the overlay key + +1. Fill in the TL structure: + + ```tlb + tonNode.shardPublicOverlayId workchain:int shard:long zero_state_file_hash:int256 = tonNode.ShardPublicOverlayId; + ``` + + For the masterchain: `workchain = -1`, `shard = -9223372036854775808` (`0x8000000000000000`). The `zero_state_file_hash` comes from the `validator > zero_state` section of the global config. + +1. Serialize and hash to get the key ID. + +1. Wrap this key ID in `pub.overlay name:bytes = PublicKey`, serialize, and hash again to get the final key. + +1. Use this key with `dht.findValue`, setting `name` to `"nodes"` in the `dht.key`. + +The result uses `dht.updateRule.overlayNodes`. After validation, the returned public keys identify nodes with blockchain data. Hash these keys to get ADNL addresses, then look up each address in the DHT to obtain connection details. + +To discover additional nodes in the same shard, use [`overlay.getRandomPeers`](https://github.com/ton-blockchain/ton/blob/ad736c6bc3c06ad54dc6e40d62acbaf5dae41584/tl/generate/scheme/ton_api.tl#L237). + +## See also + +- [ADNL specification](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/overview) +- [ADNL UDP](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/udp) +- [DHT implementation](https://github.com/ton-blockchain/ton/tree/master/dht) +- [DHT server implementation](https://github.com/ton-blockchain/ton/tree/master/dht-server) +- [The Open Network Whitepaper, Chapter 3.2](https://ton.org/whitepaper.pdf) +- [Original article by Oleg Baranov](https://github.com/xssnick/ton-deep-doc/blob/master/DHT.md) diff --git a/foundations/network/rldp.mdx b/foundations/network/rldp.mdx new file mode 100644 index 000000000..7c64af891 --- /dev/null +++ b/foundations/network/rldp.mdx @@ -0,0 +1,149 @@ +--- +title: "Reliable Large Datagram Protocol" +description: "Protocol for transferring large data blocks over ADNL UDP using Forward Error Correction" +sidebarTitle: "RLDP" +--- + +import { Aside } from "/snippets/aside.jsx"; + +The Reliable Large Datagram Protocol (RLDP) operates on top of [ADNL UDP](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/udp) and is designed for transferring large data blocks. It uses Forward Error Correction (FEC) instead of acknowledgment packets, enabling more efficient data transfer at the cost of increased traffic. + +RLDP is used throughout the TON infrastructure: downloading blocks from other nodes, transferring data, and accessing [TON Sites](/foundations/web3/ton-sites) and [TON Storage](/foundations/web3/ton-storage). + +## Protocol + +RLDP uses the following TL structures: + +```tlb +fec.raptorQ data_size:int symbol_size:int symbols_count:int = fec.Type; +fec.roundRobin data_size:int symbol_size:int symbols_count:int = fec.Type; +fec.online data_size:int symbol_size:int symbols_count:int = fec.Type; + +rldp.messagePart transfer_id:int256 fec_type:fec.Type part:int total_size:long seqno:int data:bytes = rldp.MessagePart; +rldp.confirm transfer_id:int256 part:int seqno:int = rldp.MessagePart; +rldp.complete transfer_id:int256 part:int = rldp.MessagePart; + +rldp.message id:int256 data:bytes = rldp.Message; +rldp.query query_id:int256 max_answer_size:long timeout:int data:bytes = rldp.Message; +rldp.answer query_id:int256 data:bytes = rldp.Message; +``` + +All structures are serialized, wrapped in `adnl.message.custom`, and transmitted over [ADNL UDP](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/udp). + +## Transfer mechanism + +1. Generate a random `transfer_id`. +1. Process the data through the FEC algorithm to produce encoded symbols. +1. Wrap each symbol in `rldp.messagePart` and send to the peer. +1. Continue sending until the peer responds with `rldp.complete` or a timeout occurs. + +The receiver collects `rldp.messagePart` messages, and once enough symbols are gathered, decodes them via FEC and deserializes the result into `rldp.query` or `rldp.answer`. + +## Forward Error Correction + +Three FEC algorithms are supported: RoundRobin, Online, and RaptorQ. Currently, [RaptorQ](https://www.qualcomm.com/media/documents/files/raptorq-technical-overview.pdf) is the standard choice. + +### RaptorQ + +RaptorQ divides data into fixed-size symbols (blocks). These symbols are organized into matrices where discrete mathematical operations produce an effectively unlimited number of encoded symbols from the same source data. + +This allows the receiver to recover lost packets without requesting retransmission — fewer packets are needed than if raw data segments were simply resent. + +Symbols are transmitted until the peer confirms that all data has been received and successfully decoded. + +[RaptorQ implementation in Go](https://github.com/xssnick/tonutils-go/tree/46dbf5f820af066ab10c5639a508b4295e5aa0fb/adnl/rldp/raptorq). + +## RLDP-HTTP + +[TON Sites](/foundations/web3/ton-sites) use RLDP to wrap HTTP traffic. The host runs a standard HTTP web server with `rldp-http-proxy` alongside it. Incoming requests from the TON network arrive via RLDP, the proxy converts them to HTTP and calls the local web server. On the client side, a local proxy (such as [Tonutils Proxy](https://github.com/xssnick/TonUtils-Proxy)) converts outgoing HTTP requests into RLDP. + +HTTP over RLDP uses these TL structures: + +```tlb +http.header name:string value:string = http.Header; +http.payloadPart data:bytes trailer:(vector http.header) last:Bool = http.PayloadPart; +http.response http_version:string status_code:int reason:string headers:(vector http.header) no_payload:Bool = http.Response; + +http.request id:int256 method:string url:string http_version:string headers:(vector http.header) = http.Response; +http.getNextPayloadPart id:int256 seqno:int max_chunk_size:int = http.PayloadPart; +``` + +The request-response flow: + +1. Client sends `http.request` via `rldp.query`. +1. Server checks the `Content-Length` header: + - If non-zero, server sends `http.getNextPayloadPart` requests to the client. + - Client responds with `http.payloadPart` chunks, incrementing `seqno` until `last` is true. +1. Server sends `http.response`. +1. Client checks `Content-Length`: + - If non-zero (`no_payload` is false), client sends `http.getNextPayloadPart` requests to the server using the same chunked mechanism. + +## Example: requesting a TON Site + +This example demonstrates requesting `foundation.ton`, assuming the ADNL address has been resolved via DNS and a connection has been established over [ADNL UDP](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/udp). + +### Sending a GET request + +Serialize `http.request`: + +```text +e191b161 -- TL ID http.request +116505dac8a9a3cdb464f9b5dd9af78594f23f1c295099a9b50c8245de471194 -- id (random) +03 474554 -- method = "GET" +16 687474703a2f2f666f756e646174696f6e2e746f6e2f 00 -- url = "http://foundation.ton/" +08 485454502f312e31 000000 -- http_version = "HTTP/1.1" +01000000 -- headers (1 entry) + 04 486f7374 000000 -- name = "Host" + 0e 666f756e646174696f6e2e746f6e 00 -- value = "foundation.ton" +``` + +Wrap in `rldp.query`: + +```text +694d798a -- TL ID rldp.query +184c01cb1a1e4dc9322e5cabe8aa2d2a0a4dd82011edaf59eb66f3d4d15b1c5c -- query_id (random) +0004040000000000 -- max_answer_size = 257 KB +258f9063 -- timeout (unix timestamp) +34 ... -- data (serialized http.request) +``` + +### Encoding with RaptorQ + +1. Choose a symbol size (typically 768 bytes). +1. Pad the data to a multiple of the symbol size with zero bytes. For 156 bytes of data, add 612 zero bytes to reach 768. +1. Create a RaptorQ encoder ([example](https://github.com/xssnick/tonutils-go/blob/46dbf5f820af066ab10c5639a508b4295e5aa0fb/adnl/rldp/raptorq/encoder.go#L15)). + +### Sending packets + +Send encoded symbols in `rldp.messagePart`: + +- `transfer_id` — random int256, same for all parts of this transfer. +- `fec_type` — `fec.raptorQ` with `data_size = 156`, `symbol_size = 768`, `symbols_count = 1`. +- `part` — 0 (used for very large transfers that exceed size limits). +- `total_size` — 156 (actual data size). +- `seqno` — starts at 0, incremented for each packet. +- `data` — encoded symbol (768 bytes). + +Wrap each in `adnl.message.custom` and send over ADNL UDP. Send symbols in round-robin order, continuing until `rldp.complete` is received. After sending one full round of symbols, slow down to one packet per 10 milliseconds — extra packets serve as FEC recovery data. + +[Implementation example](https://github.com/xssnick/tonutils-go/blob/be3411cf412f23e6889bf0b648904306a15936e7/adnl/rldp/rldp.go#L249). + +### Processing the response + +The response arrives as `rldp.messagePart` messages inside `adnl.message.custom`. The `transfer_id` is the original transfer ID with each byte XORed with `0xFF`. + +1. Extract FEC parameters (`data_size`, `symbol_size`, `symbols_count`) from the first received message. +1. Initialize a RaptorQ decoder ([example](https://github.com/xssnick/tonutils-go/blob/be3411cf412f23e6889bf0b648904306a15936e7/adnl/rldp/rldp.go#L137)). +1. Feed received symbols with their `seqno` into the decoder. +1. Once `symbols_count` symbols are collected, attempt decoding. On success, send `rldp.complete` ([example](https://github.com/xssnick/tonutils-go/blob/be3411cf412f23e6889bf0b648904306a15936e7/adnl/rldp/rldp.go#L168)). + +The decoded result is `rldp.answer` containing `http.response`. If `no_payload` is false, retrieve the body by sending `http.getNextPayloadPart` requests with incrementing `seqno` and `max_chunk_size` of 128 KiB (131,072 bytes) until `last` is true. + +## See also + +- [ADNL specification](https://old-docs.ton.org/v3/documentation/network/protocols/adnl/overview) +- [TON Proxy](/foundations/web3/ton-proxy) +- [TON Sites](/foundations/web3/ton-sites) +- [RLDP implementation](https://github.com/ton-blockchain/ton/tree/master/rldp) +- [RLDP v2 implementation](https://github.com/ton-blockchain/ton/tree/master/rldp2) +- [Original article by Oleg Baranov](https://github.com/xssnick/ton-deep-doc/blob/master/RLDP.md)