Skip to content

Commit 998360a

Browse files
author
cortze
committed
add basic documentation how the module works, how to use it + numbers :)
1 parent 7f33cbb commit 998360a

1 file changed

Lines changed: 162 additions & 26 deletions

File tree

‎README.md‎

Lines changed: 162 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,172 @@
11
# py-dht
2-
Small and straightforward representation of how a Kademlia-based DHT could be integrated into Ethereum, particularly at [DAS-research](https://github.com/codex-storage/das-research).
3-
4-
## Terminology
5-
All the terms and abbreviations that will be found in the code
6-
- BPID: Block-Part IDentifier (for both Row and Column IDs)
7-
8-
## Specifications
9-
10-
The code includes a simple logic implementation of a `DHTClient`, which includes:
11-
- [x] basic `Hash` and `BitArray` implementations
12-
- [x] logical `RoutingTable` and `KBucket` that can:
13-
- [x] fill its kbuckets with the XX closest peers sharing YY bits with our `NodeID`
14-
- [x] Give back the closest XX peers to a given Hash
15-
- [x] basic `DHTClient` operations, such as:
16-
- [x] create a Network interface that can link all the nodes in the network
17-
- [x] bootstrap and fill up the routing table from ZZ nodes
18-
- [x] lookup for the closest XX peers to a given Hash
19-
- [x] Provide a `BPID` to the network
20-
- [x] Retrieve a `BPID` from the network
21-
- [ ] Make the DHT compatible with random delays and error rates
22-
- [ ] make randomness and hashes deterministic
2+
small and straightforward representation of how a kademlia-based dht could be integrated into ethereum, particularly at [das-research](https://github.com/codex-storage/das-research).
3+
4+
## What can this dht do?
5+
the current work on this simulation of a kademlia-based dht network offers:
6+
7+
- [`DHTNewtork`](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/dht/dht.py#l235) object
8+
that can: spawn `dhtclients`, serve as main source to initialize the `routing table` of the `dhtclients`, resolve
9+
`connections` between `dhtclients`, handle and keep track of the interactions between `dhtclients`. the parameters to
10+
configure the a `dhtnetwork` are:
11+
- `networkid`: in case we want to simulate different network at the same time
12+
- `errorrate`: to define the number connections that will fall into an error (we can understand it as the likelines o
13+
f an error in %)
14+
- `delayrage`: range between the slowest possible delay and the biggest one. a random delay will be selected every
15+
time a connection is stablished between 2 nodes (if no error is raised)
16+
17+
the network offers the following functions:
18+
- `parallel_clilist_initializer`
19+
- `init_with_random_peers` initializes a network using a "blazingly fast" method, which can be optimized even more if
20+
a number of threads/processes is defined
21+
- `add_new_node` adds a new node to the local `Network`
22+
- `connect_to_node` returns the `Connection` obj between node `A` and `B`
23+
- `bootstrap_node` return the "best" nodes/dhtclis to compose the routing table for the given node
24+
- `summary` return the summary of the current status of the network (number of nodes, successful connections, failed
25+
ones, etc), will evolve over time
26+
27+
28+
- [`Connection`](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/dht/dht.py#l211)
29+
interface that limits how two `dhtclients` interact with each other (like if it was a closed api/protocol).
30+
it offers the possibility to client `a` (client starting the connection) to ask client `b` (remote client), applying
31+
if specified the delay at the moment of stablishing the connection and per each interaction:
32+
- `get_closest_nodes_to(hash)` will return the k closest nodes to the given `hash` that client `b` has in it's routing
33+
table. _note: it will also return the value if it's stored locally :)_
34+
35+
- `store_segment(segment)` will add the `hash` and the segment as a key-value pair in `b`'s local storage
36+
- `retrieve_segment(hash)` will ask `b` to return the segment of the given `hash` if it has it
37+
38+
39+
- [`DHTClient`](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/dht/dht.py#l14) as
40+
representation of a node in the simulated dht network. the `dhtclient` can be created using the following parameters:
41+
- `nodeid`: id of the node that hosts the `dhtclient`
42+
- `network`: referece to the network obj that where the `dhtclient` participates in
43+
- `kbucketsize`: k value, number of nodes per kbucket
44+
- `a`: number of concurrent node connections the client does while looking for a given key
45+
- `b`: target of nodes (number of nodes) returned when asking for a `hash`
46+
- `steptostop`: number of iterations without finding anyone closer to stop the `lookup` operation
47+
48+
the client serves a list of endpoints such as:
49+
- `bootstrap` uses the network reference to find the right peers for the routing table
50+
- `lookup_for_hash` will try to look the value of the `hash` in the network, and the closest nodes to it
51+
- `get_closest_nodes_to` will return the closest nodes to a `hash` from the local routing table
52+
- `provide_block_segment` will lookup for the closest nodes in the network, and store the segment on them
53+
- `store_segment` will store locally a segment value using its `hash` as key
54+
- `retrieve_segment` will return the value of a `hash` if its locally, exception raised otherwise
55+
56+
57+
- [`RoutingTable`](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/dht/routing_table.py#L21) and
58+
[`KBucket`](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/dht/routing_table.py#L76) classes to store locally the local representation of the network for a given node
59+
60+
61+
- [`Hash`](https://github.com/cortze/py-dht/blob/main/dht/hashes.py#l9) and
62+
[`BitArray`](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/dht/hashes.py#l44) classes
63+
to represent a `nodeid`/`blocksegment`/`generalobject`
64+
65+
## Dependencies
66+
The source code runs mostly on plain Python libraries. However, to speed up the performance, the plain `arrays` and `dicts`
67+
were updated to classes from `collections`. Thus, I recomend to have an specific virtual environment to use the module.
68+
69+
To install the dependencies, do:
70+
```shell
71+
# python -m venv venv
72+
# or
73+
# python -m virtualenv venv
74+
# source venv/bin/activate
75+
(venv)$ pip install -r requirements.txt
76+
```
77+
78+
## Tests
79+
The repo has a list of tests to ensure that no functionality is broken whenever a new feature is added. All the tests are
80+
triggered whenever GitHub records a `Push` or a `PullRequest`. However, there are locally runable using the `./launch_test.sh`
81+
script
82+
83+
```shell
84+
# the script will try to activate any `venv` located at the root of the directory
85+
py-dht$ bash launch_tests.sh
86+
```
87+
88+
## Benchmarks
89+
Running benchmarks is a bit trickier. First install the `py-dht` module as `editable` in the `venv`
90+
```shell
91+
py-dht$ pip install -e ./
92+
```
93+
after that, feel free to change the parameters in the `benchmarks/launch_benchmarks.sh` and run it like if it was a test:
94+
```shell
95+
# the script will try to activate any `venv` located at the root of the directory
96+
py-dht$ cd benchmarks
97+
py-dht/benchmarks$ bash launch_benchmarks.sh
98+
```
99+
100+
## Numbers and recomendations
101+
From the experience of running tests and benchmarks on the repo, I can say that the optimizations on [#8](https://github.com/cortze/py-dht/pull/8)
102+
and [#9](https://github.com/cortze/py-dht/pull/9) were more than necesary.
103+
104+
*Recomendations:*
105+
- to simulate a network -> use the `network.init_with_random_peers()` function setting the `processes` parameters
106+
the initialization of the network is by far the process that takes the longer, as it has to compute the best routing tables
107+
for each of the spawned nodes. So, please benefit from the concurrency to reduce the duration
108+
(check [this test](https://github.com/cortze/py-dht/blob/f5a1c27735bececf75942b54a7426aabf2fd28e7/tests/test_network.py#L113) as example)
109+
- At the moment using 20 cores of a `ryzen 5900x` I'm able to initialize a network of `10k dhtclients` with `k=20` in `28 secs`
110+
111+
Latest numbers of the `network becnhmark`
112+
```shell
113+
# 03.08.2023
114+
py-dht$ python benchmarks/network.py -t opt-conc -o ./csvs -i 1 -k 20 -n 10000 --threads 20
115+
-- benchmark: opt-conc_network_n_initialization --
116+
rounds : 1
117+
failed rounds : 0
118+
prep time (s) : 1.5974044799804688e-05
119+
avg (s) : 0.09420228004455566
120+
median (s) : 0.09420228004455566
121+
p90_duration (s): 0.09420228004455566
122+
-- benchmark: opt-conc_network_bootstrap_node --
123+
rounds : 1
124+
failed rounds : 0
125+
prep time (s) : 0.11602234840393066
126+
avg (s) : 0.6563236713409424
127+
median (s) : 0.6563236713409424
128+
p90_duration (s): 0.6563236713409424
129+
-- benchmark: opt-conc_network_fast_bootstrap_network --
130+
rounds : 1
131+
failed rounds : 0
132+
prep time (s) : 8.344650268554688e-06
133+
avg (s) : 174.6814157962799
134+
median (s) : 174.6814157962799
135+
p90_duration (s): 174.6814157962799
136+
-- benchmark: opt-conc_network_fast_threaded_bootstrap_network --
137+
rounds : 1
138+
failed rounds : 0
139+
prep time (s) : 0.0008330345153808594
140+
avg (s) : 28.239949226379395
141+
median (s) : 28.239949226379395
142+
p90_duration (s): 28.239949226379395
143+
```
144+
At the moment (after applying the code optimizations in #8 and the conncurrency #9) the numbers look like this:
145+
146+
| Number of nodes | Default implementation | 1st Optimization (monothread) | 2nd Optimization (8 processes) |
147+
|-----------------|------------------------|-------------------------------|--------------------------------|
148+
| 500 | 10 | 2.096 | 0.489 |
149+
| 1000 | 39 | 5.39 | 1.02 |
150+
| 1200 | 56 | 7.22 | 1.48 |
151+
| 2000 | 157.36 | 14.49 | 2.83 |
152+
| 5000 | 995.73 | 60.01 | 11.15 |
153+
| 10000 | (beyond 2 hours) | 199.97 | 39.15 |
154+
_NOTE: all the measurements displayed in the table are expresed in seconds_
155+
156+
![img.png](docs/optimization_benchmarks.png)
157+
158+
23159

24160
## Maintainer
25-
[@cortze](https://github.com/cortze)
161+
Mikel Cortes-Goicoechea - [@cortze](https://github.com/cortze)
26162

27163
## Contributing
28-
Feel free to dive in! Change proposals, issues, and PRs will be more than welcome.
164+
feel free to dive in! change proposals, issues, and prs will be more than welcome.
29165

30166
## Support
31-
- The work has been supported by [Codex](https://github.com/codex-storage)
32-
- Feel free to support this project through [Buy Me A Coffee](https://www.buymeacoffee.com/cortze); it would make my day 😊.
167+
- the work has been supported by [codex](https://github.com/codex-storage)
168+
- feel free to support this project through [buy me a coffee](https://www.buymeacoffee.com/cortze); help me getting the ship of caffeine that I need 😊.
33169

34170
## License
35-
MIT, see [LICENSE](./LICENSE) file
171+
mit, see [license](./license) file
36172

0 commit comments

Comments
 (0)