Always start as DHT Client and switch to DHT Server after 48 hours

[RubenKelevra]

Checklist

• My issue is specific & actionable.
• I am not suggesting a protocol enhancement.
• I have searched on the issue tracker for my issue.

Description

Currently, all newly started nodes do advertise as DHT Server if they can handle incoming connections.

A lot of computers are used just for a couple of hours and then restarted, put into standby or have longer outages of their internet connection.

I think go-ipfs should start a timer on startup and if there's a connection outage for more than 10 minutes, a standby event etc. the timer will be reset.

Once 48 hours are reached, the node switches to the current 'dht' setting which will decide if the node can handle incoming connections etc. and then may switch to the server profile.

This will eliminate the efforts to push data to nodes which are not capable of holding data for longer periods of time available on their network, but recover them after two days, if there's for example a connection outage.

[Jorropo]

This is surprisingly painfull thing to implement because right now this is an option to the DHT factory.

We can't just call dht.StartServer.

It's nothing an API change in go-libp2p-kad-dht can't solve, but that more work than expected.

[hsn10]

48 hours is too much, 3 hours is more realistic since 80% of IPFS nodes in my scan have uptime lower then 2h. They are probably Brave run nodes.

[RubenKelevra]

48 hours is too much, 3 hours is more realistic since 80% of IPFS nodes in my scan have uptime lower then 2h. They are probably Brave run nodes.

That's exactly the nodes we want to NOT be part of the DHT as servers. :)

[RubenKelevra]

This is surprisingly painfull thing to implement because right now this is an option to the DHT factory.

We can't just call dht.StartServer.

It's nothing an API change in go-libp2p-kad-dht can't solve, but that more work than expected.

Yeah, I kinda expected this. However, that would be pretty beneficial. :)

[aschmahmann]

IMO this seems worth investigating, but I wouldn't pick a number out of a hat here. The balance here is between having more server nodes, and having higher uptime server nodes. More server nodes means more nodes to share the load, better nodes means less churn and likely some network parameters can change to require less load from DHT clients.

cc @yiannisbot for any thoughts here

[RubenKelevra]

Hey @aschmahmann,

the idea with 48 hours was simply to avoid to have office/desktop computers which run for a night to start serving DHT. So a computer would at least run for two nights.

This would also avoid that computer which have a semi stable connection start serving. In Germany for example the Internet connections are terminated after 24 hours and you dial in again and get a different IP. If the connection quality overall is poor it sometimes just fails for a while. That's why this is usually moved to the night by the routers.

So if it recovers fast by updating the IP and reconnect it's fine IMHO, but if there's a longer outage it's just not worth that the computers take part in the DHT as servers.

While, yes, the load would go up slightly when there are less nodes serving, as they need to cover a larger range of the address space, the query time goes down, as there are far less failed dials.

Currently the assumption is that at least one of the DHT nodes provides the record for at least 12 hours (reprovide interval). If the nodes only start serving when they consider themselves healthy for 48 hours it would be save to increase the reprovide interval to a lot longer interval and thus reducing the overall traffic in the network.

Btw: I'm currently seeing very high numbers for failed dials on my nodes (tcp resets), just for pushing records to the DHT: 10-200 failed tcp connection attempts per second, while I already filter out all private IP-spaces in my config file.

[hsn10]

changed IP is not a problem, DHT is organised by Peer Id. It will quickly switch to new node IP.

[RubenKelevra]

@hsn10 never stated that this is a problem 🤔

[BigLep]

2022-07-01 conversation: the goal with this is to reduce the number of dead peers in the network.

Data sources for how many dead peers are in the network:

1. https://github.com/protocol/network-measurements/blob/master/results/rfm2-uptime-and-churn.md
2. https://stats.ipfs.network/nebula-22-07-01/

That helps establish how big of a problem this is.

We then would need data on what the ideal value to set this to. Until having data and analysis of this, maintainers aren't planning to do any work here because there are higher leverage things that can be done for the DHT performance.

[yiannisbot]

I don't have a clear view of what this would entail in terms of implementation, but it's certainly not a bad idea to consider.

Our results show that we could go on and increase the reprovide interval (or reduce the provider record replication factor) even now. Provider records seem to stay alive in ~15 peers over a 24h period.

Quick clarification: why would Brave nodes show as servers? Do you mean the Brave nodes that run on peers with publicly accessible IPs?

[guillaumemichel]

As mentioned above by @aschmahmann, the tradeoff here is to balance the churn rate vs the network size. It is important to have a large number of nodes participating in the DHT for load balancing and decentralization.

I see two main reasons for which a high churn rate has a negative impact on IPFS:

1. We don't want that all DHT server nodes storing the same Provider Record gone before the Provider Record is republished, otherwise the DHT is unable to route to the associated content.
2. When going offline, the DHT server nodes will still appear in other nodes' routing tables as stale entries. These stale entries slow down the DHT lookup process.

The following graph by @cortze in the context of RFM17 shows that the first point is not a concern. On average, 15 out of the 20 original Provider Records are still provided after 24 hours (without even being republished). IPFS republishes the Provider Records every 12 hours, so the probability that a Provider Record disappears from the DHT given the current churn rate is negligible.

In RFM19, we discovered that the number of stale entries in the routing table is low (roughly 4% for buckets 0 to 8 and 15% for bucket with higher ID). So it is not a big pain at the moment, but it can certainly be improved!

kubo the most widely spread IPFS implementation makes use of libp2p/go-libp2p-kad-dht, whose Kademlia k-bucket replacement policy roughly corresponds to refreshing the routing table every 10 minutes: the node will try to contact all the peers in its routing table and will evict the peers that fail to respond in time. Thus, stale entries cannot stay longer than 10 minutes in a peer's routing table. A stale entry will live 5 minutes on average (assuming a random uniform distribution between the time the node goes offline and the routing table refresh).

The table below shows the ratio of the stale entry liveliness vs the total uptime of the node ( $\frac{5 min}{uptime+5 min}$ ) for all nodes scanned by the Nebula Crawler from February to May 2022.

It shows us that on average peers with an uptime <1 hour will spend ~$\frac{1}{3}$ of their time being a stale entry. Nodes with an uptime between 1 and 2 hours could have stale entries living for 5.93% of their uptime on average (average uptime of this category: 1 hour and 19 minutes = 79.2 minutes $\rightarrow \frac{5 min}{79.2 + 5 min}=5.93\%$).

Note that starting the DHT server mode after X hours will significantly change the data on the graph. For instance, if the server mode is turned on after 1h, the nodes with 1-2h uptime will have a much higher stale entry liveliness vs total uptime ratio, (roughly 4x) as the average uptime of this category is 79.2 minutes, so the nodes will serve the DHT in server mode for 19.2 minutes on average.

The following graphs shows the ratio having an uptime greater than X hours/days.

Turning on the DHT server mode after 48 hours would reduce the number of DHT server peers by 35x and increase the individual load by 35x, which is probably not desirable. Excluding only the nodes with an uptime <1 hour would double the load for the rest of the network, but remove a large proportion of the potential routing table stale entries. It will also cause one additional empty k-bucket on average (non full buckets will be shifted 1 ID lower $\rightarrow$ 1 full bucket would become non full, and the last bucket would become empty).

It would make sense to start the DHT Server mode after 1 to 3 hours, the individual load would be approximately 2x to 3x. The plot below shows how the stale entry liveliness vs total uptime would look like if the DHT server mode is started respectively immediately, 1 hour, 2 hours and 3 hours after a node gets online. The graph is shifted and time starts when nodes join the DHT (respectively after 0, 1, 2 and 3 hours), so the node categories are also shifted.

We can see that the ratio roughly stays the same if we change the time DHT server mode is turned on. The overall number of stale entries will be reduced as there are more peers with uptime <1h compared with peers with uptime between 1h and 2h. That is because there will be less peers in the network, thus the performance would be left unchanged. The ratio is expected to stay approximately constant as the conditional uptime distribution, displayed below, is almost linear.

I am a bit surprised by these results, but it looks like starting the DHT server mode after some time would hardly improve the routing efficiency, but it would significantly reduce the number of peers participating in the DHT and increase the individual load. I think we should either not change anything or we could try starting the DHT server mode after 2-3 hours, which should give a small improvement in terms of routing table stale entries, but the individual load would increase by 3x.

What do you think?