DHT announce timeboxing prevents full traversal with large torrent counts

[Autodystopia]

Hello.

I have 1k infohashes.

I run qbittorrent with them, and after an hour i got 346 metainfos.
Then i run rain with same hashes, and after an hour i got 334 metainfos.
Then i run torrent with same hashes, and after an hour i got 15 metainfos. After 36 hours i got 58 metainfos.
On another server with torrent i got 10 metainfos after 12 hours.
Ofc i dont use trackers nowhere.

So, how do i figure out what the problem is?

This is config i use

config := torrent.NewDefaultClientConfig()
config.ListenPort = 14746
config.PublicIp4 = net.ParseIP("xxx.xxx.xxx.xxx")
config.DisableIPv6 = true
config.PeriodicallyAnnounceTorrentsToDht = true
config.TotalHalfOpenConns = 1000

Stats

Listen port: 14746
Extension bits: 0000000000100005 (ltep, fast, dht)
Announce key: 633e5895
Banned IPs: 0
udp DHT server at 0.0.0.0:14746:
ID: 02307c896b45d5a0da14c498d88f8d5a7842b464
(dht.ServerStats) {
GoodNodes: (int) 15,
Nodes: (int) 177,
OutstandingTransactions: (int) 2829,
SuccessfulOutboundAnnouncePeerQueries: (int64) 0,
BadNodes: (uint) 0,
OutboundQueriesAttempted: (int64) 4408672
}
(torrent.ClientStats) {
ConnStats: (torrent.ConnStats) {
BytesWritten: (torrent.Count) 1896338,
BytesWrittenData: (torrent.Count) 0,
BytesRead: (torrent.Count) 10068279,
BytesReadData: (torrent.Count) 0,
BytesReadUsefulData: (torrent.Count) 0,
BytesReadUsefulIntendedData: (torrent.Count) 0,
ChunksWritten: (torrent.Count) 0,
ChunksRead: (torrent.Count) 0,
ChunksReadUseful: (torrent.Count) 0,
ChunksReadWasted: (torrent.Count) 0,
MetadataChunksRead: (torrent.Count) 558,
PiecesDirtiedGood: (torrent.Count) 0,
PiecesDirtiedBad: (torrent.Count) 0
},
ActiveHalfOpenAttempts: (int) 0,
NumPeersUndialableWithoutHolepunch: (int) 524,
NumPeersUndialableWithoutHolepunchDialedAfterHolepunchConnect: (int) 0,
NumPeersDialableOnlyAfterHolepunch: (int) 0,
NumPeersDialedSuccessfullyAfterHolepunchConnect: (int) 0,
NumPeersProbablyOnlyConnectedDueToHolepunch: (int) 0
}
Torrents: 942

DHT Announces on each torrent: 432

Another server

Listen port: 14746
Extension bits: 0000000000100005 (ltep, fast, dht)
Announce key: -186fbcb9
Banned IPs: 0
udp DHT server at 0.0.0.0:14746:
ID: a8b056bc30cdc1d44378c90ea2edd8e973649962
(dht.ServerStats) {
GoodNodes: (int) 10,
Nodes: (int) 159,
OutstandingTransactions: (int) 2973,
SuccessfulOutboundAnnouncePeerQueries: (int64) 0,
BadNodes: (uint) 0,
OutboundQueriesAttempted: (int64) 1951953
}
(torrent.ClientStats) {
ConnStats: (torrent.ConnStats) {
BytesWritten: (torrent.Count) 38646,
BytesWrittenData: (torrent.Count) 0,
BytesRead: (torrent.Count) 3262765,
BytesReadData: (torrent.Count) 0,
BytesReadUsefulData: (torrent.Count) 0,
BytesReadUsefulIntendedData: (torrent.Count) 0,
ChunksWritten: (torrent.Count) 0,
ChunksRead: (torrent.Count) 0,
ChunksReadUseful: (torrent.Count) 0,
ChunksReadWasted: (torrent.Count) 0,
MetadataChunksRead: (torrent.Count) 197,
PiecesDirtiedGood: (torrent.Count) 0,
PiecesDirtiedBad: (torrent.Count) 0
},
ActiveHalfOpenAttempts: (int) 0,
NumPeersUndialableWithoutHolepunch: (int) 58,
NumPeersUndialableWithoutHolepunchDialedAfterHolepunchConnect: (int) 0,
NumPeersDialableOnlyAfterHolepunch: (int) 0,
NumPeersDialedSuccessfullyAfterHolepunchConnect: (int) 0,
NumPeersProbablyOnlyConnectedDueToHolepunch: (int) 0
}
Torrents: 990

DHT Announces on each torrent: 254

torrent v1.58.1, debian 12

[anacrolix]

I'm guessing you're just adding the torrents by infohash.

I will double check the conditions around connecting to peers purely to obtain the metadata, it may be off for DHT (definitely well used for trackers).

Are you attempting to DHT index, or is it just an unusual case? I have code optimised for extracting metadata from the DHT.

Thanks for the excellent report.

[anacrolix]

Ah and the stat we need is how many connections were added. I suspect it will be very low. I will see if that's recorded somewhere.

[Autodystopia]

I'm guessing you're just adding the torrents by infohash.

By magnets to be precise.
client.AddMagnet("magnet:?xt=urn:btih:" + infohash)

Are you attempting to DHT index, or is it just an unusual case? I have code optimised for extracting metadata from the DHT.

Well, i making some media library and i have ~200k torrents which i want to download. Some of them is fresh, some is very old and has been abandoned, and first i want to get all possible meta to deduplicy content and determine what try to download.

conditions around connecting to peers purely to obtain metadata, it may be off for DHT (definitely well used for trackers).

But it works in principle, albeit poorly. Something is downloading...

By the way, it seems that meta is founding much faster when instead of 1k torrents i add only 10. At the same time in the stats i see this after a couple of minutes - GoodNodes: (int) 152, Nodes: (int) 161. Also SuccessfulOutboundAnnouncePeerQueries is growing (3280 after a couple hours), while in the examples above there is 0.

[anacrolix]

Oh that is interesting. Thanks, I will fix this up.

[anacrolix]

Okay if you could include the output of /debug/vars (run with GOPPROF=http) if you need a way to get at it. It's the expvar output. That would confirm my suspicion. It looks like you're probably running into a combination of rate limiting on the DHT server, and the announces to the DHT being "timeboxed" to 5 mins. With 1000 torrents you may never traverse the network to any useful nodes before the announces are halted. That's old code that probably doesn't need to exist anymore, because I added the concept of "traversal" to the dht library that actually has a termination condition. You can muck around with

torrent/torrent.go

Line 2193 in 67fe9f4

func (t *Torrent) timeboxedAnnounceToDht(s DhtServer) error {

, and/or set your DHT server configuration to allow more packets at a time.

There should be a refactor coming soon that pools active DHT announces to prevent too many goroutines. erigontech/erigon#14647.

[Autodystopia]

Yep, seems its working. I tried 15min timeboxes and set rate.NewLimiter(250, 25) to SendLimiter.

Here is outputs of stock/tweaked runs - https://drive.google.com/drive/folders/1BD9p815jXaF3daHc4LaIXDrC-Zk25i1Z?usp=drive_link

[anacrolix]

Thank you so much for following this through. I don't have the time to look into the data you have, but I am tempted just to remove the timeboxing entirely for the reason about traversals now having a termination condition. What do you think?

[anacrolix]

I am making the values you provided the defaults, they seem reasonable and match expectations of common hardware.