(opt-in) use curl for tracker communication

[TheBaronVladimirHarkonnen]

fixes #4334
closes #8025

Goal

Connection reuse improves efficiency for both the client and server side. Coupled with support for SSL session reuse and advanced HTTP features, this implementation reduces the computing cost of implementing encryption in HTTP trackers.
Hopefully, this will stimulate more trackers to adopt encryption.

Breaking changes

• None; disabled by default. Compile time opt-in.

Breaking changes (with respect to the reference http_connection_tracker implementation):

• send_host_in_connect is ignored because it is required for curl's HTTP/1.x proxy and cannot be turned off. I suggest removing the setting and always turning it on.
• proxy_hostnames is ignored for HTTP proxies, because it is required in curl's HTTP proxy implementation.

Relevant curl issues:

• Keep-alive pings for connections in the connection cache are not processed, causing idle connections to die (https://github.com/curl/curl/blob/master/docs/TODO.md#monitor-connections-in-the-connection-pool).
• No port-hint support for HTTPS-rr DNS records feature advertisement (ECH, ALPN) (HTTPS RR ip hint and port ignored curl/curl#17045):
  • HTTPS-rr is currently experimental in libcurl and can only be enabled/disabled at compile time. It should be disabled until a baseline of features (such as port hint) is supported. For example, trackers can use a port hint to upgrade from HTTP to TLS connections on a different port.
  • Only works with libcurl DoH and c-ares resolvers (c-ares does not support ECH a.t.m.)

Questions:

• It is unclear how to run all automatic test with two HTTP stacks. Should all tests be executed for both?
• libtorrent/aux_/deadline_timer.hpp uses system_clock, which is not resistant to time jumps. Any chance of changing it to steady_clock?

Unit test coverage:

• Building with curl=on allows running all current tests for the curl HTTP stack.
• New test are added for ssrf and ip_filter because the simulator tests are not run yet (maybe remove these in favor of running the simulator tests?).
• New test are added for intrusive_list, bitmask and curl.
• No test are added for curl specific features like HTTP/2. Proper functioning of curl internals is assumed.
• Simulator support is not added yet
  Can be implemented by stubbing the curl_pool, and redirecting the curl_easy_* calls to secretly use the http_connection class. Is this desired? should this be a separate PR to keep this one small?

Testing:

• A rebased version has been tested with a custom qbittorrent docker image, based on libtorrent v2.0.11.
  https://github.com/TheBaronVladimirHarkonnen/docker-qbittorrent-nox
  • Compiled with curl_debug=on all the HTTP connection information is logged. Observed that the connections are being reused.
  • When the application is shutting down, the "stopped" events are sent. Good, the application exits cleanly.
  • Tested that curl errors are propagated properly to the UI.
• All testcases are passing with curl=on, no degradation in test running times.
• Did not run the simulator.
• Has not been tested on Windows yet, I will start working on setting up a windows box.

Implementation design:

• Uses libcurl thread-unaware multi library with boost asio async io handlers (single threaded, on the main libtorrent thread)
• All the newly added algorithms are O(1) per HTTP request, using O(N) to process N requests. However, the libcurl multi interface uses naive algorithms with O(N^2) time complexity (inside libcurl for queuing and list operations).

[arvidn]

please simplify this patch. also, review it to make sure you don't duplicate any functionality. also, make sure you have CI test coverage of the new functionality.

[arvidn]

btw. I would think the main issue with tracker announces right now is that there's no support for pipelining. If multiple torrents use the same tracker, they shouldn't each have to connect and SSL handshake.

[TheBaronVladimirHarkonnen]

btw. I would think the main issue with tracker announces right now is that there's no support for pipelining. If multiple torrents use the same tracker, they shouldn't each have to connect and SSL handshake.

You make a good point, I agree that this is both easy to implement in boost and cleaner to integrate into the current code base. The reason why I'm more interested in curl is as follows:

• With HTTP/1.1, it is possible to have more announces than you can process using a single connection. For example, a tracker expects an announce every 30 minutes, when an announce takes 0.5 seconds, you can only support at most 30*60*2=3600 torrents. H2 or H3 can solve this with doing many parallel announces, and the server can choose how many streams it wants to use.

• With HTTP/3 and QUIC, a server is no longer limited by the number of available TCP ports (0-65536), allowing it to keep more persistent connections.

• Encrypted Hello support, while this is currently not working yet. This is a cool privacy upgrade.

• HTTPS-RR support, making clients manually upgrade their tracker urls to HTTPS is not feasible. HTTP redirect for upgrades are not feasible. I want a server to upgrade all my torrents to HTTPS by simply adding a HTTPS-RR DNS record.

• boost beast has no development roadmap and has been in maintenance mode for years. I doubt it will ever support more advanced HTTP features. Writing correct/good code is their main goal, not low level efficiency.

I'm not happy with the current state of curl but it is the best library out there. Thinking about the future is important too, will the simple boost implementation be good enough 10 years from now? I would like the think we are all using the new advanced HTTP features by then.

Don't be afraid to reject the integration of curl if you don't like the idea, I would totally understand.

[TheBaronVladimirHarkonnen]

just a status update, I'm still busy reworking this.

[TheBaronVladimirHarkonnen]

needs more work, don't review yet.

[seabashed]

needs more work, don't review yet.

Looking forward to whenever you have time for this. The fact we need to create and tear down a connection with all the overhead (especially SSL announces) that entails is quite sad. Hope to see modern approaches adopted in our favorite tools! 👍

[TheBaronVladimirHarkonnen]

@seabashed
Thank you for the motivation to work on this.

Currently I'm working on creating a custom HTTP request queuing system to avoid using the curl queuing system entirely. The curl queuing was not made for applications like Libtorrent and we are better of not using it.

This queuing creates a separate queue per tracker:

• HTTP server without connection reuse, then use up to f(queue_size) connections, e.g. 5 connections per 200 queued items would avoid overloading the server.
• HTTP1.X server with connection reuse, then use up to g(queue_size) connections, e.g. 1 connection per 200 queued items.
• HTTP2/HTTP3 server with connection reuse, then use up to k(queue_size) connections, e.g. 1 connection with many streams would work.

The balance is between not creating too many connections, but at the same time being able to send all announces at startup/shutdown in a reasonable time-frame.

[seabashed]

Thanks @TheBaronVladimirHarkonnen, is this still worth pursuing or effectively replaced by @arvidn's work on 9654c51?