[10SP] OpenCMW: Rework REST/ZeroMQ interface

[frankosterfeld]

The current cpp-httplib RestBackend implementation has severe issues that prevent it from scaling up to real-world usage as the primary interface of client->broker communication.
It uses a model where each get/set as well as every long-poll request occupies a thread in a cpp-httplib internal thread pool, where esp. long-poll requests easily exhaust the available threads and then make the server stop responding.

The current implementation of RestBackend being a ZeroMQ client talking to the broker as normal zmq creates unnecessary overhead. As the REST backend is central to real-world use in the near future, it should be as efficient as possible.

Thus, the following changes should be made:

• Integrate RestBackend directly into Broker to avoid communication overhead, allowing to bind to a HTTP address analog to creating MDP/MDS endpoints, and handle HTTP connections inside the broker.
• Avoid long-poll requests blocking resources and improve scalability: Move from a thread-based server to event-driven I/O, where the HTTP server socket is polled and connections are served with replies/subscription notifications similarly as it is done right now for ZeroMQ clients. As a first step, this will be probably single-threaded, but keep the option in mind to use a thread pool for the actual I/O.
• Replace cpp-httplib with a suited replacement (probably nghttp2) to allows the aforementioned event-driven approach.
• The backend must support HTTP/1.1, HTTP/2 and should support HTTP/3. (It should be evaluated whether HTTP/3 is worth the added complexity (QUIC vs. TCP))
• Use the HTTP/2/HTTP/3 multiplexing support to use a single client->server connection for all of a client's subscriptions (instead of one connection per subscription).
• Adapt the opencmw client REST client APIs (native and wasm) to support HTTP2/3/multiplexing.
• Create a basic load test that allows to test the broker with the number of clients expected from production (10s-100s of clients), each subscribing to a few topics and using get/set commands.

Further requirements:

• Existing MDP/MDS and HTTP1.1 clients should continue working.
• External dependencies should be minimal (mostly OpenSSL)
• Replacing the HTTP library should be as easy as possible. Optional: Consider a design where multiple backends could be supported in parallel/via compile-time switch.
• Do not implement but keep in mind: WebSocket support

[RalphSteinhagen]

Please make sure to avoid using std::mutex. The existing bridge uses mutexes for locking and notification, and this makes reasoning about how it works, performance, and deadlocks much more difficult. We should keep things as lock-free as possible.

I have been working on some lock-free replacements for the existing bridge because we -- for some time -- felt that this would be the bottleneck.

LockFreeCache: https://compiler-explorer.com/z/fbEezGvbP
LockFreeTimedMap (WIP): https://compiler-explorer.com/z/8vbvExKxa

The LockFreeTimedMap is still a bit rough and not thoroughly tested, but it should give a good start.

[frankosterfeld]

Hi, I want to give an update after 2.5SP:

• nghttp2 is rather low-level and doesn't do the actual I/O itself, on the upside it gives us a lot of control about if how we want to implement the server (threads if any, event-based I/O). On the downside, getting things to work took some initial ramp-up time.
• I have a WIP working that receives GET/SET requests via SSL-encrypted HTTP/2 and converts them to MDP messages and forwards the responses. With the basics now working, adding long-polling support shouldn't be that hard anymore though.
• Protocols:
  • Protocol negotiation is working via ALPN (requires SSL enabled) and works here with curl as a client.
  • HTTP/1.1 isn't directly supported by nghttp2. We would need another code path using another HTTP parser, use some translating proxy as a front, or drop HTTP1.1 support.
  • HTTP/3 can be enabled per compile-time switch, brings a lot of dependencies though, including an OpenSSL that must be compiled with QUIC support. I didn't spend time on that yet.
• Broker/REST interface integration:
  • The HTTP server runs in a separate thread with a loop + select() inside
  • The current WIP would use two queues to pass messages between server thread and the broker
    • Pro: If we use a lock-free queue, we could avoid copying or de/serialization of large messages.
    • Con:
      • Needs a lock-free queue implentation in opencmw-cpp
      • Integration with the ZeroMQ socket polling would be trickier (to avoid inactivity on the sockets blocking processing of REST messages)
  • Alternative (my preference for now):
    • Keep using zmq for now for communication between REST server and broker
    • That would basically be what we have, architecture-wise, and one could argue that we then also can keep the whole structure (separation of Broker and RestBackend) as is...
    • Profile later and reconsider if there's a bottleneck (other solutions are thinkable, like using zmq sockets for notifying of new messages, but a queue to pass the payload)