Reimplement REST interface using HTTP/2, HTTP3 (#371)

* Reimplement REST interface using HTTP/2

Instead of using the httplib-based HTTP server using blocking I/O in
a thread per client, implement a nghttp2-based HTTP/2 server, using
single-threaded non-blocking I/O directly integrated into the Broker
zmq_poll() loop.

* HTTP/3 server implementation

Implements HTTP/3 support in the server, following the implementation
in ngtcp2's server.cc example. A lot of the function logic and structure
from there has been preserved for easier debugging and comparing to the
example if needed.

* Prepare client for HTTP/3

Use CRTP to prepare for a Http3ClientSession (not implemented yet).

Signed-off-by: Frank Osterfeld <[email protected]>

Author: frankosterfeld

CMakeLists.txt

@@ -108,6 +108,8 @@ else()
   option(OPENCMW_ENABLE_COVERAGE "Enable Coverage" OFF)
 endif()
 option(OPENCMW_ENABLE_CONCEPTS "Enable Concepts Builds" ${opencmw_MASTER_PROJECT})
+option(OPENCMW_DEBUG_HTTP "Enable verbose HTTP output for debugging" OFF)
+option(OPENCMW_PROFILE_HTTP "Enable verbose HTTP output for profiling" OFF)
 
 # Very basic PCH example
 option(ENABLE_PCH "Enable Precompiled Headers" OFF)
@@ -124,6 +126,14 @@ if(ENABLE_PCH)
     <utility>)
 endif()
 
+if(OPENCMW_DEBUG_HTTP)
+  target_compile_definitions(opencmw_project_options INTERFACE -DOPENCMW_DEBUG_HTTP)
+endif()
+
+if(OPENCMW_PROFILE_HTTP)
+  target_compile_definitions(opencmw_project_options INTERFACE -DOPENCMW_PROFILE_HTTP)
+endif()
+
 if(OPENCMW_ENABLE_TESTING)
   enable_testing()
   message("Building Tests.")

README.md

@@ -29,7 +29,7 @@ where the frame-work takes care of most of the communication, [data-serialisatio
 and buffering, settings management, Role-Based-Access-Control (RBAC), and other boring but necessary control system integrations
 while still being open to expert-level modifications, extensions or improvements.
 
-### General Schematic
+## General Schematic
 
 OpenCMW combines [ZeroMQ](https://zeromq.org/)'s [Majordomo](https://rfc.zeromq.org/spec/7/) with LMAX's [disruptor](https://lmax-exchange.github.io/disruptor/)
 ([C++ port](https://github.com/Abc-Arbitrage/Disruptor-cpp)) design pattern that both provide a very efficient lock-free mechanisms
@@ -38,7 +38,7 @@ for distributing, streaming and processing of data objects. A schematic outline
 
 ![OpenCMW architectural schematic](./assets/FAIR_microservice_schematic.svg)
 
-### Glossary
+## Glossary
 
 _Majordomo Broker_ or _'Broker':_ is the central authority where multiple workers can register their services, allowing clients to perform get, set or subscriptions requests.
 There can be multiple brokers for subset of services.
@@ -67,11 +67,11 @@ _Publisher:_ the [DataSourcePublisher](DataSourceExample.cpp) provides an interf
 ring-buffer with events from OpenCMW, REST services or other sources.
 While using disruptor ring-buffers is the preferred and most performing options, the client also supports classic patterns of registering call-back functions or returning `Future<reyly objects>` objects.
 
-### OpenCMW Majordomo Protocol
+## OpenCMW Majordomo Protocol
 
 The OpenCMW Majordomo [protocol](docs/MajordomoProtocol.md) is based on the [ZeroMQ Majordomo protocol](https://rfc.zeromq.org/spec/7/), both extending and slightly modifying it (see [the comparison](docs/Majordomo_protocol_comparison.pdf)).
 
-#### Service Names
+### Service Names
 
 Service names must always start with `/`. For consistency, this also applies to the built-in MDP broker services like `/mmi.service` (instead of `mmi.service` without leading slash as in ZeroMQ Majordomo). A service name is a non-empty alphanumerical string (also allowing `.`, `_`), that must start with `/` but not end with `/`. It contain additional `/` to denote a hierarchical structure.
 
@@ -84,7 +84,7 @@ Examples:
 - `/DeviceName/Acquisition/` - invalid (trailing slash)
 - `/a-service/` - invalid (`-` not allowed)
 
-#### Topics
+### Topics
 
 The "topic" field (frame 5 in the [OpenCMW MDP protocol](docs/Majordomo_protocol_comparison.pdf)) specifies the topic for subscriptions and GET/SET requests. It contains a URI with the service name as path and optional query parameters to specify further requests parameters and filter criteria.
 
@@ -99,7 +99,7 @@ Note that the whole path is considered the service name, and that there's no add
 
 See also the documentation for [mdp::Topic](src/core/include/Topic.hpp).
 
-#### URL to Service/Topic Mapping (mds/mdp and REST)
+### URL to Service/Topic Mapping (mds/mdp and REST)
 
 With both the MDS/MDP-based ZeroMQ clients as well as the REST interface, a common scheme is used to map from mdp/hds/http(s) URLs used for subscriptions and requests to the OpenCMW service name and topic fields.
 
@@ -120,7 +120,7 @@ Other examples are:
 - `mds://example.com:8080/DeviceName/Acquisition?signal=test` => service name `/DeviceName/Acquisition`, topic `/DeviceName/Acquisition?signal=test` (subscription via mds).
 - `mdp://example.com:8080/dashboards/dashboard1?what=header` => service name `/dashboards/dashboard1`, topic `/dashboards/dashboard1?what=header` (Request via mdp).
 
-### Compile-Time-Reflection
+## Compile-Time-Reflection
 
 The serialisers are based on a [compile-time-reflection](docs/CompileTimeSerialiser.md) that efficiently transform domain-objects to and from the given wire-format (binary, JSON, ...).
 Compile-time reflection will become part of [C++23](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p0592r4.html) as described by [David Sankel et al. , “C++ Extensions for Reflection”, ISO/IEC CD TS 23619, N4856](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/n4856.pdf).
@@ -142,7 +142,7 @@ provides also an optional light-weight `constexpr` annotation template wrapper `
 that in turn can be used to (re-)generate and document the class definition (e.g. for other programming languages or projects that do not have the primary domain-object definition at hand)
 or to generate a generic [OpenAPI](https://swagger.io/specification/) definition. More details can be found [here](docs/CompileTimeSerialiser.md).
 
-### Building from source
+## Building from source
 
 Note that building from source is only required if you want to modify opencmw-cpp itself.
 
@@ -151,7 +151,7 @@ In that case, rather take a look at the project [opencmw-cpp-example](https://gi
 
 For concrete build instructions, please check the [build instructions page](docs/BuildInstructions.md).
 
-### Example
+## Example
 
 For an example on how to implement a simple, first service using opencmw-cpp, please take a look at the project [opencmw-cpp-example](https://github.com/alexxcons/opencmw-cpp-example).
 
@@ -217,7 +217,7 @@ or RESTful (HTTP)-based high-level protocols, or through a simple RESTful web-in
 [comment]: <> (The basic HTML rendering is based on XXX template engine and can be customised. For more efficient, complex and cross-platform)
 [comment]: <> (UI designs it is planned to allow embedding of WebAssembly-based &#40;[WASM]&#40;https://en.wikipedia.org/wiki/WebAssembly&#41;&#41; applications.)
 
-### Performance
+## Performance
 
 The end-to-end transmission achieving roughly 10k messages per second for synchronous communications and
 about 140k messages per second for asynchronous and or publish-subscribe style data acquisition (TCP link via locahost).
@@ -241,17 +241,74 @@ Your mileage may vary depending on the specific domain-object, processing logic,
 but you can check and compare the results for your platform using the [RoundTripAndNotifyEvaluation](RoundTripAndNotifyEvaluation.cpp)
 and/or [MdpImplementationBenchmark](MdpImplementationBenchmark.cpp) benchmarks.
 
-### Documentation
+## Testing HTTP/3
+
+The openCMW REST interface supports both HTTP/2 and HTTP/3. When connecting from a browser, the browser typically first connects
+via HTTP/2. HTTP/2 responses contain a "alt-svc" header that informs the browser about the availability of HTTP/3. The browser
+then should switch to HTTP/3. Note that if HTTP/3 fails for any reason (certificates, server error etc.), the browser might remember
+that and not try HTTP/3 again. How to reset this depends on the browser. What I did in Google Chrome
+
+ 1. open an private tab, open the developer console, go to "Network", enable the "Protocol" column
+ 2. connect to the service; verify in the developer console that HTTP/3 is used (Protocol should switch to`H3` after the first `H2` request)
+ 3. To reset browser status, close *all* private tabs and open a new one.
+
+### SSL Certificates
+
+QUIC/HTTP/3 requires the use of TLS, unencrypted servers are not possible.
+
+At least in Google Chrome, the TLS stack used for QUIC seems quite separate from the normal settings, and its much stricter than for HTTP/1/2. *If
+anything goes wrong here, Google Chrome will silently stick with HTTP/2, or show you an error, if you're lucky*.
+
+Caveats:
+
+ 1. The certificate must be trusted inside the Chrome Certificate store (e.g. on Mac, trusting it in Keychain might silence the warning for HTTP1/2,
+but HTTP/3 will still fail. Add the server's public key under chrome://certificate-manager/.
+ 2. The hostname in the certificate must match what you connect via in the browser. "Works-everywhere self-signed certificates" I couldn't get to work.
+What I did: Create a certificate for hostname `foobar`, edit `/etc/hosts` to resolve `foobar` to the test host's IP address. Enter e.g. `https://foobar:8080`
+in the browser, instead of the IP address.
+ 3. For HTTP/2, you can ignore Chrome's warning with "Proceed anyway" or similar. This does not make the QUIC stack trust the certificate, HTTP/3 will
+not be used.
+ 4. To get the certificate's fingerprint in base64:
+```
+openssl x509 -in /path/to/demo_public.crt -noout -pubkey \
+  | openssl pkey -pubin -outform DER \
+  | openssl dgst -sha256 -binary \
+  | openssl enc -base64
+```
+ 3. Start Chrome with these parameters to make Chrome trust the certificate
+```
+.../Google\ Chrome --enable-quic \
+  --origin-to-force-quic-on=<yourhost>:8080 \
+  --ignore-certificate-errors-spki-list=<base64fingerprint> \
+  --user-data-dir=/tmp/quic-test-profile \
+  --no-sandbox
+```
+
+ For verbose logging (netlog can be viewed in the [https://netlog-viewer.appspot.com/#import](netlog viewer)), add
+```
+  --enable-logging=stderr --v=3 \
+  --log-net-log=netlog.json --quic-version=h3
+```
+
+### curl
+
+curl needs to be built with QUIC/HTTP/3 enabled, which is not the case at least on Ubuntu 24.04. I used the docker image `badouralix/curl-http3`:
+
+```
+docker run --rm badouralix/curl-http3 curl -k -vvvv --http3 https://<yourhost>:8080/loadTest?topic=1&intervalMs=40&payloadSize=4096&nUpdates=100&LongPollingIdx=Next"
+```
+
+## Documentation
 
 .... more to follow.
 
-### Don't like Cpp?
+## Don't like Cpp?
 
 For prototyping applications or services that do not interact with hardware-based systems, a Java-based
 [OpenCMW](https://github.com/fair-acc/opencmw-java) twin-project is being developed which follows the same functional style
 but takes advantage of more concise implementation and C++-based type safety.
 
-### Acknowledgements
+## Acknowledgements
 
 The implementation heavily relies upon and re-uses time-tried and well-established concepts from [ZeroMQ](https://zeromq.org/)
 (notably the [Majordomo](https://rfc.zeromq.org/spec/7/) communication pattern, see [Z-Guide](https://zguide.zeromq.org/docs/chapter4/#Service-Oriented-Reliable-Queuing-Majordomo-Pattern)

cmake/DependenciesNative.cmake

@@ -1,32 +1,118 @@
-# Build a static version of openssl to link into
+include(ExternalProject)
+include(GNUInstallDirs)
+
+set(OPENSSL_C_FLAGS "-O3 -march=x86-64-v3" CACHE STRING "OpenSSL custom CFLAGS" FORCE)
+set(OPENSSL_CXX_FLAGS "-O3 -march=x86-64-v3" CACHE STRING "OpenSSL custom CXXFLAGS" FORCE)
 set(OPENSSL_INSTALL_DIR "${CMAKE_BINARY_DIR}/openssl-install")
-add_library(OpenSSL::Crypto STATIC IMPORTED GLOBAL)
-add_library(OpenSSL::SSL STATIC IMPORTED GLOBAL)
-add_dependencies(OpenSSL::Crypto PUBLIC openssl-build)
-add_dependencies(OpenSSL::SSL PUBLIC openssl-build)
-set_target_properties(OpenSSL::Crypto PROPERTIES
+
+# Build custom OpenSSL with QUIC support
+ExternalProject_Add(OpenSslProject
+        GIT_REPOSITORY https://github.com/openssl/openssl.git
+        GIT_TAG openssl-3.5.0 # 3.5.0 required for server-side QUIC support
+        GIT_SHALLOW ON
+        BUILD_BYPRODUCTS ${OPENSSL_INSTALL_DIR}/lib64/libcrypto.a ${OPENSSL_INSTALL_DIR}/lib64/libssl.a
+        CONFIGURE_COMMAND COMMAND ./Configure CFLAGS=${OPENSSL_C_FLAGS} CXXFLAGS=${OPENSSL_CXX_FLAGS} no-shared no-tests --prefix=${OPENSSL_INSTALL_DIR} --openssldir=${OPENSSL_INSTALL_DIR} linux-x86_64
+        UPDATE_COMMAND ""
+        BUILD_COMMAND make -j
+        INSTALL_COMMAND make install_sw # only installs software components (no docs, etc)
+        BUILD_IN_SOURCE ON
+)
+
+add_library(openssl-crypto-static STATIC IMPORTED GLOBAL)
+add_dependencies(openssl-crypto-static OpenSslProject)
+set_target_properties(openssl-crypto-static PROPERTIES
         IMPORTED_LOCATION "${OPENSSL_INSTALL_DIR}/lib64/libcrypto.a"
         INTERFACE_INCLUDE_DIRECTORIES "${OPENSSL_INSTALL_DIR}/include"
 )
-set_target_properties(OpenSSL::SSL PROPERTIES
+
+add_library(openssl-ssl-static STATIC IMPORTED GLOBAL)
+add_dependencies(openssl-ssl-static OpenSslProject)
+set_target_properties(openssl-ssl-static PROPERTIES
         IMPORTED_LOCATION "${OPENSSL_INSTALL_DIR}/lib64/libssl.a"
         INTERFACE_INCLUDE_DIRECTORIES "${OPENSSL_INSTALL_DIR}/include"
 )
-get_target_property(libcryptoa OpenSSL::Crypto IMPORTED_LOCATION)
-get_target_property(libcryptoaloc OpenSSL::Crypto LOCATION)
-set(OPENSSL_C_FLAGS "-O3 -march=x86-64-v3" CACHE STRING "OpenSSL custom CFLAGS" FORCE)
-set(OPENSSL_CXX_FLAGS "-O3 -march=x86-64-v3" CACHE STRING "OpenSSL custom CXXFLAGS" FORCE)
-add_custom_command(
-        OUTPUT ${OPENSSL_INSTALL_DIR}/lib64/libcrypto.a ${OPENSSL_INSTALL_DIR}/lib64/libssl.a
-        COMMAND ${FETCHCONTENT_BASE_DIR}/openssl-source-src/Configure CFLAGS=${OPENSSL_C_FLAGS} CXXFLAGS=${OPENSSL_CXX_FLAGS} no-shared no-tests --prefix=${OPENSSL_INSTALL_DIR} --openssldir=${OPENSSL_INSTALL_DIR} linux-x86_64
-        COMMAND make -j
-        COMMAND make install_sw # only installs software components (no docs, etc)
-        COMMENT "Build openssl as a static library"
-        WORKING_DIRECTORY ${FETCHCONTENT_BASE_DIR}/openssl-source-build
+
+option(ENABLE_NGHTTP_DEBUG "Enable verbose nghttp2 debug output" OFF)
+
+ExternalProject_Add(Nghttp2Project
+        GIT_REPOSITORY https://github.com/nghttp2/nghttp2
+        GIT_TAG v1.65.0
+        GIT_SHALLOW ON
+        BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/nghttp2-install/${CMAKE_INSTALL_LIBDIR}/libnghttp2.a
+        UPDATE_COMMAND ""
+        CMAKE_ARGS
+        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}/nghttp2-install
+        -DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
+        -DENABLE_LIB_ONLY:BOOL=ON
+        -DENABLE_HTTP3:BOOL=OFF
+        -DENABLE_DEBUG:BOOL=${ENABLE_NGHTTP_DEBUG}
+        -DBUILD_STATIC_LIBS:BOOL=ON
+        -BUILD_SHARED_LIBS:BOOL=OFF
+        -DENABLE_DOC:BOOL=OFF
+)
+
+add_library(nghttp2-static STATIC IMPORTED GLOBAL)
+set_target_properties(nghttp2-static PROPERTIES
+        IMPORTED_LOCATION "${CMAKE_BINARY_DIR}/nghttp2-install/${CMAKE_INSTALL_LIBDIR}/libnghttp2.a"
+        INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_BINARY_DIR}/nghttp2-install/include"
 )
-add_custom_target(openssl-build ALL
-        DEPENDS ${OPENSSL_INSTALL_DIR}/lib64/libcrypto.a ${OPENSSL_INSTALL_DIR}/lib64/libssl.a
+add_dependencies(nghttp2-static Nghttp2Project)
+
+ExternalProject_Add(Nghttp3Project
+        GIT_REPOSITORY https://github.com/ngtcp2/nghttp3.git
+        GIT_TAG v1.10.1
+        GIT_SHALLOW ON
+        BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/nghttp3-install/${CMAKE_INSTALL_LIBDIR}/libnghttp3.a
+        UPDATE_COMMAND ""
+        CMAKE_ARGS
+        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}/nghttp3-install
+        -DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
+        -DENABLE_LIB_ONLY:BOOL=ON
+        -DENABLE_DEBUG:BOOL=${ENABLE_NGHTTP_DEBUG}
+        -DBUILD_STATIC_LIBS:BOOL=ON
+        -DBUILD_SHARED_LIBS:BOOL=OFF
+        -DENABLE_DOC:BOOL=OFF
+)
+
+ExternalProject_Add(NgTcp2Project
+        GIT_REPOSITORY https://github.com/ngtcp2/ngtcp2.git
+        GIT_TAG v1.13.0
+        GIT_SHALLOW ON
+        PREFIX ${CMAKE_BINARY_DIR}/ngtcp2-install
+        BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/ngtcp2-install/${CMAKE_INSTALL_LIBDIR}/libngtcp2.a ${CMAKE_BINARY_DIR}/ngtcp2-install/${CMAKE_INSTALL_LIBDIR}/libngtcp2_crypto_ossl.a
+        UPDATE_COMMAND ""
+        CMAKE_ARGS
+        -DOPENSSL_ROOT_DIR:PATH=${OPENSSL_INSTALL_DIR}
+        -DENABLE_OPENSSL:BOOL=ON
+        -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}/ngtcp2-install
+        -DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
+        -DENABLE_LIB_ONLY:BOOL=ON
+        -DENABLE_DEBUG:BOOL=${ENABLE_NGHTTP_DEBUG}
+        -DBUILD_STATIC_LIBS:BOOL=ON
+        -DBUILD_SHARED_LIBS:BOOL=OFF
+        DEPENDS openssl-crypto-static openssl-ssl-static
+)
+
+add_library(ngtcp2-static STATIC IMPORTED GLOBAL)
+set_target_properties(ngtcp2-static PROPERTIES
+        IMPORTED_LOCATION "${CMAKE_BINARY_DIR}/ngtcp2-install/${CMAKE_INSTALL_LIBDIR}/libngtcp2.a"
+        INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_BINARY_DIR}/ngtcp2-install/include"
+)
+add_dependencies(ngtcp2-static NgTcp2Project)
+
+add_library(ngtcp2-crypto-ossl-static STATIC IMPORTED GLOBAL)
+set_target_properties(ngtcp2-crypto-ossl-static PROPERTIES
+        IMPORTED_LOCATION "${CMAKE_BINARY_DIR}/ngtcp2-install/${CMAKE_INSTALL_LIBDIR}/libngtcp2_crypto_ossl.a"
+        INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_BINARY_DIR}/ngtcp2-install/include"
 )
+add_dependencies(ngtcp2-crypto-ossl-static NgTcp2Project)
+
+add_library(nghttp3-static STATIC IMPORTED GLOBAL)
+set_target_properties(nghttp3-static PROPERTIES
+        IMPORTED_LOCATION "${CMAKE_BINARY_DIR}/nghttp3-install/${CMAKE_INSTALL_LIBDIR}/libnghttp3.a"
+        INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_BINARY_DIR}/nghttp3-install/include"
+)
+add_dependencies(nghttp3-static Nghttp3Project)
 
 add_library(mustache INTERFACE)
 target_include_directories(mustache INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}/3rd_party/kainjow)
@@ -38,6 +124,7 @@ FetchContent_Declare(
         GIT_TAG v4.3.5 # latest as of 2025-03-27
 )
 set(ZMQ_BUILD_TESTS OFF CACHE BOOL "Build the tests for ZeroMQ")
+
 # suppress warnings for missing zeromq dependencies by disabling some features
 set(WITH_TLS OFF CACHE BOOL "TLS support for ZeroMQ WebSockets")
 set(BUILD_SHARED OFF CACHE BOOL "Build cmake shared library")
@@ -56,12 +143,6 @@ FetchContent_Declare(
         GIT_TAG v1.2.12 # latest v1.2.12
 )
 
-FetchContent_Declare(
-        openssl-source
-        GIT_REPOSITORY https://github.com/openssl/openssl.git
-        GIT_TAG openssl-3.4.1
-)
-
-FetchContent_MakeAvailable(cpp-httplib zeromq openssl-source)
+FetchContent_MakeAvailable(cpp-httplib zeromq)
 
 list(APPEND CMAKE_MODULE_PATH ${catch2_SOURCE_DIR}/contrib) # replace contrib by extras for catch2 v3.x.x

concepts/client/CMakeLists.txt

@@ -1,13 +1,10 @@
-if(NOT EMSCRIPTEN)
-  add_executable(RestSubscription_example RestSubscription_example.cpp)
-  target_link_libraries(
-    RestSubscription_example
-    PRIVATE core
-            client
-            opencmw_project_options
-            opencmw_project_warnings
-            assets::rest)
-endif()
+add_executable(LoadTest_client LoadTest_client.cpp)
+target_link_libraries(
+  LoadTest_client
+  PRIVATE core
+          client
+          opencmw_project_options
+          opencmw_project_warnings)
 
 add_executable(RestSubscription_client RestSubscription_client.cpp)
 target_link_libraries(