Ingestion with HTTP lacks request compression support

[hi-octane]

Hello Seq Team,

I've tried Seq version 2021.3 and it looks like Ingestion with HTTP (POST to /api/events/raw?clef) ignores content-encoding header and simply fails if json compressed with gzip or brotli. Consequently, intensive apps produce huge traffic sending numerous events in plain JSON into Seq.

With libs like https://github.com/alexanderkozlenko/aspnetcore-request-decompression, or using .NET Core GzipStream and BrotliStream, support for compression encodings can be added in just a few lines of code. Maybe you can add this feature to your roadmap?

[liammclennan]

Thanks for the suggestion. It's an interesting idea, and I can see some scenarios where it would be useful (very large servers, very large payloads). We will give it some thought.

If you are looking to optimize the performance of your Seq instance you might also try one of the Seq client libraries (Serilog etc) which may perform better than using the HTTP API directly.

[hi-octane]

Thanks for your interest and quick reply!
The problem I'm trying to solve is related not only to servers that produce tons of messages, but also servers that have terrible slow interconnections. I started investigating compression when found that the size of the delivery queue steadily increases if a demo app for a client is located in Ali Cloud (China), and Seq is running in Amazon EU-West. Right now I solved this problem with an ad-hoc http proxy that provides compression, but it's a no-go if the project finally reaches a production state.

I've reviewed available .NET sinks sources, and found that they differ mostly in approaches of batching and storage reliability. Serilog-sink-seq, NLog.Target.Seq, Seq.Extensions.Logging, etc. - all of them finally post events via the http api. As a result, any of them will produce the same traffic between an application server and the Seq server.

Alternatively, maybe, you can add an additional extension point, to allow a custom Seq App to register one more ASP.NET HTTP Controller on startup. The registered controller will then process incoming traffic in any compressed or binary format, unpack it, prepare events and push them into Seq's DB via an internal API? This approach may allow supporting IOT-friendly protocols like WebSockets, MQTT, etc.

[liammclennan]

Seq already supports custom input apps such as https://github.com/datalust/sqelf and https://github.com/datalust/seq-input-rabbitmq.

[nblumhardt]

The main limitation faced by (network server) custom inputs right now is that they can't share a port with the rest of the Seq API (and thus miss out on a big slice of infrastructure support, like TLS termination). Not a show-stopper - the GELF and Syslog inputs just do without - but it'd be amazing if we could lift this limitation somehow in the future.

[nblumhardt]

Hi @hi-octane; we've decided to move this suggestion to a discussion while we gather more feedback on it and consider how it might fit into a future release. We don't expect to include it in 2022.1 or .2 at this stage. Thanks for getting the ball rolling, though!