Ingestion with HTTP lacks request compression support

[hi-octane]

Hello Seq Team,

I've tried Seq version 2021.3 and it looks like Ingestion with HTTP (POST to /api/events/raw?clef) ignores content-encoding header and simply fails if json compressed with gzip or brotli. Consequently, intensive apps produce huge traffic sending numerous events in plain JSON into Seq.

With libs like https://github.com/alexanderkozlenko/aspnetcore-request-decompression, or using .NET Core GzipStream and BrotliStream, support for compression encodings can be added in just a few lines of code. Maybe you can add this feature to your roadmap?

[liammclennan]

Thanks for the suggestion. It's an interesting idea, and I can see some scenarios where it would be useful (very large servers, very large payloads). We will give it some thought.

If you are looking to optimize the performance of your Seq instance you might also try one of the Seq client libraries (Serilog etc) which may perform better than using the HTTP API directly.

[hi-octane]

Thanks for your interest and quick reply!
The problem I'm trying to solve is related not only to servers that produce tons of messages, but also servers that have terrible slow interconnections. I started investigating compression when found that the size of the delivery queue steadily increases if a demo app for a client is located in Ali Cloud (China), and Seq is running in Amazon EU-West. Right now I solved this problem with an ad-hoc http proxy that provides compression, but it's a no-go if the project finally reaches a production state.

I've reviewed available .NET sinks sources, and found that they differ mostly in approaches of batching and storage reliability. Serilog-sink-seq, NLog.Target.Seq, Seq.Extensions.Logging, etc. - all of them finally post events via the http api. As a result, any of them will produce the same traffic between an application server and the Seq server.

Alternatively, maybe, you can add an additional extension point, to allow a custom Seq App to register one more ASP.NET HTTP Controller on startup. The registered controller will then process incoming traffic in any compressed or binary format, unpack it, prepare events and push them into Seq's DB via an internal API? This approach may allow supporting IOT-friendly protocols like WebSockets, MQTT, etc.

[liammclennan]

Seq already supports custom input apps such as https://github.com/datalust/sqelf and https://github.com/datalust/seq-input-rabbitmq.

[nblumhardt]

The main limitation faced by (network server) custom inputs right now is that they can't share a port with the rest of the Seq API (and thus miss out on a big slice of infrastructure support, like TLS termination). Not a show-stopper - the GELF and Syslog inputs just do without - but it'd be amazing if we could lift this limitation somehow in the future.

[nblumhardt]

Hi @hi-octane; we've decided to move this suggestion to a discussion while we gather more feedback on it and consider how it might fit into a future release. We don't expect to include it in 2022.1 or .2 at this stage. Thanks for getting the ball rolling, though!

[Chugunenok]

Hi SEQ Team!
We are using SEQ for react-native mobile app logs (serilogger Javascript NPM package + custom persistent storage).
Compressed version of our logs is 6x times smaller (we are using Pako javascript NPM package to GZIP data), so compressed logs uploads much faster and utilizes less mobile traffic.
It would be great if we can send compressed data directly to SEQ without using middleware that unzips the data and then sends it to SEQ.
Thanks!

[mitchellsuzuki]

Hi SEQ,
this is a feature of HTTP Ingestion that would greatly help my team.
I have a number of events that fail to ingest due to exceeding the maximum event body size.

{"Error":"The event on line 1 exceeded the maximum event body size; first 1024 chars: {\"@t\":\"2024-09-11T15:51:47.206635314Z\",\"@m\":\"[15:51:47 Information] Microsoft.AspNetCore.Hosting.Diagnostics\",\"kubernetes\":{\"pod_name\":\"webapp-fintatech-client-identity-698d7d494d-f92b6\",\"namespace_name\":\"alpha-fintatech\",\"pod_id\":\"41100b6c-69af-4241-bbb9-4211858cde39\",\"host\":\"ip-10-10-0-116.us-east-2.compute.internal\",\"container_name\":\"webapp-fintatech-client-identity\",\"docker_id\":\"ef7de205eeb71b2a5aaf130b92d4bcefac0bfca59aab1b0c1a8cf841492c68b2\",\"container_hash\":\"718902333874.dkr.ecr.us-east-1.amazonaws.com/fintatech/trading-data-server/fintatech.tds.clientidentity@sha256:35b4c86a54d2eb48a59748fe0602c423fa45afec8c42a7350d44cc4ac66551e0\",\"container_image\":\"718902333874.dkr.ecr.us-east-1.amazonaws.com/fintatech/trading-data-server/fintatech.tds.clientidentity:2.3.6\"},\"fluentbit_log\":\"true\",\"@l\":\"Information\"}{\"@t\":\"2024-09-11T15:51:47.206665519Z\",\"@m\":\"Request starting HTTP/1.1 POST http://webapp-fintatech-client-identity.alpha-fintatech.svc.cluster.local:5000/connect/token application/x-www-form-urlencoded 2."}

[nblumhardt]

Hi Mitchell,

This would not be affected by request compression - the limit is applied to the JSON data ingested into the event store, so that retrieval performance is kept reasonable.

To change the limit, at the cost of event store performance, see Settings > System > Raw event body limit.

Hope this helps,
Nick