Description
I think there should be a section here about time sharding.
Our use case involves collecting logs from 100,000s of mobile devices. Logs are concatenated client-side and uploaded to object storage. Once the objects are uploaded, an event-driven pipeline parses the logs and pushes them to a Loki cluster running in Kubernetes. Sometimes a single log file can contain log lines spanning multiple days, and the logs are not necessarily concatenated in chronological order.
We struggled with out-of-order log ingestion and considered building a customized pipeline step that would aggregate and sort larger batches of logs. Luckily, after trawling through the configuration parameters, time sharding came to the rescue and elegantly solved the problem.
It would have saved us a lot of time if the time sharding feature were documented alongside other automatic sharding capabilities of Loki.