Mapping properties to labels

What is a label?

According to Grafana Loki documentation:

Labels are key value pairs and can be defined as anything! We like to refer to them as metadata to describe a log stream. Labels in Loki perform a very important task: They define a stream. More specifically, the combination of every label key and value defines the stream. If just one label value changes, this creates a new stream.

Why do we need explicit mapping?

Every unique combination of labels creates a new separate stream in Loki. Big count of streams can kill Loki and called high cardinality. In case of structured logging every log event property translates into label, which cause a lot of metadata for a simple log entry. For example, next labels are created for a standard ASP.NET Core Web API call: ActionId, ActionName, ConnectionId, Message, MessageTemplate, RequestId, RequestPath, SourceContext, Timestamp, Level. This is a lot of labels with a dynamic values.

How Serilog.Sinks.Grafana.Loki dealt with that problem in the past

First public version (v3) of this sink has only possibility to exclude some properties from mapping to the labels. This wasn't a good solution, cause you needed to add all exclusion rules explicitly, what becomes a hell with a huge amount of labels (especially in cases with external libs.

We have changed this behavior in v4. It brought us a filtrationMode and filtrationLabels, where you could select one of modes: Include/Exclude and define properties' names to map 'em to the labels. Or you could leave it blank and in this case you will face the high cardinality problem. We always recommended to use Include mode with some always present labels (such a app name). But in last discussion we have found that this behavior is straightforward, but has no sense.

The current solution - properties as labels

From v8 we propose you a clear way - you define only properties that should be mapped into a labels (or none if you don't need it). For this case use a new propertiesAsLabels configuration parameter. The rest of the properties are passed to the text formatter and included as part of the log entry body.

V9 change: a property promoted via propertiesAsLabels is now also kept in the log body (in v8 it was removed once promoted). So it is available both as a stream label and inside the JSON line for | json queries. Global labels still take priority over property-derived labels when keys collide, and a property whose key matches a global label or the level label is skipped.

High-cardinality properties: use structured metadata

propertiesAsLabels is for low-cardinality values — every distinct label value creates a new stream, so promoting something like RequestId or UserId recreates the high-cardinality problem described above. For values you still want to filter on but that change on (almost) every event, use Structured metadata instead: it is queryable like a label but is not indexed, so it carries no cardinality cost.