loki.process should support extracting data from strucutured metadata

[baurmatt]

Background

When passing log streams to loki.process, it's currently not possible to use existing structured metadata. Being able to work with existing structured metadata is crucial for transform those log entries.

Use case

We send log data to Alloy via loki.source.api. Received log entries contain a structured metadata field called pod. We need to transform it into a log label called instance.

Proposal

Add a new stage stage.structured_metadata_unpack which unpacks existing structured metadata into extracted values. This is done by 1:1 copying all data over. Optionally one could implement exclude/include parameters for this stage to limit the structured metadata to certain keys(/values).

Tip

_{React with 👍 if this issue is important to you.}

[baurmatt]

I'd be happy to implement this, just need some directions how this is preferred to be implemented.

[kalleep]

Hey yes we could implement something to handle this usecase. Could you write a proper proposal for how you envision that this would work? Then we can review it or make suggestions

[baurmatt]

Thanks! :) I've updated the description with the proposal. Two things regarding this proposal:

• Not 100% happy with the stage (structured_metadata_unpack) naming. Naming is hard 🙃
• Open to discuss different solution where we e.g. put this functionality directly into existing stages
  • include_structured_metadata option for stage.labels to have it lookup data in structured metadata as well
  • include_structured_metadata option for loki.process to have it per default include structured metadata in extracted values

[kalleep]

Thanks, will make sure to bring this up when we have our next proposal review meeting!

[dehaansa]

Maybe we could follow a similar pattern to the new truncate stage and have an (optional) source_type when a stage needs to address a specific source of data.

[baurmatt]

@dehaansa Not quite sure what you're referencing to. Do you mean currently existing stages should be extended by a source_type option or do you mean a new stage like this:

loki.process "default" {
  forward_to = [loki.write.default.receiver]

  stage.transform {
    rule {
      name             = "^key_" # Optional. Match the key name which should be copied from source to destination
      destination_type = "label" # Required. To which type of data to copy. Can be `extracted`, `label` or `structured_metadata`.
      source_type      = "structured_metadata" # Required. From which type of data to copy. Can be `extracted`, `label` or `structured_metadata`.
      override         = true # Optional. If the destination already contains a key -> Override. Default: true
      source_delete    = true # Optional. If the data should be deleted in source_type. Default: true
    }
  }
}

[dehaansa]

@dehaansa Not quite sure what you're referencing to. Do you mean currently existing stages should be extended by a source_type option or do you mean a new stage like this:
...

Extending existing stages with a source_type options that defaults to the existing behavior seems like the best choice to me.