OpenTelemetry logs, body vs attributes for structured logs unclear in spec and behaves inconsistently in exporters

[segevfiner]

There is something that is inconsistent/missing at this time from the OpenTelemetry logs specification and is causing me quite an headache while trying to implement it into our app.

The current spec defines:

Field: Body
Type: any.

Description: A value containing the body of the log record. Can be for example a human-readable string message (including multi-line) describing the event in a free form or it can be a structured data composed of arrays and maps of other values. Body MUST support any type to preserve the semantics of structured logs emitted by the applications. Can vary for each occurrence of the event coming from the same source. This field is optional.

Field: Resource
Type: Resource.

Description: Describes the source of the log, aka resource. Multiple occurrences of events coming from the same event source can happen across time and they all have the same value of Resource. Can contain for example information about the application that emits the record or about the infrastructure where the application runs. Data formats that represent this data model may be designed in a manner that allows the Resource field to be recorded only once per batch of log records that come from the same source. SHOULD follow OpenTelemetry semantic conventions for Resources. This field is optional.

But from this it is very unclear how structured logs should be mapped into this format, including what is the main message field when the body is an object rather than just a string, and results in mismatch between directly ingesting such logs versus using OpenTelemetry into various backends.

For example, the default instrumentation for Pino (Node.js) maps the message field into body and the rest into attributes, but when this is exported into Google Cloud Logs, it puts the body as textPayload and attributes as labels where labels are just a string to string mapping, which means structured fields such as err get flattened as JSON and become inconvenient to view and query, modifying the instrumentation (Directly changing its code) to put everything into body results in the expected mapping of putting it all under jsonPayload, but for Google Cloud Logs to recognize the main message field and display it nicely, it must be called message which is not part of the spec and I had to do manually in the configuration of Pino, otherwise it becomes inconvenient to view them messages as it just shows the body JSON payload instead of the message when browsing messages.

Trying the same with SigNoz, when the message is mapped into body and attributes, it again doesn't handle the structured err attribute yet and just shows it as a JSON string which is inconvenient to query and view. When mapping it all into body, it will parse the body JSON as expected, but it will display the entire body JSON in the table, with no concept of main message field, and no option to display a field of the body as a column, though you seem able to query them but without autocompletion of fields in query message box, which is also inconvient.

I'm not sure what's the canonical use for body and attributes here, and how I should handle it then to get the correct and most convenient display in different exporter as they don't seem to act the same...

See also open-telemetry/opentelemetry-js-contrib#3012

[trask]

hi @segevfiner! it's canonical to use attributes and not body. see https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/events.md#general-event-semantics

complex attributes are supported on logs (and will be soon on all signals), so you don't need to json encode

it may take some time for implementations to catch up to the specification though

[segevfiner]

So maybe what needs to change is the GCP exporter in the OpenTelemetry collector to account for that? It would work better to merge body and attributes into one JSON object and stick it into jsonPayload for it.

Also, this detail about that the body should normally be the main message field even for structured loggers should also be in the main LogRecord spec...

[segevfiner]

I wonder if we don't want to modify the GCP exporter to (possibly with a configuration):

1. Put body in jsonPayload.message
2. Put attributes in jsonPayload
3. Have a configuration to treat certain attributes as labels in addition to the one for resources.

[pellared]

Should this not be discussed under https://github.com/open-telemetry/opentelemetry-collector-contrib?

[mbrevda]

@dashpole - any chance you can weigh in here or tag someone that can?

[dashpole]

Thanks for bringing this up. I think OTel instrumentation is doing the right thing by using attributes. The solution for GCP (and likely other telemetry backends) is going to be to more elegantly handle complex attribute values. I've shared the feedback with the appropriate folks internally. I don't think changing the log exporter mapping in the collector GCP exporter is the right approach.

[mbrevda]

Thanks!

I don't think changing the log exporter mapping in the collector GCP exporter is the right approach

Any guidance on using the collector until telemetry.googleapis.com supports logs would be very helpful.

[dashpole]

OTTL doesn't have iterators yet (that I know of), so you would need to know the keys you want to move to the body beforehand. But you should be able to use the transform processor to put things into a structured body if you want:

processors:
  transform:
    log_statements:
      - context: log
        statements:
          # 1. Move the current body (string/map) into a nested field
          # This converts the body into a map if it wasn't one already.
          - set(cache, body)
          - set(body, {})
          - set(body["original_body"], cache)
          
          # 2. Merge your map-typed attribute into the now-structured body
          - merge_maps(body, attributes["app_metadata"], "upsert")
            where attributes["app_metadata"] != nil
          
          # 3. Clean up the original attribute
          - delete_key(attributes, "app_metadata")
            where attributes["app_metadata"] != nil

Note: I used an AI to generate the yaml above. It looks correct to me, but I haven't tested it myself.