Skip to content

Improvements in OTLP encoding logic required for internal logging safety #1746

@jmacd

Description

@jmacd

Following #1735, here are a list of improvements we desire in the OTLP ProtoBuffer encoding object of crates/pdata/src/otlp/common.rs

  1. Ability to encode into a limited-size buffer to avoid unbounded memory use
  2. Ability to encode into a single pre-allocated &mut [u8]
  3. Ability to safely truncate long strings appending a "..." or other notice
  4. Ability to safely truncate a list of top-level fields (e.g., the attributes list of a log record)
  5. Ability to automatically generate the dropped_attributes_count field of LogRecord, Span, Resource etc by reserving 5 bytes
  6. Ability to choose (somehow) the number of bytes dedicated to placeholders; the current 4-byte approach is wasteful for small buffers

Ultimately with all of this done we will be able to safely log into pre-allocated buffers and obtain usefully truncated and/or formatted messages without allocating memory inside a tokio tracing event.

Part of #1736

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions