[Libbeat][Filebeat][outputs] Add Codec and exporter to serialize and export batch of beat events in `OTLP` format

[shivanshuraj1333]

Describe the enhancement:

To make Filebeat compatible with Opentelemetry, i.e. exporting beat events in OTLP format so that opentelemtry collector can ingest Filebeat beat events, we need a codec to serialize the batch of beat events into OTLP Log Data Model.

To do that, we need to add:

1. a Codec which will be converting a batch of beat events into an OTLP log data model, the codec should be added here libbeat/outputs/codec in addition to existing format and JSON codecs.
2. an Exporter which will be supporting OTLP over HTTP/gRPC protocol, this exporter needs to be added here libbeat/outputs alongside kafka, logstash, and redis exporters and can be called as otlp

Implementation details:

The above diagram explains the translation of a batch of beat events into OTLP Log Data Model.

Describe a specific use case for the enhancement or feature:

Filebeat is a great tool, and with the increased adoption of Opentelemetry it's really nice to make Filebeat compatible with Opentelemetry.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[shivanshuraj1333]

@andrewkroh , can you help me in getting this reviewed? So that I can open a PR.
Thanks!

[cmacknz]

Hi @shivanshu1333, we are thinking about this internally. I'm not sure when we'll have a decision.

For context, adding new outputs has a long tail of work in the form of ongoing user deployment support after they are introduced. Writing the code is only part of work here. We need to be careful about when we introduce new outputs to ensure we can properly support and test them in the various environments and configurations they will be deployed in.

[shivanshuraj1333]

Hi @cmacknz, thanks for responding. Do we have any decision yet?

The changes required for outputs are ready and tested locally; the changes include

1. one output codec for otlp which will be added here (libbeat/outputs/codec)
2. one otlp exporter which will be added here (libbeat/outputs)
   These two changes alone are complete to add this feature.

Can you please elaborate on

"adding new outputs has a long tail of work in the form of ongoing user deployment support after they are introduced"

To resolve this

We need to be careful about when we introduce new outputs to ensure we can adequately support and test them in the various environments and configurations they will be deployed in

How about letting the changes in? Let's not release it until it's thoroughly tested and mark this as an alpha feature whenever we're ready to release; eventually, we can mark it as a beta feature.

This way, we can start progressing in the right direction, the changes will be well tested, and we'll make Filebeat compatible with OpenTelemetry Collector, increasing adoption of Filebeat among users of OpenTelemetry Collector.

Kindly let me know your thoughts.

[cmacknz]

We are still thinking about when and how best to support OpenTelemetry, I don't think there will be a decision with respect to Beats soon.

Besides the Beats' maintainers making the time to support and test an OpenTelemetry output to the extent we would like, we would ideally like to wait for the OpenTelemetry logging protocol to stabilize (currently experimental).

There is also an active proposal to adopt the Elastic Common Schema we use in our logs into OpenTelemetry (see open-telemetry/oteps#197) and we would ideally wait for a decision on that proposal as well.

[hartfordfive]

There are multiple configuration options in filebeat which are also indicated as experimental (see full config for v8.4. Having this considered, would it not be reasonable to add in this feature and clearly state that it is experimental? I think it's safe to assume Opentelemetry is here to stay especially as it's been a CNCF project since May 2019. It's starting to gain some popularity in the observability realm for many reasons. Having said that, I realize that Elastic does have ECS to adress consistency with log schemas but that's not an option that everyone wants to necessarily implement as it's solution specific.

[shivanshuraj1333]

@joshdover this is the original tracking issue with the idea to introduce OTLP codec and exporter, now we have modified our code to do serialization on the fly in the exporter itself. And we've a separate package for metrics.

This is in continuation with your discussion with Lalit.
Should we raise a PR?

[hartfordfive]

Regarding the following comment in the linked issue which asks if a special receiver should be built in the opentelemetry-collector or if filebeat should implement an OTLP output option, it would be beneficial for filebeat to have the capability to encode in the OTLP format. I'm wondering if filebeat could also add a new processor (e.g.: encode_otlp) which could allow the log events collected to be transformed to the OTLP log data model defined here. This could also potentially simplify applying a specific schema directly in filebeat for some of the sub-fields. (e.g.: standardizing key names in the "Attributes" section). Considering OpenTelemetry has adopted ECS for schema standardization, I feel this approach seems it would make sense.

As an example, someone could encode messages in the OTLP model and then publish them to Kakfa. They could then consume them from the topic, process them with any tool and then index them into Elasticsearch.

[secustor]

As an example, someone could encode messages in the OTLP model and then publish them to Kafka.

This is a similar setup we use and we are considering dropping beats and logstash as we want to standardize on OTLP.