New Blog Post: Announcing OpenTelemetry Protocol with Apache Arrow - Phase 2

[jmacd]

The OpenTelemetry Protocol with Apache Arrow project would post a project update as we embark on Phase 2 of the project. This post will be the third in a series, preceded by:
https://opentelemetry.io/blog/2023/otel-arrow/
https://opentelemetry.io/blog/2024/otel-arrow-production/

OTel-Arrow Phase 2 (as originally described in the OTEP, in the donation) will investigate an end-to-end pipeline for OpenTelemetry data using Apache Arrow data frames as the pipeline data representation. We anticipate that using OpenTelemetry Protocol with Apache Arrow (OTAP) instead OTLP as the intermediate representation will result in substantial savings, lower CPU and memory usage across the entire pipeline. We will experiment with writing new, column-oriented processing logic for OpenTelemetry data.

We choose the Rust language for this project due to its popularity in the Apache Arrow ecosystem. We are specifically interested in using the Apache DataFusion project's embedded query engine as a component in OpenTelemetry pipelines.

We will investigate the use of an OTAP-first OpenTelemetry-Rust SDK. We are specifically interested in using the OpenTelemetry Weaver project to generate type-safe bindings for exporting into an OTAP pipeline.

The blog post will summarize these points and remind users that they can contribute, where to find us, and so on.

[tiffany76]

Thanks, @jmacd! We look forward to reading your draft.

For the sake of completeness, are the SIG and sponsor the same as last time?

SIG: OTel-Arrow
Sponsor: @lquerel

[austinlparker]

FYI, the GC/TC would like to discuss this before a blog goes up about it.