The OTAP Dataflow library is a set of core Rust crates which combine to produce an OpenTelemetry pipeline support, for use as an embedded software component, providing a framework for collecting OpenTelemetry data.
[!NOTE] These Rust libraries are the main deliverable of Phase 2 of the OTel-Arrow project, as defined in the project phases. The
df_enginemain program built throughcargoinsrc/main.rsis provided as a means to test and validate OTAP pipelines built using the dataflow engine.
The OTAP Dataflow engine consists of a number of major pieces. Here are the major aspects of its design:
- OTel-Arrow first. The Apache Arrow record batch is the underlying data type used to represent "OTAP records", at the lowest level in our pipeline. The OTAP records format consists of a varying number of Arrow record batches per unit of data, representing hierarchical OpenTelemetry data in a column-oriented form, under a "star schema".
- Zero-copy to and from OTLP bytes. OTLP bytes are the standard for representing OpenTelemetry data on the wire. Through a custom implementation of Google Protocol buffers, we convert OTAP records directly to and from OTLP bytes without constructing intermediate data objects.
- Thread-per-core approach. Our design aims to support single-threaded
nothing-shared pipelines as our first priority. We make use of a
Local async
runtime
freeing the pipeline from synchronizing instructions.
Multi-threaded components are possible using
sharedadapters, but we choose single-threadedlocalcomponents when possible.
The basic unit of data in an OTAP Dataflow pipeline is the OTAP
pipeline data object, otap_df_otap::pdata::OtapPdata. In the
hierarchy of types a pipeline component interacts with, this crate
otap_df_otap::pdata crate is a focal point. The OtapPdata data
type is a struct consisting of "context" and "payload", where context
is used for routing "Ack" and "Nack" responses, and payload has two
equivalent, signal-specific representations:
- OTLP bytes (Logs, Traces, Metrics): A signal-specific enum of
Vec<u8>corresponding with one of the export requests (e.g.,ExportLogsServiceRequest). - OTAP records (Logs, Traces, Metrics): A signal-specific array of
Arrow
RecordBatchobjects defining aspects of the OpenTelemetry data model, where unused columns are omitted. For example, The Logs form of OTAP records consists of four record batches, corresponding with Logs, Log Attributes, Scope Attributes, and Resource Attributes.
Refer to the OTAP basics documentation.
The OTAP data model contains diagrams of the many N-to-1 relationships expressed within an OTAP request.
The otap_df_engine crate is located in crates/engine, here we
find the engine's overall architecture expressed:
- Local (unsynchronized) and shared (
Sync + Send) code paths - Queue-oriented message passing
- Separate control and data plane
- Effect handler for interacting with pipeline.
The engine's main entry point,
otap_df_engine::PipelineFactory<PData>, supports building a
single-thread pipeline for generic type PData. Generally, users do
not construct these, as they are managed by a controller instance.
Here are the key files to know about:
crates/engine/lib.rs: Effect handler extensions, pipeline factory
|-- attributes.rs: Host, process/container IDs
|-- context.rs: CPU, NUMA, thread context
|-- control.rs: NodeControlMsg, AckMsg, NackMsg
|-- effect_handler.rs: Component interfaces (network, clock, ack/nack)
|-- error.rs: Structured errors
|-- exporter.rs: Pipeline component (output only)
|-- message.rs: The data and control plane messages
|-- node.rs: The basic NodeId type
|-- pipeline_ctrl.rs: Timer state, channel to all nodes
|-- processor.rs: Pipeline component (input/input)
|-- receiver.rs: Pipeline component (input only)
|-- runtime_pipeline.rs: Builds the graph of component channels
The OTAP pipeline data type is defined here, therefore all of our
built-in components are provided here as well. The main entry point
into this crate is the otap_df_otap::pdata::OtapPdata type with its
two alternate representations, OTAP records and OTLP bytes, specific
by signal type.
The PData type also facilitates datatype-aware aspects of interacting
with the pipeline engine, including ProducerEffectHandlerExtension,
for receivers and processors to subscribe to the NodeControlMsg::Ack
and NodeControlMsg::Nack messages, and
ConsumerEffectHandlerExtension for processors and exporters to
notify the next recipient in the chain of subscribers.
Here are the key files to know that support the components in this crate:
crates/otap/lib.rs: OTAP Dataflow pipeline factory
|-- compression.rs: OTLP and OTAP compression settings
|-- encoder.rs: Computes OTAP from OTLP view representations
|-- metrics.rs Metrics logic shared by several components
|-- pdata.rs The OtapPdata type, effect handler extensions
|-- otap_grpc/ OTLP and OTAP shared gRPC support
|-- fixtures.rs Test support
|-- otap_mock.rs Test support
|-- testing/ Test support
All gRPC services are implemented using Tonic.
The major OTAP Dataflow components of otap_df_otap are listed next.
This component supports efficient low-level manipulation of OTAP records. For example, this component supports O(1) column renaming for OpenTelemetry data.
A simple component that prints information about the data passing through, with configurable level of detail.
A simple component that returns a constant error message. All requests fail.
A simple component that returns success. All requests succeed.
A simple component to produce synthetic data from semantic convention registries.
An batching processor that works directly with OTAP records. This is
based on lower-level support in the otal_arrow_rust
crate.
The OTAP streaming gRPC exporter. This corresponds with the
otelarrowexporter Collector-Contrib exporter component
(i.e., this project's Phase 1 deliverable), based on Arrow IPC
streams over gRPC streams.
The OTAP streaming gRPC receiver. This corresponds with the
otelarrowreceiver Collector-Contrib receiver component,
this project's Phase 1 deliverable based on Arrow IPC streams over
gRPC streams.
The OTLP unary gRPC exporter. This corresponds with the otlp
Collector exporter component, exports standard OTLP bytes.
The OTLP unary gRPC exporter. This corresponds with the otlp
Collector receiver component, receives standard OTLP bytes using
a Tonic server.
The parquet exporter records the OTel-Arrow representation using Parquet. While there is a direct translation between the OTel-Arrow representation and Parquet, it requires changing several data types to be compatible with Parquet. This component uses 32-bit identifiers, as opposed to 16-bit identifiers used in OTel-Arrow batches, making large batches of telemetry available for external engines to process.
A simple component that collects and prints statistics about the number of requests and items it sees, used for monitoring our benchmarks.
The retry processor supports a configurable number of retries and exponential back-off.
Supports routing OTAP data by signal type, enabling signal-specific route destinations.
The Syslog/CEF receiver is considered a core component used to establish the performance of the OTAP Dataflow system.
The otap_df_controller crate is located in crates/controller is
the main entry point to construct an OTAP Dataflow pipeline instance. The
controller type, otap_df_controller::Controller<PData>, manages building
and running one or more pipelines.
This component is responsible for making the assignment between OTAP
dataflow pipeline and individually-numbered CPU instances. The
Controller::run_forever() method is called to execute the pipeline.
Like the engine, the pipeline datatype PData is opaque to this crate.
Here, the configuration model for the OTAP Dataflow engine defines the structs and conventions used to configure as well as observe the pipeline, the engine, and the pipeline components.
A number of example configurations are listed in
./configs. These are deserialized into the
otap_df_config::engine::EngineConfig structs, defined in this crate.
Defines the low-level queues used in the OTAP dataflow pipeline,
otap_df_channel::mpsc and otap_df_channel::mpmc.
Defines a standard SendError<T> used to return failures throughout
the codebase to enable recovering from try_send().
Defines an administrative portal for operators, an HTTP service capable of displaying the current pipeline state, pipeline configuration, debugging logs, and Prometheus metrics. This supports primitive controls such as the ability to shut down the pipeline.
Low-level library supporting the state transition diagram, enabling
observability for the state of the Controller.
The OTAP Dataflow system is built using a bespoke telemetry system, as we needed to ensure NUMA-awareness from the start. Moreover, this project is taking up a charter to investigate an OTel-Arrow first approach to telemetry, hence we are working with the experimental telemetry SDK here.
The views sub-module contains zero-copy machinery for:
- interpreting OTLP bytes using views to build OTAP records
- interpreting OTAP records using views to encode OTLP bytes
Requirements:
- Rust >= 1.86.0
- Cargo
Clone & Build:
git clone https://github.com/open-telemetry/otel-arrow
cd otel-arrow/rust/otap-dataflow
cargo build --workspaceRun Tests:
cargo test --workspaceRun Examples:
cargo run --example <example_name>With Docker:
docker build --build-context otel-arrow=../../ -f Dockerfile -t df_engine .- Contribution Guidelines
- Code of Conduct (TBD)
Before submitting a PR, please run the following commands:
# Prepare and check the entire project before submitting a PR or a commit
cargo xtask checkLicensed under the Apache License, Version 2.0.
CNCF Slack channel: #otel-arrow
See our detailed Roadmap for upcoming features and improvements.