From 27101ce6034e4c8d6fa5e6642a36a510ceee19ed Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Sun, 7 Dec 2025 20:44:24 +0100 Subject: [PATCH 1/2] Add documentation for EDOT data streams --- docs/reference/data-streams.md | 101 +++++++++++++++++++++++++++++++++ docs/reference/toc.yml | 1 + 2 files changed, 102 insertions(+) create mode 100644 docs/reference/data-streams.md diff --git a/docs/reference/data-streams.md b/docs/reference/data-streams.md new file mode 100644 index 00000000..ec394e07 --- /dev/null +++ b/docs/reference/data-streams.md @@ -0,0 +1,101 @@ +--- +navigation_title: Data streams +description: Learn how EDOT stores traces, metrics, and logs in Elasticsearch. Understand OTel-native and ECS-compatible storage modes, exporter behavior, and differences between Managed OTLP Endpoint and local EDOT gateways. +applies_to: + stack: + serverless: + observability: +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# EDOT data streams + +{{edot}} writes telemetry data to two types of data stream schemas in {{es}}: + +- OTel-native data streams (default) +- ECS-compatible data streams (for backwards compatibility with {{product.apm}}) + +To learn how to route OpenTelemetry (OTel) signals to custom data streams, see [Data stream routing](docs-content://solutions/observability/apm/opentelemetry/data-stream-routing.md). + +## Exporter behavior: `otel` and `ecs` + +EDOT may use two {{es}} exporters: + +- `otel`: sends data to OTel-native data streams +- `ecs`: sends data to ECS-compatible data streams + +The `ecs` exporter ensures compatibility with legacy {{product.apm}} data formats. + +## Data streams used by a local deployment + +When EDOT runs as a local gateway or Collector, it writes telemetry data to different data streams depending on the storage mode and exporter configuration: + +### OTel-native mode + +This is the default mode. When ingesting standard OTel signals, EDOT writes to: + +| Signal | Data stream | +|--------|-------------| +| Traces | `traces-otel-*` | +| Metrics | `metrics-otel-*` (TSDB-backed using `mode: time_series`) | +| Logs | `logs-otel-*` | +| Aggregated service destination metrics | `metrics-service_destination.[1m\|10m\|60m].otel-*` | + +These streams follow OTel naming conventions and field structures. + +### ECS-compatible mode + +When the `ecs` exporter is enabled, or when ingesting data from {{product.apm}} agents using `elasticapmintake`, EDOT writes to: + +- `traces-apm-*` +- `traces-apm.sampled-*` +- `metrics-apm.internal-*` +- `metrics-apm.*-*` +- `metrics-apm.service_destination.[interval]-*` +- `logs-apm.error-*` +- `logs-apm.app.*-*` + +These match legacy {{product.apm-server}} behavior. + +## Managed OTLP Endpoint + +The {{motlp}} follows the same stream selection logic as local EDOT deployments: + +- OTel-native streams for standard OTel telemetry +- ECS-compatible streams for {{product.apm}}-formatted events + +The main difference is that {{motlp}} is a managed cloud service that handles ingestion and storage, whereas local EDOT deployments require you to manage the Collector infrastructure yourself. Ingest and storage behavior remain consistent between both approaches. + +## Field duplication + +You may notice fields like: + +- `resource.attributes.service.name` +- `service.name` + +These are not true duplicates. Here's why: + +- OTel defines attributes at multiple levels (for example: resource, span). +- EDOT stores resource attributes under `resource.attributes.*`, according to OTel specifications. +- {{kib}} dashboards and {{product.apm}} UI rely on ECS-style top-level fields like `service.name`. + +To bridge these models, EDOT aliases or copies key fields so that: + +- Dashboards continue to function +- Filters and searches behave consistently +- OTel structure remains intact for downstream compatibility + +This duplication is intentional. + +## Storage engines used by EDOT + +| Data type | Storage engine | +|-----------|----------------| +| Metrics | TSDB (`mode: time_series`) | +| Logs | LogsDB | +| Traces | Standard time-series model | + +EDOT uses the same index templates and mappings as the rest of the Observability solution. \ No newline at end of file diff --git a/docs/reference/toc.yml b/docs/reference/toc.yml index a62cd288..cb9ae43b 100644 --- a/docs/reference/toc.yml +++ b/docs/reference/toc.yml @@ -19,6 +19,7 @@ toc: - file: compatibility/nomenclature.md - file: compatibility/data-streams.md - file: motlp.md + - file: data-streams.md - file: central-configuration.md - file: edot-sdks/index.md - file: edot-cloud-forwarder/index.md From 52087034d7a0c62171742548c22c74752659c68a Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Sun, 7 Dec 2025 20:51:43 +0100 Subject: [PATCH 2/2] Enhance documentation for EDOT data streams --- docs/reference/compatibility/data-streams.md | 20 +++++++++++--------- docs/reference/data-streams.md | 2 ++ 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/reference/compatibility/data-streams.md b/docs/reference/compatibility/data-streams.md index 80f8ab94..f035da29 100644 --- a/docs/reference/compatibility/data-streams.md +++ b/docs/reference/compatibility/data-streams.md @@ -18,7 +18,9 @@ The {{edot}} (EDOT) stores telemetry data using a storage model optimized for Op This architecture is designed for scalable observability workloads. It supports dynamic attributes, reduces mapping complexity, and avoids issues like mapping explosions or manual dimension setup. -EDOT uses Elasticsearch’s [Logs data stream (LogsDB)](docs-content://manage-data/data-store/data-streams/logs-data-stream.md) and [Time Series Data Streams (TSDS)](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md) as storage backends. These are purpose-built to handle the scale and variety of observability data and improve the storage efficiency. +EDOT uses {{es}}’s [Logs data stream (LogsDB)](docs-content://manage-data/data-store/data-streams/logs-data-stream.md) and [Time Series Data Streams (TSDS)](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md) as storage backends. These are purpose-built to handle the scale and variety of observability data and improve the storage efficiency. + +This page provides a detailed comparison of EDOT data streams with classic {{product.apm}} and ECS-based integrations. For a practical reference on which data streams EDOT uses, exporter behavior, and storage engines, see [EDOT data streams](../data-streams.md). ## Logs and traces in LogsDB @@ -29,7 +31,7 @@ Log and trace data is stored in [LogsDB](docs-content://manage-data/data-store/d ## Metrics in TSDS -Metric data is stored using Elasticsearch’s [TSDS](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md). Benefits include: +Metric data is stored using {{es}}’s [TSDS](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md). Benefits include: * Efficient storage using columnar compression * Fast aggregations @@ -40,19 +42,19 @@ Metric data is stored using Elasticsearch’s [TSDS](docs-content://manage-data/ This table highlights key differences between classic Elastic APM data streams and EDOT with `mapping_mode: otel`: -| Feature | Classic APM (ECS-based) | EDOT (`mapping_mode: otel`) | +| Feature | Classic {{product.apm}} (ECS-based) | EDOT (`mapping_mode: otel`) | |---|---|---| -| Index mode | General-purpose data streams (logs, traces, metrics)

TSDS is not supported for classic APM. | LogsDB (logs/traces), TSDS (metrics) | +| Index mode | General-purpose data streams (logs, traces, metrics)

TSDS is not supported for classic {{product.apm}}. | LogsDB (logs/traces), TSDS (metrics) | | Mapping style | Nested objects are mapped as structured fields. Some exceptions exist, such as `labels.*` and `numeric_labels.*`, where dots in field names are replaced with underscores.

ECS supports multiple field types (keyword, long, double, date, boolean, and so on) as defined in the schema. | Native OpenTelemetry fields with `passthrough`, preserving types and structure. | -| Attribute handling | Dynamic mapping. Custom attributes are stored under `labels.*` (strings) or `numeric_labels.*` (numbers); dots in field names are replaced with underscores.

See [Document examples - classic APM](#classic-apm) | Dynamic mapping with native types under `attributes.*`, preserving dots in field names.

See [Document examples - EDOT](#edot) | +| Attribute handling | Dynamic mapping. Custom attributes are stored under `labels.*` (strings) or `numeric_labels.*` (numbers); dots in field names are replaced with underscores.

See [Document examples - classic {{product.apm}}](#classic-apm) | Dynamic mapping with native types under `attributes.*`, preserving dots in field names.

See [Document examples - EDOT](#edot) | ### Query compatibility with classic APM data streams -EDOT is designed to make OpenTelemetry data queryable using many of the same field names as classic APM (ECS-based) data streams. This helps preserve compatibility with existing dashboards, saved searches, and queries. +EDOT is designed to make OpenTelemetry data queryable using many of the same field names as classic {{product.apm}} (ECS-based) data streams. This helps preserve compatibility with existing dashboards, saved searches, and queries. Query compatibility is achieved through: -* **`passthrough` fields:** Make nested OpenTelemetry fields available at the top level so they can be queried. For example, while the service name is stored at `resource.attributes.service.name`, you can query it as `service.name` (the same field name as the one used in the classic APM data stream). +* **`passthrough` fields:** Make nested OpenTelemetry fields available at the top level so they can be queried. For example, while the service name is stored at `resource.attributes.service.name`, you can query it as `service.name` (the same field name as the one used in the classic {{product.apm}} data stream). * **Field aliases:** Map fields with different names in ECS and OpenTelemetry semantic conventions to a common query name to make migration easier. #### Limitations @@ -120,7 +122,7 @@ severity_text: INFO ## Comparison with ECS-based integrations -While classic APM and EDOT represent two ingestion paths for application telemetry, Elastic’s integrations (for example Nginx, MySQL, Kubernetes) also produce ECS-based data streams for logs, metrics, and events. These use ECS mappings and integration-specific pipelines optimized for their domain. +While classic {{product.apm}} and EDOT represent two ingestion paths for application telemetry, Elastic’s integrations (for example Nginx, MySQL, Kubernetes) also produce ECS-based data streams for logs, metrics, and events. These use ECS mappings and integration-specific pipelines optimized for their domain. | Stream type | Typical field layout | Custom attributes / dot notation | |--------------|----------------------|----------------------------------| @@ -148,7 +150,7 @@ user: ## Summary of all data stream types -| Feature | Classic APM (ECS-based) | Integration ECS-based streams | EDOT (`mapping_mode: otel`) | +| Feature | Classic {{product.apm}} (ECS-based) | Integration ECS-based streams | EDOT (`mapping_mode: otel`) | |----------|-------------------------|-------------------------------|-----------------------------| | **Index mode** | General-purpose data streams (logs, traces, metrics); TSDS not supported | ECS-style data streams (logs, metrics, events) using integrations | LogsDB for logs/traces, TSDS for metrics | | **Mapping style** | ECS object mappings; nested fields preserved. `labels.*` / `numeric_labels.*` flatten dots. | ECS mappings or integration-altered schemas (flattening, renaming). | OTel-native nested layout with `passthrough`, preserving types and structure. | diff --git a/docs/reference/data-streams.md b/docs/reference/data-streams.md index ec394e07..e8a2e677 100644 --- a/docs/reference/data-streams.md +++ b/docs/reference/data-streams.md @@ -18,6 +18,8 @@ products: - OTel-native data streams (default) - ECS-compatible data streams (for backwards compatibility with {{product.apm}}) +This page provides a practical reference for which data streams EDOT uses, exporter behavior, and storage engines. For a detailed comparison of OTel data streams with classic {{product.apm}} and ECS-based integrations, see [OTel data streams compared to classic {{product.apm}}](./compatibility/data-streams.md). + To learn how to route OpenTelemetry (OTel) signals to custom data streams, see [Data stream routing](docs-content://solutions/observability/apm/opentelemetry/data-stream-routing.md). ## Exporter behavior: `otel` and `ecs`