You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/architecture/hosts_vms.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,11 +13,11 @@ products:
13
13
14
14
# Hosts and VMs environments
15
15
16
-
On host or virtual machine environments, deploy local, per-host OpenTelemetry Collector instances at the[edge](index.md#understanding-edge-deployment), close to your data sources.
16
+
On host or virtual machine environments, deploy local, per-host OpenTelemetry Collector instances at these[edge](index.md#understanding-edge-deployment) host or VMs where your applications are running.
17
17
18
18

19
19
20
-
These edge collectors have two main purposes:
20
+
Collectors deployed on these edge environments have two main purposes:
21
21
22
22
1. The collection of local logs and infrastructure metrics. Refer to [this sample config file](https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/managed_otlp/platformlogs_hostmetrics.yml) for recommended Collector receiver configurations for hostmetrics and logs.
23
23
2. Enriching application telemetry from OTel SDKs that run within the instrumented applications on corresponding hosts with resource information.
@@ -36,23 +36,25 @@ Elastic's Observability solution is technically compatible with edge setups that
Users can send their OTel data from the edge setup in OTel-native format through OTLP without any additional requirements for self-managed preprocessing of data.
39
+
Users can send data direct from the Collectors or SDKs deployed on the edge environment via OTLP without any additional requirements for managing an ingestion layer.
40
40
41
41
### {{ech}}
42
42
```{applies_to}
43
43
ess:
44
44
stack: preview 9.2
45
45
```
46
46
47
-
{{ech}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data. Users can send their OTel data from the edge setup in OTel-native format through OTLP without any additional requirements for self-managed preprocessing of data.
47
+
{{ech}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data. Users can send data direct from the Collectors or SDKs deployed on the edge environment via OTLP without any additional requirements for managing an ingestion layer.
48
48
49
49
Alternatively, you can run a self-hosted EDOT Collector in gateway mode to ingest OTel data from the edge setup. The EDOT Collector in gateway mode enriches and pre-aggregates the data before ingesting it directly into {{es}}. If required, you can build your custom, EDOT-like Collector [following these instructions](elastic-agent://reference/edot-collector/custom-collector.md).
50
50
51
51

52
52
53
53
### Self-managed
54
54
55
-
In a self-managed deployment scenario, you need to host an EDOT Collector in gateway mode that preprocesses and ingests OTel data from the edge setup into the self-managed {{stack}}.
55
+
In a self-managed Elastic deployment, we recommend running an EDOT Collector in gateway mode as a unified ingestion layer for OTLP telemetry from OpenTelemetry collectors or SDKs running at the edge. This gateway can receive all signals (logs, metrics and traces), apply processing as needed, and cover the same use cases that previously required components like APM Server or Logstash.
56
+
57
+
Depending on your scalability and durability needs, this can be a single collector that scales horizontally, or a chain of collectors where each tier handles a specific concern. For high availability and stronger durability guarantees, you can insert Kafka between tiers so that ingestion is buffered and resilient to downstream outages.
Copy file name to clipboardExpand all lines: docs/reference/architecture/index.md
+8-6Lines changed: 8 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -20,20 +20,22 @@ The following sections outline the recommended architectures for Elastic Distrib
20
20
21
21
## Understanding edge deployment
22
22
23
-
In the context of OpenTelemetry architecture, 'edge' refers to collectors deployed close to your data sources: running on individual hosts, virtual machines, or as daemon sets on Kubernetes nodes. These edge collectors perform two primary functions:
23
+
In the context of OpenTelemetry architecture, 'edge' refers to the environment where applications are running: individual hosts, virtual machines, or as daemon sets on Kubernetes nodes. Collectors deployed in these environments perform two primary functions:
24
24
25
25
1. They gather logs, infrastructure metrics, application traces, profiles, and other telemetry from the local environment.
26
26
2. They enrich application telemetry from OpenTelemetry SDKs running on the same host or node with resource metadata, such as host information and Kubernetes attributes.
27
27
28
-
### Collector flexibility at the edge
28
+
### Vendor neutral collector deployments on edge environments
29
29
30
30
You can use any OpenTelemetry Collector distribution at the edge, including:
31
31
32
32
- The contrib OpenTelemetry Collector.
33
33
- Custom-built Collector distributions.
34
34
- The {{edot}} Collector.
35
+
- Any shipper capable of sending valid OTLP data.
35
36
36
-
The only requirement is that edge collectors send data using the OpenTelemetry Protocol (OTLP) to either a gateway Collector or directly to {{product.observability}}.
37
+
38
+
The only requirement is that collectors deployed on edge environments send data using valid OpenTelemetry Protocol (OTLP) to Elastic Cloud (via managed OTLP endpoint) or to the OTLP receiver of a gateway EDOT collector for self-managed deployments.
37
39
38
40
While any OTLP-compatible collector works at the edge, using EDOT provides a more streamlined experience with:
39
41
@@ -45,12 +47,12 @@ While any OTLP-compatible collector works at the edge, using EDOT provides a mor
45
47
46
48
The need for an EDOT Collector in gateway mode depends on your Elastic deployment type:
47
49
48
-
**{{serverless-full}} and {{ech}}**: Edge collectors can send OTLP data directly to the [Managed OTLP Endpoint](/reference/motlp.md) without requiring a Gateway Collector. The Managed OTLP Endpoint handles all necessary preprocessing and data transformation.
50
+
**{{serverless-full}} and {{ech}}**: Edge collectors can send OTLP data directly to the [Managed OTLP Endpoint](/reference/motlp.md) without requiring a Gateway Collector. The Managed OTLP Endpoint provides a managed durable and resilient ingestion layer.
49
51
50
-
**Self-managed, ECE, and ECK deployments**: An EDOT Collector in Gateway mode is required between your edge collectors and {{es}}. The gateway Collector performs essential preprocessing, including:
52
+
**Self-managed, ECE, and ECK deployments**: An EDOT Collector in Gateway mode is required as part of the stack, this collector will expose the OTLP endpoint that collectors deployed at the edge can send data to. The gateway Collector handles essential preprocessing, including:
51
53
52
54
- Data enrichment and transformation.
53
-
- Metrics aggregation.
55
+
- Metrics aggregation for traces and logs which improves APM UI performance with lower latency.
54
56
- Format conversion for optimal storage in {{es}}.
Copy file name to clipboardExpand all lines: docs/reference/architecture/k8s.md
+9-7Lines changed: 9 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,21 +19,23 @@ The recommended OTel architecture for Kubernetes clusters includes a set of Open
19
19
20
20
## Daemon form
21
21
22
-
The Collector in Daemon form is deployed on each Kubernetes node as an [edge collector](index.md#understanding-edge-deployment) (close to your data sources) to collect node-local logs and host metrics.
22
+
The Collector as a DaemonSet resource kind, deploys an instance on each Kubernetes node as an [edge collector](index.md#understanding-edge-deployment) (close to your data sources) to collect node-local logs and host metrics.
23
23
24
-
The daemon collector also receives telemetry from applications instrumented with OTel SDKs and running on corresponding nodes.
24
+
The DaemonSet collectors also can funnel and enrich telemetry from applications instrumented with OTel SDKs and running on corresponding nodes.
25
25
26
26
That Collector enriches the application telemetry with resource information such as host and Kubernetes metadata.
27
27
28
28
All data is then sent through OTLP to the OTel or EDOT Collector in gateway mode.
29
29
30
-
## Cluster form
30
+
## Cluster Collector
31
31
32
-
The Collector in Cluster form collects Kubernetes cluster-level metrics and sends them to the OTel or EDOT Gateway Collector using OTLP.
32
+
The Cluster collector is a Deployment resource kind that collects Kubernetes cluster-level metrics and sends them to the OTel or EDOT Gateway Collector using OTLP.
33
+
34
+
This instance of the collector helps with collecting cluster level metrics which otherwise would be duplicated by the DaemonSet instances.
33
35
34
36
## Gateway form
35
37
36
-
The OTel or EDOT Collector in gateway form gathers the OTel data from all other collectors and ingests it into the Elastic backend.
38
+
The OTel or EDOT Collector gateway deployed on the edge Kubernetes environment is not an ingestion layer to Elastic, it centralizes OTel data from all other collectors to help define a unified data pipeline sending OTLP data into the Elastic backend corresponding OTLP endpoint.
37
39
38
40
For self-managed, ECE, and ECK deployment models the gateway Collector does some additional preprocessing of data.
39
41
@@ -69,11 +71,11 @@ Alternatively, you can run a self-hosted EDOT Collector in gateway mode. The gat
69
71
70
72
### Self-managed
71
73
72
-
With a self-managed scenario the gateway Collector ingests data directly into the self-managed {{es}} instance.
74
+
With a self-managed deployment the gateway Collector ingests data directly into the self-managed {{es}} instance.
The gateway Collector does some preprocessing and aggregation of OTel data before ingesting it into {{es}}.
78
+
The gateway Collector processes APM data and logs to improve latency on solution UIs before is ingested into {{es}}.
77
79
78
80
While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the [required components and preprocessing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).
0 commit comments