Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 15 additions & 22 deletions docs/reference/architecture/hosts_vms.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,58 +13,51 @@ products:

# Hosts and VMs environments

On host or virtual machine environments, deploy local, per-host OpenTelemetry Collector instances, here referred to as OTel Collector in Agent Mode.
On host or virtual machine environments, deploy local, per-host OpenTelemetry Collector instances at the [edge](index.md#understanding-edge-deployment), close to your data sources.

![VM-Edge](../images/arch-vm-edge.png)

These collectors have two main purposes:
These edge collectors have two main purposes:

1. The collection of local logs and infrastructure metrics. Refer to [this sample config file](https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/managed_otlp/platformlogs_hostmetrics.yml) for recommended Collector receiver configurations for hostmetrics and logs.
2. Enriching application telemetry from OTel SDKs that run within the instrumented applications on corresponding hosts with resource information.

## Deployment scenarios

See the recommended architectures per Elastic deployment scenarios:
Refer to the recommended architectures per Elastic deployment scenarios.

:::{note}
Elastic's Observability solution is technically compatible with edge setups that are fully based on contrib OTel components as long as the ingestion path follows the recommendations outlined in the following sections.
:::

### Elastic Cloud Serverless
### {{serverless-full}}

{{serverless-full}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.

![VM-Serverless](../images/arch-vm-serverless.png)

Users can send their OTel data from the edge setup in OTel-native format through OTLP without any additional requirements for self-managed preprocessing of data.

### Elastic Cloud Hosted
### {{ech}}
```{applies_to}
ess:
stack: preview 9.2
```

As of Elastic Stack version 9.0 on {{ech}} (ECH), you need to run a self-hosted EDOT Collector in Gateway Mode to ingest OTel data from the edge setup in OTel-native format into the Elastic-hosted {{es}}.
{{ech}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data. Users can send their OTel data from the edge setup in OTel-native format through OTLP without any additional requirements for self-managed preprocessing of data.

![VM-ECH](../images/arch-vm-ech.png)

The EDOT Collector in Gateway mode enriches and pre-aggregates the data for a seamless experience in the Elastic Observability solution before ingesting it directly into {{es}}.

If required, users can build their custom, EDOT-like Collector
[following these instructions](elastic-agent://reference/edot-collector/custom-collector.md).

:::{note}
The EDOT Gateway Collector does not send data through Elastic's Integration / APM Server on ECH to ingest data into {{es}}.
:::
Alternatively, you can run a self-hosted EDOT Collector in gateway mode to ingest OTel data from the edge setup. The EDOT Collector in gateway mode enriches and pre-aggregates the data before ingesting it directly into {{es}}. If required, you can build your custom, EDOT-like Collector [following these instructions](elastic-agent://reference/edot-collector/custom-collector.md).

:::{important}
If self-managing an EDOT Gateway is not a valid option for you, refer to [Elastic's classic ingestion path for OTel data on ECH](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md).
:::
![VM-ECH](../images/arch-vm-ech.png)

### Self-managed

In a self-managed deployment scenario, you need to host an EDOT Collector in Gateway mode that pre-processes and ingests OTel data from the edge setup into the self-managed Elastic Stack.
In a self-managed deployment scenario, you need to host an EDOT Collector in gateway mode that preprocesses and ingests OTel data from the edge setup into the self-managed {{stack}}.

![VM-self-managed](../images/arch-vm-self-managed.png)

:::{note}
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Collector in gateway mode there is no need for {{product.apm-server}} anymore.

Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
Refer to [EDOT data streams compared to classic {{product.apm}}](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
:::
39 changes: 39 additions & 0 deletions docs/reference/architecture/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,42 @@ The following sections outline the recommended architectures for Elastic Distrib
- [Hosts and VMs](hosts_vms.md)
- [Kubernetes](k8s.md)

## Understanding edge deployment

In the context of OpenTelemetry architecture, 'edge' refers to collectors deployed close to your data sources: running on individual hosts, virtual machines, or as daemon sets on Kubernetes nodes. These edge collectors perform two primary functions:

1. They gather logs, infrastructure metrics, application traces, profiles, and other telemetry from the local environment.
2. They enrich application telemetry from OpenTelemetry SDKs running on the same host or node with resource metadata, such as host information and Kubernetes attributes.

### Collector flexibility at the edge

You can use any OpenTelemetry Collector distribution at the edge, including:

- The contrib OpenTelemetry Collector.
- Custom-built Collector distributions.
- The {{edot}} Collector.

The only requirement is that edge collectors send data using the OpenTelemetry Protocol (OTLP) to either a gateway Collector or directly to {{product.observability}}.

While any OTLP-compatible collector works at the edge, using EDOT provides a more streamlined experience with:

- Preconfigured components optimized for {{product.observability}}.
- Curated receivers and processors for common use cases.
- Simplified setup and configuration.

### When gateway mode is required

The need for an EDOT Collector in gateway mode depends on your Elastic deployment type:

**{{serverless-full}} and {{ech}}**: Edge collectors can send OTLP data directly to the [Managed OTLP Endpoint](/reference/motlp.md) without requiring a Gateway Collector. The Managed OTLP Endpoint handles all necessary preprocessing and data transformation.

**Self-managed, ECE, and ECK deployments**: An EDOT Collector in Gateway mode is required between your edge collectors and {{es}}. The gateway Collector performs essential preprocessing, including:

- Data enrichment and transformation.
- Metrics aggregation.
- Format conversion for optimal storage in {{es}}.

:::{note}
Refer to the [deployment mode documentation](elastic-agent://reference/edot-collector/modes.md) for detailed information about Agent and Gateway modes and their specific requirements.
:::

51 changes: 22 additions & 29 deletions docs/reference/architecture/k8s.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,73 +19,66 @@ The recommended OTel architecture for Kubernetes clusters includes a set of Open

## Daemon form

The Collector in Daemon form is deployed on each Kubernetes node to collect nodes-local logs and host metrics.
The Collector in Daemon form is deployed on each Kubernetes node as an [edge collector](index.md#understanding-edge-deployment) (close to your data sources) to collect node-local logs and host metrics.

The daemon collector also receives telemetry from applications instrumented with OTel SDKs and running on corresponding nodes.

That Collector enriches the application telemetry with resource information such as host and Kubernetes metadata.

All data is then being sent through OTLP to the OTel or EDOT Gateway Collector.
All data is then sent through OTLP to the OTel or EDOT Collector in gateway mode.

## Cluster form

The Collector in Cluster form collects Kubernetes cluster-level metrics and sends them to the OTel or EDOT Gateway Collector using OTLP.

## Gateway form

The OTel or EDOT Collector in Gateway form gathers the OTel data from all other collectors and ingests it into the Elastic backend.
The OTel or EDOT Collector in gateway form gathers the OTel data from all other collectors and ingests it into the Elastic backend.

For self-managed and {{ech}} deployment models the Gateway Collector does some additional pre-processing of data.
For self-managed, ECE, and ECK deployment models the gateway Collector does some additional preprocessing of data.

## Deployment scenarios

See the recommended architectures per Elastic deployment scenarios:
Refer to the recommended architectures per Elastic deployment scenarios.

::::{note}
Elastic's Observability solution is technically compatible with setups that are fully based on contrib OTel components, as long as the ingestion path follows the recommendations outlined in following sub-sections for the different Elastic deployment options.
Elastic's Observability solution is technically compatible with setups that are fully based on contrib OTel components, as long as the ingestion path follows the recommendations outlined in following sections for the different Elastic deployment options.
::::

### Elastic Cloud Serverless
### {{serverless-full}}

{{serverless-full}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.

![K8s-Serverless](../images/arch-k8s-serverless.png)

For a Kubernetes setup, that means the Gateway Collector passes through the OTel data in native format using the OTLP protocol to the managed OTLP endpoint. There is no need for the Gateway Collector to do any Elastic-specific pre-processing.
For a Kubernetes setup, that means the gateway Collector passes through the OTel data in native format using the OTLP protocol to the Managed OTLP Endpoint. There is no need for the gateway Collector to do any Elastic-specific preprocessing.

### Elastic Cloud Hosted
### {{ech}}
```{applies_to}
ess:
stack: preview 9.2
```

With {{ech}} (ECH), OTel data is being directly ingested into the Elastic-hosted {{es}} instance.
{{ech}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.

![K8s-ECH](../images/arch-k8s-ech.png)

The Gateway Collector needs to do some preprocessing, aggregation of metrics and, finally, it uses the {{es}} exporter to ingest data into ECH.

While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the Gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the
[required components and pre-processing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).

If required, users can build their custom, EDOT-like Collector [following these instructions](elastic-agent://reference/edot-collector/custom-collector.md).
For a Kubernetes setup, that means the gateway Collector can pass through the OTel data in native format using the OTLP protocol to the Managed OTLP Endpoint. There is no need for the gateway Collector to do any Elastic-specific preprocessing.

::::{note}
The EDOT Gateway Collector does not send data through Elastic's Integration / APM Server on ECH to ingest data into {{es}}.
::::
Alternatively, you can run a self-hosted EDOT Collector in gateway mode. The gateway Collector needs to do some preprocessing, aggregation of metrics and, finally, it uses the {{es}} exporter to ingest data into ECH. While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the [required components and pre-processing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).

::::{important}
If self-managing an EDOT Gateway is not a valid option for you, refer to [Elastic's classic ingestion path for OTel data on ECH](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md).
::::
![K8s-ECH](../images/arch-k8s-ech.png)

### Self-managed

With a self-managed scenario the Gateway Collector ingests data directly into the self-managed {{es}} instance.
With a self-managed scenario the gateway Collector ingests data directly into the self-managed {{es}} instance.

![K8s-self-managed](../images/arch-k8s-self-managed.png)

The Gateway Collector does some preprocessing and aggregation of OTel data before ingesting it into {{es}}.
The gateway Collector does some preprocessing and aggregation of OTel data before ingesting it into {{es}}.

While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the Gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the [required components and pre-processing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).
While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the [required components and preprocessing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).

::::{note}
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT gateway Collector there is no need for {{product.apm-server}} anymore.

Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
Refer to [EDOT data streams compared to classic {{product.apm}}](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
::::
Loading