Skip to content

Commit be1ae00

Browse files
authored
docs(otel): document delivery guarantees for OTel mode (#11560)
When running in OTel mode, delivery guarantees for Beats receivers are only possible with a specific combination of retry settings, the sending queue, and Beats queue options, so document that.
1 parent f9b07f8 commit be1ae00

File tree

1 file changed

+43
-0
lines changed

1 file changed

+43
-0
lines changed

docs/hybrid-agent-beats-receivers.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,6 +193,7 @@ receivers:
193193
paths:
194194
- /var/log/*.log
195195
type: filestream
196+
queue.mem.flush.timeout: 0s
196197
metricbeatreceiver:
197198
metricbeat:
198199
modules:
@@ -202,6 +203,7 @@ receivers:
202203
metricsets:
203204
- cpu
204205
module: system
206+
queue.mem.flush.timeout: 0s
205207
exporters:
206208
elasticsearch/_agent-component/default:
207209
api_key: placeholder
@@ -214,6 +216,24 @@ exporters:
214216
enabled: true
215217
mapping:
216218
mode: bodymap
219+
220+
retry:
221+
enabled: true
222+
initial_interval: 1s
223+
max_interval: 1m0s
224+
max_retries: 3
225+
226+
sending_queue:
227+
enabled: true
228+
wait_for_result: true
229+
block_on_overflow: true
230+
num_consumers: 1
231+
queue_size: 3200
232+
batch:
233+
max_size: 1600
234+
min_size: 0
235+
flush_timeout: 10s
236+
sizer: items
217237
service:
218238
pipelines:
219239
logs:
@@ -223,3 +243,26 @@ service:
223243
- filebeatreceiver
224244
- metricbeatreceiver
225245
```
246+
247+
### Beats receivers delivery guarantees in OTel mode
248+
249+
When Beat receivers are used in OTel mode, event delivery guarantees depend on the configuration of the OpenTelemetry Collector `sending_queue` and retry settings.
250+
Unlike standalone Beats, the EDOT pipeline allows users to customize queue behavior through the Collector configuration.
251+
This flexibility is useful, but it also means that not every option combination is compatible with reliable delivery.
252+
253+
Elastic Agent in OTel mode provides an **at least once** delivery guarantee for Beat receivers **only when using the supported `sending_queue` settings described below**.
254+
These settings mirror Beats pipeline behavior closely enough to preserve durability expectations.
255+
256+
If users provide arbitrary `sending_queue` or Beat queue overrides, delivery semantics become **undefined** and **at least once delivery cannot be guaranteed**.
257+
These combinations are not tested and may result in event loss during backpressure or shutdown.
258+
259+
To achieve the intended delivery guarantee, the exporter that receives events from Beat receivers must define a `sending_queue` with the following characteristics:
260+
261+
- `enabled: true`: The queue must be active.
262+
- `wait_for_result: true`: The pipeline must wait for the exporter response before removing events.
263+
- `block_on_overflow: true`: Prevents event drops when the queue is full.
264+
- The `batch` configuration must include explicit `max_size`, `min_size`, and `flush_timeout` values to ensure events are grouped and flushed in predictable, controlled batches.
265+
266+
Additionally, the retry settings must be enabled on the exporter, using a backoff policy that retries until the operation succeeds. By default, `max_retries` is set to 3, which is how most Beats behave. Standalone Filebeat, however, retries indefinitely. Beats receivers don't support unlimited retries yet, and this is being tracked at https://github.com/elastic/beats/issues/47892.
267+
268+
Beat receivers also require the Beat-internal memory queue to run in synchronous mode for delivery guarantees. This is enabled by setting `queue.mem.flush.timeout: 0s` in each receiver configuration, as shown in the example above.

0 commit comments

Comments
 (0)