Skip to content

Commit fa58e3f

Browse files
gizasvigneshshanmugam
gizas
and
vigneshshanmugam
authored
Updating EDOT collector configuration samples for mOTLP input (#10569)
* updating motel config Signed-off-by: Andreas Gkizas <[email protected]> * updating with seperate exporters rest examples Signed-off-by: Andreas Gkizas <[email protected]> * Update changelog/fragments/1760515775-batchingconfig.yaml Co-authored-by: Vignesh Shanmugam <[email protected]> --------- Signed-off-by: Andreas Gkizas <[email protected]> Co-authored-by: Vignesh Shanmugam <[email protected]>
1 parent 4320a02 commit fa58e3f

File tree

9 files changed

+164
-32
lines changed

9 files changed

+164
-32
lines changed

‎changelog/fragments/1760515775-batchingconfig.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Kind can be one of:
2+
# - breaking-change: a change to previously-documented behavior
3+
# - deprecation: functionality that is being removed in a later release
4+
# - bug-fix: fixes a problem in a previous version
5+
# - enhancement: extends functionality but does not break or fix existing behavior
6+
# - feature: new functionality
7+
# - known-issue: problems that we are aware of in a given version
8+
# - security: impacts on the security of a product or a user’s deployment.
9+
# - upgrade: important information for someone upgrading from a prior version
10+
# - other: does not fit into any of the other categories
11+
kind: other
12+
13+
# Change summary; a 80ish characters long description of the change.
14+
summary: Updating EDOT gateway collector batch configuration when ingesting to mOTLP endpoint.
15+
16+
# Long description; in case the summary is not enough to describe the change
17+
# this field accommodate a description without length limits.
18+
# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
19+
#description:
20+
21+
# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
22+
component: elastic-agent
23+
24+
# PR URL; optional; the PR number that added the changeset.
25+
# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
26+
# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
27+
# Please provide it if you are adding a fragment for a different PR.
28+
#pr: https://github.com/owner/repo/1234
29+
30+
# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
31+
# If not present is automatically filled by the tooling with the issue linked to the PR number.
32+
#issue: https://github.com/owner/repo/1234

‎deploy/helm/edot-collector/kube-stack/managed_otlp/logs-values.yaml‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -234,22 +234,27 @@ collectors:
234234
http:
235235
endpoint: ${env:MY_POD_IP}:4318
236236
processors:
237-
batch:
238-
send_batch_size: 1000
239-
timeout: 1s
240-
send_batch_max_size: 1500
241237
exporters:
242238
debug:
243239
otlp/ingest:
244240
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
245241
headers:
246242
Authorization: ApiKey ${env:ELASTIC_API_KEY}
243+
sending_queue:
244+
enabled: true
245+
sizer: bytes
246+
queue_size: 50000000 # 50MB uncompressed
247+
block_on_overflow: true
248+
batch:
249+
flush_timeout: 1s
250+
min_size: 1_000_000 # 1MB uncompressed
251+
max_size: 4_000_000 # 4MB uncompressed
247252
timeout: 15s
248253
service:
249254
pipelines:
250255
logs:
251256
receivers: [otlp]
252-
processors: [batch]
257+
processors: []
253258
exporters: [debug, otlp/ingest]
254259
# For more details on OpenTelemetry's zero-code instrumentation, see:
255260
# https://opentelemetry.io/docs/concepts/instrumentation/zero-code/

‎deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml‎

Lines changed: 30 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -545,35 +545,50 @@ collectors:
545545
http:
546546
endpoint: ${env:MY_POD_IP}:4318
547547
processors:
548-
batch:
549-
send_batch_size: 1000
550-
timeout: 1s
551-
send_batch_max_size: 1500
552-
batch/metrics:
553-
# explicitly set send_batch_max_size to 0, as splitting metrics requests may cause version_conflict_engine_exception in TSDB
554-
send_batch_max_size: 0
555-
timeout: 1s
556548
exporters:
557549
debug:
558-
otlp/ingest:
550+
otlp/ingest_logs:
559551
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
560552
headers:
561553
Authorization: ApiKey ${env:ELASTIC_API_KEY}
554+
sending_queue:
555+
enabled: true
556+
sizer: bytes
557+
queue_size: 50000000 # 50MB uncompressed
558+
block_on_overflow: true
559+
batch:
560+
flush_timeout: 1s
561+
min_size: 1_000_000 # 1MB uncompressed
562+
max_size: 4_000_000 # 4MB uncompressed
563+
timeout: 15s
564+
otlp/ingest_metrics_traces:
565+
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
566+
headers:
567+
Authorization: ApiKey ${env:ELASTIC_API_KEY}
568+
sending_queue:
569+
enabled: true
570+
sizer: bytes
571+
queue_size: 50000000 # 50MB uncompressed
572+
block_on_overflow: true
573+
batch:
574+
flush_timeout: 1s
575+
min_size: 1_000_000 # 1MB uncompressed
576+
max_size: 4_000_000 # 4MB uncompressed
562577
timeout: 15s
563578
service:
564579
pipelines:
565580
metrics:
566581
receivers: [otlp]
567-
processors: [batch/metrics]
568-
exporters: [debug, otlp/ingest]
582+
processors: []
583+
exporters: [debug, otlp/ingest_metrics_traces]
569584
logs:
570585
receivers: [otlp]
571-
processors: [batch]
572-
exporters: [debug, otlp/ingest]
586+
processors: []
587+
exporters: [debug, otlp/ingest_logs]
573588
traces:
574589
receivers: [otlp]
575-
processors: [batch]
576-
exporters: [debug, otlp/ingest]
590+
processors: []
591+
exporters: [debug, otlp/ingest_metrics_traces]
577592
# For more details on OpenTelemetry's zero-code instrumentation, see:
578593
# https://opentelemetry.io/docs/concepts/instrumentation/zero-code/
579594
instrumentation:

‎internal/pkg/otel/samples/darwin/managed_otlp/logs_metrics_traces.yml‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,35 +79,57 @@ processors:
7979
enabled: true
8080

8181
exporters:
82-
otlp/ingest:
82+
otlp/ingest_logs:
8383
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
8484
headers:
8585
Authorization: ApiKey ${env:ELASTIC_API_KEY}
86+
sending_queue:
87+
enabled: true
88+
sizer: bytes
89+
queue_size: 50000000 # 50MB uncompressed
90+
block_on_overflow: true
91+
batch:
92+
flush_interval: 1s
93+
min_size: 1_000_000 # 1MB uncompressed
94+
max_size: 4_000_000 # 4MB uncompressed
95+
otlp/ingest_metrics_traces:
96+
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
97+
headers:
98+
Authorization: ApiKey ${env:ELASTIC_API_KEY}
99+
sending_queue:
100+
enabled: true
101+
sizer: bytes
102+
queue_size: 50000000 # 50MB uncompressed
103+
block_on_overflow: true
104+
batch:
105+
flush_interval: 1s
106+
min_size: 1_000_000 # 1MB uncompressed
107+
max_size: 4_000_000 # 4MB uncompressed
86108

87109
service:
88110
extensions: [file_storage]
89111
pipelines:
90112
traces/fromsdk:
91113
receivers: [otlp/fromsdk]
92114
processors: []
93-
exporters: [otlp/ingest]
115+
exporters: [otlp/ingest_metrics_traces]
94116

95117
metrics/fromsdk:
96118
receivers: [otlp/fromsdk]
97119
processors: []
98-
exporters: [otlp/ingest]
120+
exporters: [otlp/ingest_metrics_traces]
99121

100122
logs/fromsdk:
101123
receivers: [otlp/fromsdk]
102124
processors: []
103-
exporters: [otlp/ingest]
125+
exporters: [otlp/ingest_logs]
104126

105127
metrics/hostmetrics:
106128
receivers: [hostmetrics/system]
107129
processors: [resourcedetection]
108-
exporters: [otlp/ingest]
130+
exporters: [otlp/ingest_metrics_traces]
109131

110132
logs/platformlogs:
111133
receivers: [filelog/platformlogs]
112134
processors: [resourcedetection]
113-
exporters: [otlp/ingest]
135+
exporters: [otlp/ingest_logs]

‎internal/pkg/otel/samples/darwin/managed_otlp/platformlogs.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,15 @@ exporters:
5757
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
5858
headers:
5959
Authorization: ApiKey ${env:ELASTIC_API_KEY}
60+
sending_queue:
61+
enabled: true
62+
sizer: bytes
63+
queue_size: 50000000 # 50MB uncompressed
64+
block_on_overflow: true
65+
batch:
66+
flush_interval: 1s
67+
min_size: 1_000_000 # 1MB uncompressed
68+
max_size: 4_000_000 # 4MB uncompressed
6069

6170
service:
6271
extensions: [file_storage]

‎internal/pkg/otel/samples/darwin/managed_otlp/platformlogs_hostmetrics.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,15 @@ exporters:
7878
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
7979
headers:
8080
Authorization: ApiKey ${env:ELASTIC_API_KEY}
81+
sending_queue:
82+
enabled: true
83+
sizer: bytes
84+
queue_size: 50000000 # 50MB uncompressed
85+
block_on_overflow: true
86+
batch:
87+
flush_interval: 1s
88+
min_size: 1_000_000 # 1MB uncompressed
89+
max_size: 4_000_000 # 4MB uncompressed
8190

8291
service:
8392
extensions: [file_storage]

‎internal/pkg/otel/samples/linux/managed_otlp/logs_metrics_traces.yml‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,35 +86,57 @@ processors:
8686
enabled: true
8787

8888
exporters:
89-
otlp/ingest:
89+
otlp/ingest_logs:
9090
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
9191
headers:
9292
Authorization: ApiKey ${env:ELASTIC_API_KEY}
93+
sending_queue:
94+
enabled: true
95+
sizer: bytes
96+
queue_size: 50000000 # 50MB uncompressed
97+
block_on_overflow: true
98+
batch:
99+
flush_interval: 1s
100+
min_size: 1_000_000 # 1MB uncompressed
101+
max_size: 4_000_000 # 4MB uncompressed
102+
otlp/ingest_metrics_traces:
103+
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
104+
headers:
105+
Authorization: ApiKey ${env:ELASTIC_API_KEY}
106+
sending_queue:
107+
enabled: true
108+
sizer: bytes
109+
queue_size: 50000000 # 50MB uncompressed
110+
block_on_overflow: true
111+
batch:
112+
flush_interval: 1s
113+
min_size: 1_000_000 # 1MB uncompressed
114+
max_size: 4_000_000 # 4MB uncompressed
93115

94116
service:
95117
extensions: [file_storage]
96118
pipelines:
97119
traces/fromsdk:
98120
receivers: [otlp/fromsdk]
99121
processors: []
100-
exporters: [otlp/ingest]
122+
exporters: [otlp/ingest_metrics_traces]
101123

102124
metrics/fromsdk:
103125
receivers: [otlp/fromsdk]
104126
processors: []
105-
exporters: [otlp/ingest]
127+
exporters: [otlp/ingest_metrics_traces]
106128

107129
logs/fromsdk:
108130
receivers: [otlp/fromsdk]
109131
processors: []
110-
exporters: [otlp/ingest]
132+
exporters: [otlp/ingest_logs]
111133

112134
metrics/hostmetrics:
113135
receivers: [hostmetrics/system]
114136
processors: [resourcedetection]
115-
exporters: [otlp/ingest]
137+
exporters: [otlp/ingest_metrics_traces]
116138

117139
logs/platformlogs:
118140
receivers: [filelog/platformlogs]
119141
processors: [resourcedetection]
120-
exporters: [otlp/ingest]
142+
exporters: [otlp/ingest_logs]

‎internal/pkg/otel/samples/linux/managed_otlp/platformlogs.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,15 @@ exporters:
5757
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
5858
headers:
5959
Authorization: ApiKey ${env:ELASTIC_API_KEY}
60+
sending_queue:
61+
enabled: true
62+
sizer: bytes
63+
queue_size: 50000000 # 50MB uncompressed
64+
block_on_overflow: true
65+
batch:
66+
flush_interval: 1s
67+
min_size: 1_000_000 # 1MB uncompressed
68+
max_size: 4_000_000 # 4MB uncompressed
6069

6170
service:
6271
extensions: [file_storage]

‎internal/pkg/otel/samples/linux/managed_otlp/platformlogs_hostmetrics.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,15 @@ exporters:
8585
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
8686
headers:
8787
Authorization: ApiKey ${env:ELASTIC_API_KEY}
88+
sending_queue:
89+
enabled: true
90+
sizer: bytes
91+
queue_size: 50000000 # 50MB uncompressed
92+
block_on_overflow: true
93+
batch:
94+
flush_interval: 1s
95+
min_size: 1_000_000 # 1MB uncompressed
96+
max_size: 4_000_000 # 4MB uncompressed
8897

8998
service:
9099
extensions: [file_storage]

0 commit comments

Comments
 (0)