Skip to content

Commit 64e4512

Browse files
pkoutsovasilispkoutsovasilis
committed
feat: implement system integration as chart built-in
1 parent 932a09a commit 64e4512

File tree

8 files changed

+253
-19
lines changed

8 files changed

+253
-19
lines changed

‎deploy/helm/elastic-agent/README.md‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -123,12 +123,21 @@ The chart built-in [kubernetes integration](https://docs.elastic.co/integrations
123123
| kubernetes.system.metrics.enabled | bool | `true` | enable system metric stream (kubelet) [ref](https://www.elastic.co/docs/current/integrations/kubernetes/kubelet#system) |
124124
| kubernetes.system.metrics.vars | object | `{}` | system metric stream vars |
125125

126-
### 3 - User Extra Integrations
126+
### 4 - System integration
127+
| Key | Type | Default | Description |
128+
|-----|------|---------|-------------|
129+
| system.enabled | bool | `false` | enable System integration. |
130+
| system.output | string | `"default"` | name of the output used in kubernetes integration. Note that this output needs to be defined in [outputs](#1-outputs) |
131+
| system.namespace | string | `"default"` | output namespace |
132+
| system.logs.enabled | bool | `true` | enable logs |
133+
| system.metrics.enabled | bool | `true` | enable metrics |
134+
135+
### 5 - User Extra Integrations
127136
| Key | Type | Default | Description |
128137
|-----|------|---------|-------------|
129138
| extraIntegrations | object | `{}` | extra [user-defined integrations](https://www.elastic.co/guide/en/fleet/current/elastic-agent-input-configuration.html) to be added to the Elastic Agent An example can be found [here](./examples/nginx-custom-integration/README.md) |
130139

131-
### 3 - Elastic-Agent Configuration
140+
### 6 - Elastic-Agent Configuration
132141
| Key | Type | Default | Description |
133142
|-----|------|---------|-------------|
134143
| agent.version | string | `"9.0.0"` | elastic-agent version |
@@ -137,7 +146,7 @@ The chart built-in [kubernetes integration](https://docs.elastic.co/integrations
137146
| agent.unprivileged | bool | `false` | enable unprivileged mode |
138147
| agent.presets | map[string]{} | `{ "perNode" : {...}, "clusterWide": {...}, "ksmSharded": {...} }` | Map of deployment presets for the Elastic Agent. The key of the map is the name of the preset. See more for the presets required by the built-in Kubernetes integration [here](./values.yaml) |
139148

140-
### 3.1 - Elastic-Agent Managed Configuration
149+
### 6.1 - Elastic-Agent Managed Configuration
141150
| Key | Type | Default | Description |
142151
|-----|------|---------|-------------|
143152
| agent.fleet.enabled | bool | `false` | enable elastic-agent managed |

‎deploy/helm/elastic-agent/templates/NOTES.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ Installed integrations:
2222
{{- if eq $.Values.kubernetes.enabled true }}
2323
- kubernetes [built-in chart integration]
2424
{{- end }}
25+
{{- if eq $.Values.system.enabled true }}
26+
- system [built-in chart integration]
27+
{{- end }}
2528
{{- range $customInputName, $customInputVal := $.Values.extraIntegrations }}
2629
- {{$customInputName}} [user-defined integration]
2730
{{- end }}

‎deploy/helm/elastic-agent/templates/agent/_helpers.tpl‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ Initialise input templates if we are not deploying as managed
6565
{{- if eq $.Values.agent.fleet.enabled false -}}
6666
{{/* standalone agent so initialise inputs */}}
6767
{{- include "elasticagent.kubernetes.init" $ -}}
68+
{{- include "elasticagent.system.init" $ -}}
6869
{{- range $customInputName, $customInputVal := $.Values.extraIntegrations -}}
6970
{{- $customInputPresetName := ($customInputVal).preset -}}
7071
{{- $presetVal := get $.Values.agent.presets $customInputPresetName -}}

‎deploy/helm/elastic-agent/templates/integrations/_system/_system.tpl‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{{- define "elasticagent.system.init" -}}
2+
{{- if eq $.Values.system.enabled true -}}
3+
{{- include "elasticagent.system.config.logs.init" $ -}}
4+
{{- include "elasticagent.system.config.metrics.init" $ -}}
5+
{{- end -}}
6+
{{- end -}}

‎deploy/helm/elastic-agent/templates/integrations/_system/_system_logs.tpl‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{{- define "elasticagent.system.config.logs.init" -}}
2+
{{- if $.Values.system.logs.enabled}}
3+
{{- $preset := $.Values.agent.presets.perNode -}}
4+
{{- $inputVal := (include "elasticagent.system.config.logs.input" $ | fromYamlArray) -}}
5+
{{- include "elasticagent.preset.mutate.inputs" (list $ $preset $inputVal) -}}
6+
{{- include "elasticagent.preset.applyOnce" (list $ $preset "elasticagent.kubernetes.pernode.preset") -}}
7+
{{- end -}}
8+
{{- end -}}
9+
10+
{{- define "elasticagent.system.config.logs.input" -}}
11+
- id: system-logs
12+
type: logfile
13+
use_output: {{ $.Values.system.output }}
14+
data_stream:
15+
namespace: {{ $.Values.system.namespace }}
16+
streams:
17+
- data_stream:
18+
dataset: system.auth
19+
type: logs
20+
paths:
21+
- /var/log/auth.log*
22+
- /var/log/secure*
23+
exclude_files:
24+
- .gz$
25+
multiline:
26+
pattern: ^\s
27+
match: after
28+
processors:
29+
- add_locale: null
30+
ignore_older: 72h
31+
- data_stream:
32+
dataset: system.syslog
33+
type: logs
34+
paths:
35+
- /var/log/messages*
36+
- /var/log/syslog*
37+
exclude_files:
38+
- .gz$
39+
multiline:
40+
pattern: ^\s
41+
match: after
42+
processors:
43+
- add_locale: null
44+
ignore_older: 72h
45+
{{- end -}}

‎deploy/helm/elastic-agent/templates/integrations/_system/_system_metrics.tpl‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
{{- define "elasticagent.system.config.metrics.init" -}}
2+
{{- if $.Values.system.metrics.enabled}}
3+
{{- $preset := $.Values.agent.presets.perNode -}}
4+
{{- $inputVal := (include "elasticagent.system.config.metrics.input" $ | fromYamlArray) -}}
5+
{{- include "elasticagent.preset.mutate.inputs" (list $ $preset $inputVal) -}}
6+
{{- include "elasticagent.preset.applyOnce" (list $ $preset "elasticagent.kubernetes.pernode.preset") -}}
7+
{{- end -}}
8+
{{- end -}}
9+
10+
{{- define "elasticagent.system.config.metrics.input" -}}
11+
- id: system-metrics
12+
type: system/metrics
13+
use_output: {{ $.Values.system.output }}
14+
data_stream:
15+
namespace: {{ $.Values.system.namespace }}
16+
streams:
17+
- data_stream:
18+
dataset: system.cpu
19+
type: metrics
20+
period: 10s
21+
cpu.metrics:
22+
- percentages
23+
- normalized_percentages
24+
metricsets:
25+
- cpu
26+
- data_stream:
27+
dataset: system.diskio
28+
type: metrics
29+
period: 10s
30+
diskio.include_devices: null
31+
metricsets:
32+
- diskio
33+
- data_stream:
34+
dataset: system.filesystem
35+
type: metrics
36+
period: 1m
37+
metricsets:
38+
- filesystem
39+
processors:
40+
- drop_event.when.regexp:
41+
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
42+
- data_stream:
43+
dataset: system.fsstat
44+
type: metrics
45+
period: 1m
46+
metricsets:
47+
- fsstat
48+
processors:
49+
- drop_event.when.regexp:
50+
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
51+
- data_stream:
52+
dataset: system.load
53+
type: metrics
54+
condition: '${host.platform} != ''windows'''
55+
period: 10s
56+
metricsets:
57+
- load
58+
- data_stream:
59+
dataset: system.memory
60+
type: metrics
61+
period: 10s
62+
metricsets:
63+
- memory
64+
- data_stream:
65+
dataset: system.network
66+
type: metrics
67+
period: 10s
68+
network.interfaces: null
69+
metricsets:
70+
- network
71+
- data_stream:
72+
dataset: system.process
73+
type: metrics
74+
period: 10s
75+
processes:
76+
- .*
77+
process.include_top_n.by_cpu: 5
78+
process.include_top_n.by_memory: 5
79+
process.cmdline.cache.enabled: true
80+
process.cgroups.enabled: false
81+
process.include_cpu_ticks: false
82+
metricsets:
83+
- process
84+
process.include_cpu_ticks: false
85+
- data_stream:
86+
dataset: system.process_summary
87+
type: metrics
88+
period: 10s
89+
metricsets:
90+
- process_summary
91+
- data_stream:
92+
dataset: system.socket_summary
93+
type: metrics
94+
period: 10s
95+
metricsets:
96+
- socket_summary
97+
- data_stream:
98+
type: metrics
99+
dataset: system.uptime
100+
metricsets:
101+
- uptime
102+
period: 10s
103+
{{- end -}}

‎deploy/helm/elastic-agent/values.schema.json‎

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,53 @@
1818
}
1919
]
2020
},
21+
"system": {
22+
"type": "object",
23+
"description": "Configuration for system integration.",
24+
"properties": {
25+
"enabled": {
26+
"type": "boolean",
27+
"description": "Enable system integration."
28+
},
29+
"output": {
30+
"type": "string",
31+
"description": "Name of the output used in system integration. Must be defined in outputs."
32+
},
33+
"namespace": {
34+
"type": "string",
35+
"description": "output namespace.",
36+
"default": "default"
37+
},
38+
"logs": {
39+
"type": "object",
40+
"properties": {
41+
"enabled": {
42+
"type": "boolean",
43+
"description": "Enable system logs stream."
44+
}
45+
},
46+
"required": [
47+
"enabled"
48+
]
49+
},
50+
"metrics": {
51+
"type": "object",
52+
"properties": {
53+
"enabled": {
54+
"type": "boolean",
55+
"description": "Enable system metrics stream."
56+
}
57+
},
58+
"required": [
59+
"enabled"
60+
]
61+
}
62+
},
63+
"required": [
64+
"enabled",
65+
"output"
66+
]
67+
},
2168
"kubernetes": {
2269
"type": "object",
2370
"description": "Configuration for Kubernetes integration.",
@@ -528,7 +575,8 @@
528575
"password"
529576
]
530577
}
531-
},{
578+
},
579+
{
532580
"if": {
533581
"properties": {
534582
"type": {

‎deploy/helm/elastic-agent/values.yaml‎

Lines changed: 34 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ kubernetes:
6464
# results in overriding and *disabling all* the respective state streams
6565
# @section -- 2 - Kubernetes integration
6666
enabled: true
67-
# -- deploy kube-state-metrics service as a sidecar container to the elastic agent of `ksmShared` preset.
67+
# -- deploy kube-state-metrics service as a sidecar container to the elastic agent of `ksmSharded` preset.
6868
# If set to `false`, kube-state-metrics will *not* get deployed and `clusterWide` agent preset will be used for
6969
# collecting kube-state-metrics.
7070
# @section -- 2 - Kubernetes integration
@@ -287,55 +287,74 @@ kubernetes:
287287
# -- system metric stream vars
288288
# @section -- 2 - Kubernetes integration
289289
vars: {}
290+
system:
291+
# -- enable System integration.
292+
# @section -- 4 - System integration
293+
enabled: false
294+
# -- name of the output used in kubernetes integration. Note that this output needs to be defined in
295+
# [outputs](#1-outputs)
296+
# @section -- 4 - System integration
297+
output: default
298+
# -- output namespace
299+
# @section -- 4 - System integration
300+
namespace: default
301+
logs:
302+
# -- enable logs
303+
# @section -- 4 - System integration
304+
enabled: true
305+
metrics:
306+
# -- enable metrics
307+
# @section -- 4 - System integration
308+
enabled: true
290309
# -- extra [user-defined integrations](https://www.elastic.co/guide/en/fleet/current/elastic-agent-input-configuration.html) to be added to the Elastic Agent
291310
# An example can be found [here](./examples/nginx-custom-integration/README.md)
292-
# @section -- 3 - User Extra Integrations
311+
# @section -- 5 - User Extra Integrations
293312
extraIntegrations: {}
294313
agent:
295314
# -- elastic-agent version
296-
# @section -- 3 - Elastic-Agent Configuration
315+
# @section -- 6 - Elastic-Agent Configuration
297316
version: 9.0.0
298317
# -- image configuration
299-
# @section -- 3 - Elastic-Agent Configuration
318+
# @section -- 6 - Elastic-Agent Configuration
300319
image:
301320
repository: docker.elastic.co/beats/elastic-agent
302321
pullPolicy: IfNotPresent
303322
tag: "9.0.0-SNAPSHOT"
304323
# -- generate kubernetes manifests or [ECK](https://github.com/elastic/cloud-on-k8s) CRDs
305-
# @section -- 3 - Elastic-Agent Configuration
324+
# @section -- 6 - Elastic-Agent Configuration
306325
engine: k8s # k8s or eck
307326
# -- enable unprivileged mode
308-
# @section -- 3 - Elastic-Agent Configuration
327+
# @section -- 6 - Elastic-Agent Configuration
309328
unprivileged: false
310329
fleet:
311330
# -- enable elastic-agent managed
312-
# @section -- 3.1 - Elastic-Agent Managed Configuration
331+
# @section -- 6.1 - Elastic-Agent Managed Configuration
313332
enabled: false
314333
# -- Fleet server URL
315-
# @section -- 3.1 - Elastic-Agent Managed Configuration
334+
# @section -- 6.1 - Elastic-Agent Managed Configuration
316335
url: ""
317336
# -- Fleet enrollment token
318-
# @section -- 3.1 - Elastic-Agent Managed Configuration
337+
# @section -- 6.1 - Elastic-Agent Managed Configuration
319338
token: ""
320339
# -- Fleet insecure url
321-
# @section -- 3.1 - Elastic-Agent Managed Configuration
340+
# @section -- 6.1 - Elastic-Agent Managed Configuration
322341
insecure: false
323342
# -- Kibana host to fallback if enrollment token is not supplied
324-
# @section -- 3.1 - Elastic-Agent Managed Configuration
343+
# @section -- 6.1 - Elastic-Agent Managed Configuration
325344
kibanaHost: ""
326345
# -- Kibana username to fallback if enrollment token is not supplied
327-
# @section -- 3.1 - Elastic-Agent Managed Configuration
346+
# @section -- 6.1 - Elastic-Agent Managed Configuration
328347
kibanaUser: ""
329348
# -- Kibana password to fallback if enrollment token is not supplied
330-
# @section -- 3.1 - Elastic-Agent Managed Configuration
349+
# @section -- 6.1 - Elastic-Agent Managed Configuration
331350
kibanaPassword: ""
332351
# -- Agent preset to deploy
333-
# @section -- 3.1 - Elastic-Agent Managed Configuration
352+
# @section -- 6.1 - Elastic-Agent Managed Configuration
334353
preset: perNode
335354
# -- Map of deployment presets for the Elastic Agent. The key of the map is the name of the preset. See more for the presets
336355
# required by the built-in Kubernetes integration [here](./values.yaml)
337356
# @notationType -- map[string]{}
338-
# @section -- 3 - Elastic-Agent Configuration
357+
# @section -- 6 - Elastic-Agent Configuration
339358
# @default -- `{ "perNode" : {...}, "clusterWide": {...}, "ksmSharded": {...} }`
340359
presets:
341360
# examplePreset:

0 commit comments

Comments
 (0)