Skip to content

Commit 2f88655

Browse files
thbkrkrthbkrkr
authored
Logs and metrics delivery via Stack Monitoring documentation (#4677) (#4707)
Add a documentation page to describe how to enable Stack Monitoring on Elasticsearch and Kibana, what are the limitations, how it works, when to use it and how to enable audit logging.
1 parent 24252b0 commit 2f88655

File tree

3 files changed

+163
-3
lines changed

3 files changed

+163
-3
lines changed

‎docs/advanced-topics/advanced-topics.asciidoc

+2
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ endif::[]
1515
- <<{p}-traffic-splitting>>
1616
- <<{p}-network-policies>>
1717
- <<{p}-webhook-namespace-selectors>>
18+
- <<{p}-stack-monitoring>>
1819
--
1920

2021
include::openshift.asciidoc[leveloffset=+1]
@@ -23,3 +24,4 @@ include::service-meshes.asciidoc[leveloffset=+1]
2324
include::traffic-splitting.asciidoc[leveloffset=+1]
2425
include::network-policies.asciidoc[leveloffset=+1]
2526
include::webhook-namespace-selectors.asciidoc[leveloffset=+1]
27+
include::stack-monitoring.asciidoc[leveloffset=+1]

‎docs/advanced-topics/stack-monitoring.asciidoc

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
:page_id: stack-monitoring
2+
ifdef::env-github[]
3+
****
4+
link:https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-{page_id}.html[View this document on the Elastic website]
5+
****
6+
endif::[]
7+
8+
[id="{p}-{page_id}"]
9+
= Stack Monitoring
10+
11+
You can enable link:https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html[Stack Monitoring]
12+
on Elasticsearch and Kibana to collect and ship their metrics and logs to a dedicated monitoring cluster.
13+
14+
To enable stack monitoring, simply reference the monitoring Elasticsearch cluster in the `spec.monitoring` section of their specification.
15+
16+
[source,yaml,subs="attributes,callouts"]
17+
----
18+
apiVersion: elasticsearch.k8s.elastic.co/{eck_crd_version}
19+
kind: Elasticsearch
20+
metadata:
21+
name: monitored-sample
22+
namespace: production
23+
spec:
24+
version: {version}
25+
monitoring:
26+
metrics:
27+
elasticsearchRefs:
28+
- name: monitoring
29+
namespace: observability <1>
30+
logs:
31+
elasticsearchRefs:
32+
- name: monitoring
33+
namespace: observability <1>
34+
nodeSets:
35+
- name: default
36+
count: 1
37+
config:
38+
node.store.allow_mmap: false
39+
---
40+
apiVersion: kibana.k8s.elastic.co/{eck_crd_version}
41+
kind: Kibana
42+
metadata:
43+
name: monitored-sample
44+
namespace: production
45+
spec:
46+
version: {version}
47+
elasticsearchRef:
48+
name: monitored-sample
49+
namespace: production <2>
50+
monitoring:
51+
metrics:
52+
elasticsearchRefs:
53+
- name: monitoring
54+
namespace: observability <1>
55+
logs:
56+
elasticsearchRefs:
57+
- name: monitoring
58+
namespace: observability <1>
59+
count: 1
60+
----
61+
62+
<1> The use of `namespace` is optional if the monitoring Elasticsearch cluster and the monitored Elasticsearch cluster are running in the same namespace.
63+
<2> The use of `namespace` is optional if the Elasticsearch cluster and the Kibana instance are running in the same namespace.
64+
65+
CAUTION: You cannot configure an Elasticsearch cluster to monitor itself, the monitoring cluster has to be a separate cluster.
66+
67+
IMPORTANT: The monitoring cluster must be managed by ECK in the same Kubernetes cluster as the monitored one.
68+
69+
You can send metrics and logs to two different Elasticsearch monitoring clusters.
70+
71+
You can also enable Stack Monitoring on Elasticsearch only or on Kibana only. In the latter case, Kibana will not be available on the Stack Monitoring Kibana page (see link:https://www.elastic.co/guide/en/kibana/current/monitoring-data.html#monitoring-data[View monitoring data in Kibana]).
72+
73+
== When to use it
74+
75+
This feature is a good solution if you need to monitor your Elastic applications in restricted Kubernetes environments where you cannot grant advanced permissions:
76+
77+
- to Metricbeat to allow queriying the k8s API
78+
- to Filebeat to deploy a privileged DaemonSet
79+
80+
However, for maximum efficiency and minimising resource consumption, or advanced use cases that require specific Beats configurations, you can deploy a standalone Metricbeat Deployment and a Filebeat Daemonset. See the <<{p}-beat-configuration-examples,Beats configuration Examples>> for more information.
81+
82+
== How it works
83+
84+
In the background, Metricbeat and Filebeat are deployed as sidecar containers in the same Pod as Elasticsearch and Kibana.
85+
86+
Metricbeat is used to collect monitoring metrics and Filebeat to monitor the Elasticsearch log files and collect log events.
87+
88+
The two Beats are configured to ship data directly to the monitoring cluster(s) using HTTPS and dedicated Elastic users managed by ECK.
89+
90+
== Audit logging
91+
92+
Audit logs are collected and shipped to the monitoring cluster referenced in the `monitoring.logs` section when audit logging is enabled (it is disabled by default).
93+
94+
[source,yaml,subs="attributes,callouts"]
95+
----
96+
apiVersion: elasticsearch.k8s.elastic.co/v1
97+
kind: Elasticsearch
98+
spec:
99+
monitoring:
100+
metrics:
101+
elasticsearchRefs:
102+
- name: monitoring
103+
namespace: observability
104+
logs:
105+
elasticsearchRefs:
106+
- name: monitoring
107+
namespace: observability
108+
nodeSets:
109+
- name: default
110+
config:
111+
# https://www.elastic.co/guide/en/elasticsearch/reference/current/enable-audit-logging.html
112+
xpack.security.audit.enabled: true
113+
---
114+
apiVersion: kibana.k8s.elastic.co/v1
115+
kind: Kibana
116+
spec:
117+
monitoring:
118+
metrics:
119+
elasticsearchRefs:
120+
- name: monitoring
121+
namespace: observability
122+
logs:
123+
elasticsearchRefs:
124+
- name: monitoring
125+
namespace: observability
126+
config:
127+
# https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html
128+
xpack.security.audit.enabled: true
129+
----
130+
131+
== Override the Beats Pod Template
132+
133+
You can customize the Filebeat and Metricbeat containers through the Pod template. Your configuration is merged with the values of the default Pod template that ECK uses.
134+
135+
[source,yaml,subs="attributes,callouts"]
136+
----
137+
apiVersion: elasticsearch.k8s.elastic.co/v1
138+
kind: Elasticsearch
139+
spec:
140+
nodeSets:
141+
- name: default
142+
monitoring:
143+
metrics:
144+
elasticsearchRef:
145+
name: monitoring
146+
namespace: observability
147+
logs:
148+
elasticsearchRef:
149+
name: monitoring
150+
namespace: observability
151+
podTemplate:
152+
spec:
153+
containers:
154+
- name: metricbeat
155+
env:
156+
- foo: bar
157+
- name: filebeat
158+
env:
159+
- foo: bar
160+
----

‎docs/release-notes/highlights-1.7.0.asciidoc

+1-3
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,7 @@ Starting with this release, the `CustomResourceDefinitions` (CRD) and the `Valid
1717
[id="{p}-170-stack-monitoring"]
1818
==== Stack Monitoring
1919

20-
In this release, the Elasticsearch and Kibana resources have been enhanced to let you specify a reference to a monitoring cluster. When specified, sidecar containers are automatically setup by ECK to ship logs and metrics to the referenced Elasticsearch cluster.
21-
22-
*Add a link to documentation*
20+
In this release, the Elasticsearch and Kibana resources have been enhanced to let you specify a reference to a monitoring cluster. When specified, sidecar containers are automatically setup by ECK to ship logs and metrics to the referenced Elasticsearch cluster. Refer to the <<{p}-stack-monitoring,Stack Monitoring documentation>> for more details.
2321

2422
[float]
2523
[id="{p}-170-autoscaling"]

0 commit comments

Comments
 (0)