Skip to content

Reestructure gathering the observability data from multiple clusters … #89924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Reestructure gathering the observability data from multiple clusters … #89924

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 123 additions & 92 deletions observability/otel/otel-config-multicluster.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,11 @@ For a multicluster configuration, you can create one OpenTelemetry Collector ins

.Procedure

. Mount the following certificates in the OpenTelemetry Collector instance, skipping already mounted certificates.
.Generate Certificates

.. An Issuer to generate the certificates by using the {cert-manager-operator}.
. Skipping if certificates are already mounted on the clusters, certificates will be generated using the {cert-manager-operator}. The following steps assume you already have it installed on central cluster.

..An Issuer to generate the certificates by using the {cert-manager-operator}.
+
[source,yaml]
----
Expand Down Expand Up @@ -55,7 +57,7 @@ spec:
group: cert-manager.io
----

.. A CA issuer.
.. Create a CA issuer.
+
[source,yaml]
----
Expand All @@ -68,7 +70,7 @@ spec:
secretName: ca-secret
----

.. The client and server certificates.
.. The server certificates user in Central cluster.
+
[source,yaml]
----
Expand All @@ -77,83 +79,159 @@ kind: Certificate
metadata:
name: server
spec:
secretName: server-tls
secretName: server-tls #<1>
isCA: false
usages:
- server auth
- client auth
dnsNames:
- "otel.observability.svc.cluster.local" # <1>
- "central.observability-cluster.com" # <2>
issuerRef:
name: ca-issuer
---
----
<1> Secret will contain certificate
<2> List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.

.. The client certificates, this certificate will be used in the Edge cluster
+
[source,yaml]
----
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: client
spec:
secretName: client-tls
secretName: client-tls #<1>
isCA: false
usages:
- server auth
- client auth
dnsNames:
- "otel.observability.svc.cluster.local" # <2>
- "edge.observability-cluster.com" # <2>
issuerRef:
name: ca-issuer
----
<1> List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.
<1> Secret will contain certificate
<2> List of exact DNS names to be mapped to a solver in the client OpenTelemetry Collector instance.

. Create a service account for the OpenTelemetry Collector instance.

.Central Cluster

. Create the YAML file to define the `OpenTelemetryCollector` custom resource (CR) in the central cluster.
+
.Example ServiceAccount
.Example `OpenTelemetryCollector` custom resource for the central cluster
[source,yaml]
----
apiVersion: v1
kind: ServiceAccount
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-collector-deployment
name: otlp-receiver
namespace: chainsaw-multi-cluster-receive
spec:
config: |
receivers:
otlp:
protocols:
http:
tls: # <1>
cert_file: /certs/server.crt
key_file: /certs/server.key
client_ca_file: /certs/ca.crt
grpc:
tls:
cert_file: /certs/server.crt
key_file: /certs/server.key
client_ca_file: /certs/ca.crt
exporters:
otlp:
endpoint: "tempo-<simplest>-distributor:4317" #<2>
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp]
ingress:
route:
termination: passthrough #<3>
type: route
mode: deployment
volumeMounts:
- mountPath: /certs
name: otel-certs
volumes:
- name: otel-certs #<4>
secret:
name: server-tls
----
<1> The Collector receiver requires the certificates listed in the Generate Certificates step.
<2> The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is `"tempo-simplest-distributor:4317"` and already created.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [error] OpenShiftAsciiDoc.SuggestAttribute: Use the AsciiDoc attribute '{TempoName}' or '{TempoShortName}' rather than the plain text product term 'Tempo', unless your use case is an exception.

<3> The ingress type should be configure to route and termination will be passthrough, in order to not terminate TLS and forward certificates to the collector.
<4> The volume with the secrets generated by cert manager following the Generate Certificates section should be mounted

. Create a cluster role for the service account.
.For Edge cluster
. Create ServiceAccount , ClusterRole and ClusterRoleBinding for the OpenTelemetry collector, this is neeede to use resourceatributesdetector processor.
+
.Example ClusterRole
[source,yaml]
----
apiVersion: v1
kind: ServiceAccount
metadata:
name: otel-edge-multi-cluster
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: otel-collector
name: otel-edge-multi-cluster
rules:
# <1>
# <2>
- apiGroups: ["", "config.openshift.io"]
resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
verbs: ["get", "watch", "list"]
----
<1> The `k8sattributesprocessor` requires permissions for pods and namespace resources.
<2> The `resourcedetectionprocessor` requires permissions for infrastructures and status.

. Bind the cluster role to the service account.
+
.Example ClusterRoleBinding
[source,yaml]
----
- apiGroups:
- config.openshift.io
resources:
- infrastructures
- infrastructures/status
verbs:
- get
- watch
- list
- apiGroups:
- apps
resources:
- replicasets
verbs:
- get
- watch
- list
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- watch
- list
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: otel-collector
subjects:
- kind: ServiceAccount
name: otel-collector-deployment
namespace: otel-collector-<example>
name: otel-edge-multi-cluster
roleRef:
kind: ClusterRole
name: otel-collector
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: otel-edge-multi-cluster
subjects:
- kind: ServiceAccount
name: otel-edge-multi-cluster
----

+
. Create the YAML file to define the `OpenTelemetryCollector` custom resource (CR) in the edge clusters.
+
.Example `OpenTelemetryCollector` custom resource for the edge clusters
Expand All @@ -166,7 +244,7 @@ metadata:
namespace: otel-collector-<example>
spec:
mode: daemonset
serviceAccount: otel-collector-deployment
serviceAccount: otel-edge-multi-cluster
config: |
receivers:
jaeger:
Expand All @@ -192,7 +270,7 @@ spec:
detectors: [openshift]
exporters:
otlphttp:
endpoint: https://observability-cluster.com:443 # <1>
endpoint: https://central.observability-cluster.com:443 # <1>
tls:
insecure: false
cert_file: /certs/server.crt
Expand All @@ -207,57 +285,10 @@ spec:
volumes:
- name: otel-certs
secret:
name: otel-certs
volumeMounts:
name: client-tls
volumeMounts: #<2>
- name: otel-certs
mountPath: /certs
----
<1> The Collector exporter is configured to export OTLP HTTP and points to the OpenTelemetry Collector from the central cluster.

. Create the YAML file to define the `OpenTelemetryCollector` custom resource (CR) in the central cluster.
+
.Example `OpenTelemetryCollector` custom resource for the central cluster
[source,yaml]
----
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otlp-receiver
namespace: observability
spec:
mode: "deployment"
ingress:
type: route
route:
termination: "passthrough"
config: |
receivers:
otlp:
protocols:
http:
tls: # <1>
cert_file: /certs/server.crt
key_file: /certs/server.key
client_ca_file: /certs/ca.crt
exporters:
logging: {}
otlp:
endpoint: "tempo-<simplest>-distributor:4317" # <2>
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: []
exporters: [otlp]
volumes:
- name: otel-certs
secret:
name: otel-certs
volumeMounts:
- name: otel-certs
mountPath: /certs
----
<1> The Collector receiver requires the certificates listed in the first step.
<2> The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is `"tempo-simplest-distributor:4317"` and already created.
<2> The volume with the secrets generated by cert manager for the client certificated following the Generate Certificates section should be mounted,