OBSDOCS-1142: Add Tempo troubleshooting page

Signed-off-by: Ruben Vargas <[email protected]>

Author: rubenvp8510

Diff for: _topic_maps/_topic_map.yml

@@ -2929,6 +2929,8 @@ Topics:
       File: distr-tracing-tempo-installing
     - Name: Configuring
       File: distr-tracing-tempo-configuring
+    - Name: Troubleshooting
+      File: distr-tracing-tempo-troubleshooting
     - Name: Upgrading
       File: distr-tracing-tempo-updating
     - Name: Removing

Diff for: modules/distr-tracing-tempo-getting-the-tempostack-instance-logs.adoc

@@ -0,0 +1,36 @@
+// Module included in the following assemblies:
+//
+// * observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-troubleshooting.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="problems-ingesting-traces_{context}"]
+= Troubleshooting issues ingesting traces
+
+When the TempoStack instance is failing to ingest traces, you can troubleshoot it as follows.
+
+.Procedure
+
+. Review the components in the ingestion path to diagnose the issue:
++
+* The spans are not sent correctly.
+* The spans are not sampled.
+
+. Check for errors in the logs that identify unhealthy ingesters, which might be failing due to out-of-memory (OOM) issues or scale-down events.
++
+In high-volume tracing environments, the default trace limits might not be adequate.
+If spans are being refused due to these limits, you can discover this in the distributor logs.
++
+You can also see that the `tempo_discarded_spans_total` metric is incremented.
+
+. Check the following two metrics to verify that the TempoStack instance is successfully receiving traces:
++
+[source,yaml]
+----
+tempo_distributor_spans_received_total
+tempo_ingester_traces_created_total
+----
++
+** If both of these metrics are greather than zero, the TempoStack instance is successfully ingesting data.
+** If `tempo_distributor_spans_received_total` is `0`, the distributor is not receiving spans.
+Troubleshoot by checking the network configuration, protocols, and ports of the applications that are attempting to send traces to the distributor.
+** If `tempo_ingester_traces_created_total is `0`, there is a communication issue between the ingester and the distributor. Troubleshoot by inspecting the logs for both to find errors.

Diff for: modules/distr-tracing-tempo-troubleshooting-issues-ingesting-traces.adoc

@@ -0,0 +1,25 @@
+// Module included in the following assemblies:
+//
+// * observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-troubleshooting.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="problems-quering-traces_{context}"]
+= Troubleshooting issues with quering traces
+
+Wheb the issues remain even though you have verified that the TempoStack instance is successfully ingesting data, troubleshoot for issues with querying the data.
+
+.Procedure
+
+. Check the logs of the `query-frontend` pod, which runs with two containers: `query-frontend` and `querier`. Logs containing errors similar to the following indicate an error in the query path:
++
+[source,terminal]
+----
+level=info ts=<...> caller=frontend.go:63 method=GET traceID=<...> url=/api/traces/<...> duration=<...> status=500
+could not dial 10.X.X.X:3200 connection refused
+----
+
+. Review the logs of the `tempo-querier` and `tempo-queryfrontend` services to determine if either has crashed and as a result unable to communicate with each other.
+
+. Check your network settings and policies to ensure proper communication between them.
+
+. Review the `cortex_query_frontend_connected_clients` metric, which is exposed by the `query-frontend`. If `cortex_query_frontend_connected_clients` is greater than `0`, it is an indication than the `queriers` are connected to the `query-frontend`.

Diff for: modules/distr-tracing-tempo-troubleshooting-issues-with-quering-traces.adoc

@@ -0,0 +1,17 @@
+// Module included in the following assemblies:
+//
+// * observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-troubleshooting.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="getting-tempo-stack-logs_{context}"]
+= Getting the TempoStack instance logs
+
+You can get the logs from a TempoStack component as follows.
+
+.Procedure
+
+. Locate the component you want to get the logs for. You can do this by listing all the deployments on an specific namespace.
+
+. Identify the pods that belong to this component.
+
+. Watch the logs by using `oc logs` command or retrieve the logs by using the web console.

Diff for: observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-troubleshooting.adoc

@@ -0,0 +1,22 @@
+:_mod-docs-content-type: ASSEMBLY
+[id="dist-tracing-tempo-troubleshooting"]
+= Troubleshooting
+include::_attributes/common-attributes.adoc[]
+:context: dist-tracing-tempo-troubleshooting
+
+toc::[]
+
+You can diagnose and fix TempoStack instance issues by using various troubleshooting methods.
+
+You can inspect the logs of the different components along with some useful metrics.
+
+If the the TempoStack instance has no traces, there are two possible causes:
+
+* TempoStack instance is not ingesting traces.
+* There is an issue with the query path.
+
+include::modules/distr-tracing-tempo-getting-the-tempostack-instance-logs.adoc[leveloffset=+1]
+
+include::modules/distr-tracing-tempo-troubleshooting-issues-ingesting-traces.adoc[leveloffset=+1]
+
+include::modules/distr-tracing-tempo-troubleshooting-issues-with-quering-traces.adoc[leveloffset=+1]