|
| 1 | +--- |
| 2 | +id: sdk-metrics-setup |
| 3 | +title: Monitor SDK metrics with Prometheus and Grafana |
| 4 | +sidebar_label: SDK Metrics |
| 5 | +description: Set up Temporal SDK metrics with Prometheus and Grafana for monitoring Workers and Client performance. |
| 6 | +slug: /cloud/metrics/sdk-metrics-setup |
| 7 | +toc_max_heading_level: 4 |
| 8 | +keywords: |
| 9 | + - temporal sdk metrics |
| 10 | + - prometheus scrape endpoint |
| 11 | + - sdk metrics setup |
| 12 | + - temporal sdk monitoring |
| 13 | + - grafana sdk metrics |
| 14 | + - worker metrics |
| 15 | + - temporal sdk prometheus |
| 16 | + - sdk metrics dashboard |
| 17 | +tags: |
| 18 | + - Metrics |
| 19 | + - Observability |
| 20 | + - Temporal Cloud |
| 21 | +--- |
| 22 | + |
| 23 | +import { ZoomingImage } from '@site/src/components'; |
| 24 | + |
| 25 | +SDK metrics are emitted by SDK Clients used to start your Workers and to start, signal, or query your Workflow Executions. |
| 26 | +Unlike [Temporal Cloud metrics](/cloud/metrics/), which are exposed through a Prometheus HTTP API endpoint, SDK metrics require you to set up a Prometheus scrape endpoint in your application code for Prometheus to collect and aggregate. |
| 27 | + |
| 28 | +For a full list of available SDK metrics and their descriptions, see the [SDK metrics reference](/references/sdk-metrics). |
| 29 | + |
| 30 | +The process for setting up SDK metrics includes the following steps: |
| 31 | + |
| 32 | +1. [Expose a metrics endpoint](#sdk-metrics-setup) in your application code where Prometheus can scrape SDK metrics. |
| 33 | +2. [Configure Prometheus](#prometheus-configuration) to scrape your SDK metrics endpoints. |
| 34 | +3. [Add an SDK metrics data source](#grafana-data-source-configuration) in Grafana. |
| 35 | +4. [Set up dashboards](#grafana-dashboards-setup) to visualize SDK metrics. |
| 36 | + |
| 37 | +Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud. |
| 38 | +Ensure Prometheus and Grafana are installed. |
| 39 | + |
| 40 | +- [Go](/develop/go/temporal-client#connect-to-temporal-cloud) |
| 41 | +- [Java](/develop/java/temporal-client#connect-to-temporal-cloud) |
| 42 | +- [Python](/develop/python/temporal-client#connect-to-temporal-cloud) |
| 43 | +- [TypeScript](/develop/typescript/core-application#connect-to-temporal-cloud) |
| 44 | +- [.NET](/develop/dotnet/temporal-client#connect-to-temporal-cloud) |
| 45 | + |
| 46 | +## Expose a metrics endpoint {#sdk-metrics-setup} |
| 47 | + |
| 48 | +You must configure a Prometheus scrape endpoint for Prometheus to collect and aggregate your SDK metrics. |
| 49 | +Each language development guide has details on how to set this up. |
| 50 | + |
| 51 | +- [Go SDK](/develop/go/observability#metrics) |
| 52 | +- [Java SDK](/develop/java/observability#metrics) |
| 53 | +- [TypeScript SDK](/develop/typescript/observability#metrics) |
| 54 | +- [Python](/develop/python/observability#metrics) |
| 55 | +- [.NET](/develop/dotnet/observability#metrics) |
| 56 | + |
| 57 | +For working examples of how to configure metrics in each SDK, see the metrics samples: |
| 58 | + |
| 59 | +- [Go SDK Samples](https://github.com/temporalio/samples-go/tree/main/metrics) |
| 60 | +- [Java SDK Samples](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics) |
| 61 | +- [TypeScript SDK Samples](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) |
| 62 | +- [Python SDK Samples](https://github.com/temporalio/samples-python/tree/main/custom_metric) |
| 63 | +- [.NET SDK Samples](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry/DotNetMetrics) |
| 64 | + |
| 65 | +Some examples use OpenTelemtry to instrument metrics. It is useful to use a |
| 66 | +[Prometheus exporter with OpenTelemetry](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter) to expose metrics for scraping. |
| 67 | + |
| 68 | +## Configure Prometheus {#prometheus-configuration} |
| 69 | + |
| 70 | +For Temporal SDKs, you must have Prometheus running and configured to listen on the scrape endpoints exposed in your application code. |
| 71 | + |
| 72 | +For this example, you can run Prometheus locally or as a Docker container. |
| 73 | +In either case, ensure that you set the listen targets to the ports where you expose your scrape endpoints. |
| 74 | +This configuration assumes the scrape endpoint is set to port 8077 as in the [SDK metrics setup](#sdk-metrics-setup) example. |
| 75 | + |
| 76 | +```yaml |
| 77 | +global: |
| 78 | + scrape_interval: 30s # Set the scrape interval to every 30 seconds. Default is every 1 minute. |
| 79 | +#... |
| 80 | + |
| 81 | +# Set your scrape configuration targets to the ports exposed on your endpoints in the SDK. |
| 82 | +scrape_configs: |
| 83 | + - job_name: 'temporalsdkmetrics' |
| 84 | + metrics_path: /metrics |
| 85 | + scheme: http |
| 86 | + static_configs: |
| 87 | + - targets: |
| 88 | + # This is the scrape endpoint where Prometheus listens for SDK metrics. |
| 89 | + - localhost:8077 |
| 90 | + # You can have multiple targets here, provided they are set up in your application code. |
| 91 | +``` |
| 92 | + |
| 93 | +See the [Prometheus documentation](https://prometheus.io/docs/introduction/first_steps/) for more details on how you can run Prometheus locally or using Docker. |
| 94 | + |
| 95 | +To check whether Prometheus is receiving metrics from your SDK target, go to [http://localhost:9090](http://localhost:9090) and navigate to **Status > Targets**. |
| 96 | +The status of your target endpoint defined in your configuration appears here. |
| 97 | + |
| 98 | +## Add an SDK metrics data source in Grafana {#grafana-data-source-configuration} |
| 99 | + |
| 100 | +Depending on how you use Grafana, you can either install and run it locally, run it as a Docker container, or log in to Grafana Cloud to set up your data sources. |
| 101 | + |
| 102 | +If you have installed and are running Grafana locally, go to [http://localhost:3000](http://localhost:3000) and sign in. |
| 103 | + |
| 104 | +To add the SDK metrics Prometheus endpoint as a data source, do the following: |
| 105 | + |
| 106 | +1. Go to **Configuration > Data sources**. |
| 107 | +2. Select **Add data source > Prometheus**. |
| 108 | +3. Enter a name for your SDK metrics data source, such as _Temporal SDK metrics_. |
| 109 | +4. In the **HTTP** section, enter your Prometheus endpoint in the URL field. |
| 110 | + If running Prometheus locally as described in the examples in this article, enter `http://localhost:9090`. |
| 111 | +5. For this example, enable **Skip TLS Verify** in the **Auth** section. |
| 112 | +6. Click **Save and test** to verify that the data source is working. |
| 113 | + |
| 114 | +If you see issues in setting this data source, check whether the endpoints set in your SDKs are showing metrics. |
| 115 | +If you don't see your SDK metrics at the scrape endpoints defined, check whether your Workers and Workflow Executions are running. |
| 116 | +If you see metrics on the scrape endpoints, but Prometheus shows your targets are down, then there is an issue with connecting to the targets set in your SDKs. |
| 117 | +Verify your Prometheus configuration and restart Prometheus. |
| 118 | + |
| 119 | +If you're running Grafana as a container, you can set your SDK metrics Prometheus data source in your Grafana configuration. |
| 120 | +See the example Grafana configuration described in the [Prometheus and Grafana setup for open-source Temporal Service](/self-hosted-guide/monitoring#grafana) article. |
| 121 | + |
| 122 | +## Set up Grafana dashboards {#grafana-dashboards-setup} |
| 123 | + |
| 124 | +To set up SDK metrics dashboards in Grafana, you can use the UI or configure them directly in your Grafana deployment. |
| 125 | + |
| 126 | +:::tip |
| 127 | + |
| 128 | +Temporal provides community-driven [example dashboards for Temporal SDKs](https://github.com/temporalio/dashboards/tree/master/sdk) that you can customize to meet your needs. |
| 129 | + |
| 130 | +::: |
| 131 | + |
| 132 | +To import a dashboard in Grafana: |
| 133 | + |
| 134 | +1. In the navigation bar, select **Dashboards** > **Import dashboard**. |
| 135 | +2. You can either copy and paste the JSON from the [Temporal SDK sample dashboards](https://github.com/temporalio/dashboards/tree/master/sdk), or import the JSON files into Grafana. |
| 136 | +3. Save the dashboard and review the metrics data in the graphs. |
| 137 | + |
| 138 | +To configure dashboards with the UI: |
| 139 | + |
| 140 | +1. Go to **Create > Dashboard** and add an empty panel. |
| 141 | +2. On the **Panel configuration** page, in the **Query** tab, select the "Temporal SDK metrics" data source that you configured earlier. |
| 142 | +3. Expand the **Metrics browser** and select the metrics you want. |
| 143 | + A list of Worker performance metrics is described in the [Developer's Guide - Worker performance](/develop/worker-performance). |
| 144 | + All SDK-related metrics are listed in the [SDK metrics](/references/sdk-metrics) reference. |
| 145 | +4. The graph should now display data based on your selected queries. |
| 146 | + Note that SDK metrics will only show if you have Workflow Execution data and running Workers. |
| 147 | + If you don't see SDK metrics, run your Worker and Workflow Executions, then monitor the dashboard. |
0 commit comments