-
Notifications
You must be signed in to change notification settings - Fork 52
Support for DataDog SDK dashboards and conf.yaml support. Doc improvements #70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
c23e2a7
datadog dashboard and conf.yaml support
mnichols 2502dc7
provide links to things everyone will need. tidy up the documentation
mnichols 64f7e98
overcommunicate to make it easy to find
mnichols c482444
fix indentation broken config
mnichols c624e45
mixing in Antonio remarks
mnichols File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| .idea |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Temporal Cloud Dashboards | ||
|
|
||
| ## Setup | ||
| * [Temporal Cloud](https://docs.temporal.io/cloud/metrics/) | ||
|
|
||
| ## Available metrics | ||
| * [Temporal Cloud metrics](https://docs.temporal.io/production-deployment/cloud/metrics/reference) | ||
|
|
||
| ## Dashboards | ||
| * **Grafana** [here](temporal_cloud.json) | ||
| * **DataDog** integration details and Dashboard access are found [here](https://docs.datadoghq.com/integrations/temporal-cloud/). | ||
| * Related [Blog post](https://temporal.io/blog/temporal-cloud-metrics-in-datadog). | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,24 @@ | ||
| # Temporal Grafana SDK Dashboards | ||
| # Temporal SDK Dashboards | ||
|
|
||
| This repository contains the following dashboards: | ||
| ## Setup | ||
|
|
||
| * [Golang](https://docs.temporal.io/develop/go/observability) | ||
| * [Java](https://docs.temporal.io/develop/java/observability) | ||
| * [Python](https://docs.temporal.io/develop/python/observability) | ||
| * [TypeScript](https://docs.temporal.io/develop/typescript/observability) | ||
| * [.NET](https://docs.temporal.io/develop/dotnet/observability) | ||
| * [PHP](https://docs.temporal.io/develop/php/observability) | ||
|
|
||
| ## Available metrics | ||
| [Temporal SDK metrics](https://docs.temporal.io/references/sdk-metrics) | ||
|
|
||
| ## Grafana dashboards | ||
| - [temporal-go-java-sdks-tally.json](temporal-go-java-sdks-tally.json) for [Go](https://github.com/temporalio/sdk-go) and [Java](https://github.com/temporalio/sdk-java) SDKs using Uber Tally to emits metrics. | ||
|
|
||
| - [temporal-go-sdk-otel.json](temporal-go-sdk-otel.json) for [Go](https://github.com/temporalio/sdk-go) SDK using OpenTelemetry to emit metrics. | ||
|
|
||
| - [temporal-core-sdks-otel.json](temporal-core-sdks-otel.json) for [Core](https://github.com/temporalio/sdk-core) based SDKs. In Core based SDKs, metrics of the type Histogram | ||
| are measured in milliseconds by default, so the dashboard is configured accordingly to display them in milliseconds. | ||
| - [temporal-core-sdks-otel.json](temporal-core-sdks-otel.json) for [Core](https://github.com/temporalio/sdk-core) based SDKs. In Core based SDKs, metrics of the type Histogram | ||
| are measured in milliseconds by default, so the dashboard is configured accordingly to display them in milliseconds. | ||
|
|
||
| ## DataDog dashboards | ||
| DataDog dashboards and related configuration are [here](datadog). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Temporal DataDog SDK Dashboards | ||
|
|
||
| The [Dashboard](temporal_sdk_dashboard.json) here works with [this openmetrics configuration](openmetrics.h_conf.yaml). | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| 1. [DataDog Agent](https://docs.datadoghq.com/getting_started/agent/), installed. | ||
|
mnichols marked this conversation as resolved.
Outdated
|
||
| 2. Advanced `Percentile` configuration for Distribution metrics. | ||
| 1. Which metrics? Basically, any metrics [here](https://docs.temporal.io/references/sdk-metrics) that are a `Histogram` and for which you want to report percentiles (p95/p99). | ||
|
|
||
| ### Put it all together | ||
| 1. Visit [here](https://docs.datadoghq.com/integrations/openmetrics/) for OpenMetrics configuration with the DataDog agent for details about configuring your DD Agent. | ||
| 2. Drop this [conf.yaml](openmetrics.h_conf.yaml) at your Agent `openmetrics.d` config path. | ||
|
mnichols marked this conversation as resolved.
Outdated
|
||
| 3. Import the [Dashboard](temporal_sdk_dashboard.json) | ||
| 4. Be sure you enable the `Percentile` configuration for relevant metrics. | ||
|
|
||
| > **Example for configuring `temporal_request_latency`**: | ||
| > * Visit https://app.datadoghq.com/metric/summary?metric=temporal_request_latency | ||
| > * Set `Advanced > Percentiles > Configure > ON` | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| ## OpenMetrics Configuration - CONVERT BUCKETS TO DISTRIBUTIONS | ||
|
|
||
| init_config: | ||
|
|
||
| instances: | ||
| - openmetrics_endpoint: "http://localhost:9464/metrics" | ||
| tags: | ||
| - "service:temporal-worker" | ||
| - "env:production" | ||
|
|
||
| metrics: | ||
| - "temporal_*" | ||
|
|
||
| # KEY: Convert histogram buckets to distributions | ||
| histogram_buckets_as_distributions: true # Convert to distributions | ||
| collect_histogram_buckets: true | ||
|
|
||
| # Send buckets as distributions instead of individual metrics | ||
| send_distribution_buckets: true # Send as distributions | ||
| send_distribution_counts_as_monotonic: true # For proper count handling | ||
| send_histograms_buckets: false | ||
|
|
||
| # Include all labels including "le" for bucket conversion | ||
| include_labels: | ||
| - "le" # NEEDED for bucket conversion | ||
| - "namespace" | ||
| - "operation" | ||
| - "workflow_type" | ||
| - "activity_type" | ||
| - "task_queue" | ||
| - "worker_type" | ||
| - "service_name" | ||
| - "error_type" | ||
| - "poller_type" | ||
| - "failure_reason" | ||
|
|
||
| exclude_labels: | ||
| - "job" | ||
| - "instance" | ||
| - "__name__" | ||
|
|
||
| timeout: 30 | ||
| min_collection_interval: 15 | ||
|
|
||
| # This will create distribution metrics that you can then enable percentiles for | ||
| # in the Datadog Metrics Summary page: | ||
| # temporal_request_latency (distribution) | ||
| # temporal_workflow_endtoend_latency (distribution) | ||
| # etc. | ||
| # Example: You want to enable the `p95` percentile for `temporal_request_latency`: | ||
| # 1. Visit https://app.datadoghq.com/metric/summary?metric=temporal_request_latency | ||
| # 2. Set `Advanced > Percentiles > Configure > ON` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.