Include Stage name in emitted metrics and dashboards #1488

AndreKurait · 2025-04-29T14:13:16Z

Description

Add stage name to metrics with trace sampling.

Fix dependency ordering with otel collector and service container, fixes otel sdk connection error during task shutdown / startup when the service is running without the collector.

Issues Resolved

https://opensearch.atlassian.net/browse/MIGRATIONS-1949

Testing

Verified metrics with dimension and dashboards in jenkins environment

C&R Dashboard Example:

Check List

New functionality includes testing
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait · 2025-05-22T16:32:51Z

...Capture/dockerSolution/src/main/docker/composeExtensions/configs/otel-config-everything.yaml

  otlp:
    protocols:
      grpc:
+        endpoint: :4317


It appears the default here changed from :4317 to localhost:4317 which broke some of our inter-container communication without moving this back.

AndreKurait · 2025-05-22T17:03:49Z

TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugDetailed.yaml

@@ -1,5 +1,5 @@
 exporters:
-  logging:
+  debug:


Needed to upgrade otel, logging is deprecated/removed

AndreKurait · 2025-05-22T17:04:23Z

TrafficCapture/dockerSolution/src/main/docker/grafana/Dockerfile

@@ -1,3 +1,3 @@
-FROM grafana/grafana:11.1.0
+FROM grafana/grafana:11.6.1


This wasn't needed, i can revert if we want

If all tests are passing with this version update, then let's keep it.

jugal-chauhan · 2025-05-29T14:46:01Z

TrafficCapture/dockerSolution/src/main/docker/otelCollector/Dockerfile

@@ -1,4 +1,4 @@
-FROM public.ecr.aws/aws-observability/aws-otel-collector:v0.38.0
+FROM public.ecr.aws/aws-observability/aws-otel-collector:v0.43.3


Just curious, does this version update have any breaking changes impact our codebase?

Yes, see earlier comment which i adjusted to fix, as well as logging -> debug

It appears the default here changed from :4317 to localhost:4317 which broke some of our inter-container communication without moving this back.

jugal-chauhan · 2025-05-29T14:50:07Z

TrafficCapture/dockerSolution/src/main/docker/otelCollector/otel-config-aws.yaml

+        action: delete
 extensions:
  health_check:
+    endpoint: :13133


Will having an endpoint for health checks help us differentiate between an actual request vs health check requests in our kafka logs? If yes, then I think that Capture and Replay Dashboard widget for request/response mismatches should exclude health check requests.

This is otel healthcheck endpoint, will not appear on any dashboards or kafka

jugal-chauhan · 2025-05-29T15:14:09Z

deployment/cdk/opensearch-service-migration/lib/constructs/migration-dashboard.ts

            'REGION': props.region,
-            'MA_STAGE': props.stage
+            'MA_STAGE': props.stage,
+            'MA_QUALIFIER': props.stage


Should we change this to props.qualifier ?? props.stage ? in case in the future we want to diverge the use-cases of qualifier from stage

jugal-chauhan · 2025-05-29T15:18:59Z

TrafficCapture/dockerSolution/otelConfigs/configSnippets/awsCloudWatch.yaml

 exporters:
  awsemf:
    namespace: 'OpenSearchMigrations'
+    dimension_rollup_option: NoDimensionRollup # Reduce number of metrics by only publishing with all dimensions


Can you elaborate on this ? Our dashboards will make an assumption that dimensions are guaranteed to be present.

This only publishes with the most granular dimension, if this was excluded, as an example, we would be publishing the metrics as stage-specific metrics as well as not stage specific

jugal-chauhan

Approving this PR as the changes are consistent and all tests appear to be passing.

codecov · 2025-05-30T14:13:45Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (3db3f15) to head (57401a4).
Report is 148 commits behind head on main.

Additional details and impacted files

@@     Coverage Diff      @@
##   main   #1488   +/-   ##
============================
============================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 14:13 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 14:13 — with GitHub Actions Failure

Add stage name to metrics with trace sampling

f7dc7c5

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the OtelConfig branch from 8cf8b49 to f7dc7c5 Compare April 29, 2025 14:15

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 14:15 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 14:15 — with GitHub Actions Failure

AndreKurait marked this pull request as ready for review April 29, 2025 14:15

AndreKurait requested review from chelma, gregschohn, jugal-chauhan, lewijacn, mikaylathompson, peternied and sumobrian as code owners April 29, 2025 14:15

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 15:03 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 15:03 — with GitHub Actions Inactive

AndreKurait force-pushed the OtelConfig branch from 255c038 to 591ce80 Compare April 29, 2025 15:25

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 15:25 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 15:25 — with GitHub Actions Inactive

AndreKurait force-pushed the OtelConfig branch from 591ce80 to 47ad23d Compare April 29, 2025 15:44

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 15:44 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 15:44 — with GitHub Actions Failure

AndreKurait force-pushed the OtelConfig branch from 47ad23d to 47ee54c Compare April 29, 2025 16:50

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 16:50 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 16:50 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 17:19 — with GitHub Actions Failure

Update otel config default aws

f44a26a

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the OtelConfig branch from 47ee54c to f44a26a Compare April 29, 2025 17:58

AndreKurait had a problem deploying to migrations-cicd April 29, 2025 17:58 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd April 29, 2025 17:58 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd May 19, 2025 15:20 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd May 19, 2025 15:20 — with GitHub Actions Inactive

Update cloudwatch otel config in config snippints

be2dc90

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait temporarily deployed to migrations-cicd May 19, 2025 18:18 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd May 19, 2025 18:18 — with GitHub Actions Failure

Add explicit healthcheck configuration

0ce03bc

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait temporarily deployed to migrations-cicd May 19, 2025 18:22 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd May 19, 2025 18:22 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd May 19, 2025 18:25 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd May 19, 2025 18:25 — with GitHub Actions Failure

AndreKurait had a problem deploying to migrations-cicd May 19, 2025 19:00 — with GitHub Actions Failure

AndreKurait temporarily deployed to migrations-cicd May 19, 2025 19:00 — with GitHub Actions Inactive

AndreKurait had a problem deploying to migrations-cicd May 20, 2025 15:40 — with GitHub Actions Failure

AndreKurait had a problem deploying to migrations-cicd May 21, 2025 14:20 — with GitHub Actions Failure

AndreKurait had a problem deploying to migrations-cicd May 21, 2025 21:03 — with GitHub Actions Failure

AndreKurait had a problem deploying to migrations-cicd May 22, 2025 14:42 — with GitHub Actions Failure

Add explicit configuration for prometheus pull exporter

57401a4

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the OtelConfig branch from 0b5846d to 57401a4 Compare May 22, 2025 15:33

AndreKurait temporarily deployed to migrations-cicd May 22, 2025 15:33 — with GitHub Actions Inactive

AndreKurait commented May 22, 2025

View reviewed changes

jugal-chauhan reviewed May 29, 2025

View reviewed changes

jugal-chauhan approved these changes May 29, 2025

View reviewed changes

AndreKurait merged commit 6b663c1 into opensearch-project:main May 30, 2025
58 checks passed

		@@ -1,3 +1,3 @@
		FROM grafana/grafana:11.1.0
		FROM grafana/grafana:11.6.1

		@@ -1,4 +1,4 @@
		FROM public.ecr.aws/aws-observability/aws-otel-collector:v0.38.0
		FROM public.ecr.aws/aws-observability/aws-otel-collector:v0.43.3

Include Stage name in emitted metrics and dashboards #1488

Include Stage name in emitted metrics and dashboards #1488

Uh oh!

Conversation

AndreKurait commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Testing

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jugal-chauhan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented May 30, 2025

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AndreKurait commented Apr 29, 2025 •

edited

Loading