Skip to content

Include Stage name in emitted metrics and dashboards #1488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
May 30, 2025
Merged

Include Stage name in emitted metrics and dashboards #1488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f7dc7c5
Add stage name to metrics with trace sampling
AndreKurait Apr 29, 2025
f44a26a
Update otel config default aws
AndreKurait Apr 29, 2025
9489b32
Update otel to take advantage of improvements in env provider
AndreKurait Apr 29, 2025
26e765a
Update aws otel config for resource cleanup, detailed metrics, fewer …
AndreKurait Apr 29, 2025
2d96f6c
Rename stage to qualifier in otel metric exporting
AndreKurait Apr 29, 2025
99bf768
Merge branch 'DashboardsRegionalization' into OtelConfig
AndreKurait Apr 29, 2025
cd2535b
Update dashboard with qualifier
AndreKurait Apr 29, 2025
5dbd515
Update dashboards for qualifier
AndreKurait Apr 29, 2025
37a4cdb
Update dashboard
AndreKurait Apr 30, 2025
badde92
Simplify otel config
AndreKurait May 1, 2025
4ae5d37
Update qualifier metric name selector
AndreKurait May 1, 2025
0c3d0f0
Update Capture and replay and otel config for qualifier
AndreKurait May 7, 2025
d25c0c3
Merge branch 'main' into OtelConfig
AndreKurait May 7, 2025
1a55a8d
Update otel logging exporter to debug https://github.com/open-telemet…
AndreKurait May 19, 2025
be2dc90
Update cloudwatch otel config in config snippints
AndreKurait May 19, 2025
0ce03bc
Add explicit healthcheck configuration
AndreKurait May 19, 2025
57401a4
Add explicit configuration for prometheus pull exporter
AndreKurait May 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/awsCloudWatch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,30 @@
exporters:
awsemf:
namespace: 'OpenSearchMigrations'
dimension_rollup_option: NoDimensionRollup # Reduce number of metrics by only publishing with all dimensions
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on this ? Our dashboards will make an assumption that dimensions are guaranteed to be present.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only publishes with the most granular dimension, if this was excluded, as an example, we would be publishing the metrics as stage-specific metrics as well as not stage specific

resource_to_telemetry_conversion:
enabled: true

processors:
cumulativetodelta:
resource/metrics:
attributes:
- key: qualifier
value: ${env:QUALIFIER}
action: upsert
resource/remove_default_attributes:
attributes:
- key: telemetry.sdk.name
action: delete
- key: telemetry.sdk.version
action: delete
- key: telemetry.sdk.language
action: delete
- key: service.name
action: delete

service:
pipelines:
metrics:
exporters: [ awsemf ]
processors: [resource/remove_default_attributes, resource/metrics, cumulativetodelta]
exporters: [ awsemf ]
3 changes: 3 additions & 0 deletions TrafficCapture/dockerSolution/otelConfigs/configSnippets/base.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ receivers:
otlp:
protocols:
grpc:
endpoint: :4317
http:
endpoint: :4318

processors:

Expand Down
6 changes: 5 additions & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/batchTraces.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
processors:
probabilistic_sampler/traces:
sampling_percentage: ${env:TRACE_SAMPLING_PERCENTAGE:-1}

service:
pipelines:
traces:
processors: [ batch ]
processors: [probabilistic_sampler/traces, batch]
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugDetailed.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
exporters:
logging:
debug:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed to upgrade otel, logging is deprecated/removed

verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugInfo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
exporters:
logging:
debug:
verbosity: info
sampling_initial: 5
sampling_thereafter: 200
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugLogsDetailed.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
service:
pipelines:
logs:
exporters: [logging]
exporters: [debug]
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugLogsInfo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
service:
pipelines:
logs:
exporters: [logging]
exporters: [debug]
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugMetricsDetailed.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
service:
pipelines:
metrics:
exporters: [logging]
exporters: [debug]
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugMetricsInfo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
service:
pipelines:
metrics:
exporters: [logging]
exporters: [debug]
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugTracesDetailed.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
service:
pipelines:
traces:
exporters: [logging]
exporters: [debug]
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/otelConfigs/configSnippets/debugTracesInfo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
service:
pipelines:
traces:
exporters: [logging]
exporters: [debug]
1 change: 1 addition & 0 deletions TrafficCapture/dockerSolution/otelConfigs/configSnippets/healthCheck.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
extensions:
health_check:
endpoint: :13133

service:
extensions: [ health_check ]
8 changes: 8 additions & 0 deletions TrafficCapture/dockerSolution/otelConfigs/configSnippets/prometheus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ exporters:
enable_open_metrics: true

service:
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: '0.0.0.0'
port: 8888
pipelines:
metrics:
exporters: [ prometheus ]
45 changes: 39 additions & 6 deletions ...ture/dockerSolution/src/main/docker/composeExtensions/configs/otel-config-everything.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,49 @@ receivers:
otlp:
protocols:
grpc:
endpoint: :4317
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears the default here changed from :4317 to localhost:4317 which broke some of our inter-container communication without moving this back.

http:
endpoint: :4318
processors:
batch:
timeout: 10s
send_batch_size: 8192
send_batch_max_size: 10000
probabilistic_sampler/traces:
sampling_percentage: ${env:TRACE_SAMPLING_PERCENTAGE:-1}
cumulativetodelta:
resource/metrics:
attributes:
- key: qualifier
value: ${env:QUALIFIER}
action: upsert
resource/remove_default_attributes:
attributes:
- key: telemetry.sdk.name
action: delete
- key: telemetry.sdk.version
action: delete
- key: telemetry.sdk.language
action: delete
- key: service.name
action: delete
extensions:
zpages:
endpoint: :55679
pprof:
endpoint: :1888
health_check:
endpoint: :13133
exporters:
logging:
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
awsemf:
namespace: 'OpenSearchMigrations'
dimension_rollup_option: NoDimensionRollup # Reduce number of metrics by only publishing with all dimensions
resource_to_telemetry_conversion:
enabled: true
awsxray:
index_all_attributes: true
prometheus:
Expand All @@ -36,13 +61,21 @@ service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, awsemf, prometheus]
processors: [batch, resource/remove_default_attributes, resource/metrics, cumulativetodelta]
exporters: [debug, awsemf, prometheus]
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, awsxray, otlp/jaeger]
processors: [probabilistic_sampler/traces, batch]
exporters: [debug, awsxray, otlp/jaeger]
logs:
receivers: [otlp]
processors:
exporters: [logging]
exporters: [debug]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: '0.0.0.0'
port: 8888
24 changes: 19 additions & 5 deletions ...ckerSolution/src/main/docker/composeExtensions/configs/otel-config-prometheus-jaeger.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,25 @@ receivers:
otlp:
protocols:
grpc:
endpoint: :4317
http:
endpoint: :4318
processors:
batch:
timeout: 10s
send_batch_size: 8192
send_batch_max_size: 10000
probabilistic_sampler/traces:
sampling_percentage: ${env:TRACE_SAMPLING_PERCENTAGE:-1}
extensions:
zpages:
endpoint: :55679
pprof:
endpoint: :1888
health_check:
endpoint: :13133
exporters:
logging:
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
Expand All @@ -33,12 +39,20 @@ service:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, prometheus]
exporters: [debug, prometheus]
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp/jaeger]
processors: [probabilistic_sampler/traces, batch]
exporters: [debug, otlp/jaeger]
logs:
receivers: [otlp]
processors:
exporters: [logging]
exporters: [debug]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: '0.0.0.0'
port: 8888
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/src/main/docker/grafana/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
FROM grafana/grafana:11.1.0
FROM grafana/grafana:11.6.1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't needed, i can revert if we want

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If all tests are passing with this version update, then let's keep it.


COPY datasources.yaml /usr/share/grafana/conf/provisioning/datasources/
12 changes: 12 additions & 0 deletions ...docker/migrationConsole/lib/console_link/tests/data/cloudwatch_list_metrics_response.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
{
"Name": "OTelLib",
"Value": "captureProxy"
},
{
"Name": "qualifier",
"Value": "dev"
}
]
},
Expand All @@ -17,6 +21,10 @@
{
"Name": "OTelLib",
"Value": "captureProxy"
},
{
"Name": "qualifier",
"Value": "dev"
}
]
},
Expand All @@ -27,6 +35,10 @@
{
"Name": "OTelLib",
"Value": "replayer"
},
{
"Name": "qualifier",
"Value": "dev"
}
]
}
Expand Down
2 changes: 1 addition & 1 deletion TrafficCapture/dockerSolution/src/main/docker/otelCollector/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM public.ecr.aws/aws-observability/aws-otel-collector:v0.38.0
FROM public.ecr.aws/aws-observability/aws-otel-collector:v0.43.3
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, does this version update have any breaking changes impact our codebase?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see earlier comment which i adjusted to fix, as well as logging -> debug

It appears the default here changed from :4317 to localhost:4317 which broke some of our inter-container communication without moving this back.


COPY otel-config*.yaml /etc/
CMD ["--config", "/etc/otel-config-debug-only.yaml"]
Expand Down
37 changes: 31 additions & 6 deletions TrafficCapture/dockerSolution/src/main/docker/otelCollector/otel-config-aws-debug.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,59 @@ receivers:
otlp:
protocols:
grpc:
endpoint: :4317
http:
endpoint: :4318
processors:
batch:
timeout: 10s
send_batch_size: 8192
send_batch_max_size: 10000
probabilistic_sampler/traces:
sampling_percentage: ${env:TRACE_SAMPLING_PERCENTAGE:-1}
cumulativetodelta:
resource/metrics:
attributes:
- key: qualifier
value: ${env:QUALIFIER}
action: upsert
resource/remove_default_attributes:
attributes:
- key: telemetry.sdk.name
action: delete
- key: telemetry.sdk.version
action: delete
- key: telemetry.sdk.language
action: delete
- key: service.name
action: delete
extensions:
health_check:
endpoint: :13133
exporters:
logging:
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
awsemf:
namespace: 'OpenSearchMigrations'
dimension_rollup_option: NoDimensionRollup # Reduce number of metrics by only publishing with all dimensions
resource_to_telemetry_conversion:
enabled: true
awsxray:
index_all_attributes: true
service:
extensions: [health_check]
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, awsemf]
processors: [batch, resource/remove_default_attributes, resource/metrics, cumulativetodelta]
exporters: [debug, awsemf]
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, awsxray]
processors: [probabilistic_sampler/traces, batch]
exporters: [debug, awsxray]
logs:
receivers: [otlp]
processors:
exporters: [logging]
exporters: [debug]
Loading
Loading