Skip to content

Commit d8a49d3

Browse files
fix(observability): stop stamping node on the SM/PM scraping path
The prometheusCR pipeline ran set_node_name/promote_node_name, unconditionally overwriting k8s.node.name with the scraping agent's own node. That is only correct when per-node placement succeeds; under the consistent-hashing fallback (cross-node target) it mislabels the metric, and it clobbers any node label the target/relabeling already set. Per-node placement is verified via the Target Allocator logs, so the stamp is not needed here -- remove both transforms from the prometheusCR pipeline. The shared transform definitions stay for the node-level pipelines (cadvisor, node_exporter, etc.) where the agent's node is the metric's node.
1 parent d57f318 commit d8a49d3

1 file changed

Lines changed: 0 additions & 8 deletions

File tree

charts/amazon-cloudwatch-observability/templates/linux/_otel-container-insights-config.tpl

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -859,19 +859,11 @@ service:
859859
{{- if or .Values.otelContainerInsights.serviceMonitor.enabled .Values.otelContainerInsights.podMonitor.enabled }}
860860
metrics/cw_k8s_ci_v0_prometheuscr:
861861
receivers: [prometheus/cw_k8s_ci_v0_prometheuscr]
862-
# Node enrichment: set_node_name/promote_node_name stamp
863-
# resource.attributes["k8s.node.name"] = the scraping agent's own node
864-
# (${env:K8S_NODE_NAME}), unconditionally. The chart does NOT emit a target_node
865-
# label. To verify per-node allocation end-to-end, add a target_node relabeling
866-
# to your ServiceMonitor/PodMonitor (from the target's node metadata) and check
867-
# that target_node == k8s.node.name.
868862
processors:
869863
- filter/cw_k8s_ci_v0_scrape_metadata
870864
- metricstarttime/cw_k8s_ci_v0
871865
- transform/cw_k8s_ci_v0_set_cluster_name
872866
- transform/cw_k8s_ci_v0_set_scope_prometheuscr
873-
- transform/cw_k8s_ci_v0_set_node_name
874-
- transform/cw_k8s_ci_v0_promote_node_name
875867
- resourcedetection/cw_k8s_ci_v0
876868
- batch/cw_k8s_ci_v0_metrics_dest
877869
exporters:

0 commit comments

Comments
 (0)