Pod metrics scrape - from multiple container ports #2142
Description
Requirement - Able to scrape pod metrics from multiple ports.
**Scrape jobs with autodiscover enabled, should be able to
- monitoring metrics on multiple ports/apps
- should be able to enable/disable depending on requirements.**
Current customer issue - pod with 2 containers, one of them is the app and the other is an envoy sidecar (using c-advisor). Want k8s-monitoring to scrape both of them. (although - the envoy sidecar is only there for egress connections). The metrics endpoint and path are as follow
app
port: 9100
path: /metrics
envoy
port: 9901
path: /stats/prometheus
and the configuration on k8s-monitoring is using the annotationAutodiscovery feature to discover and scrape the metrics.
annotationAutodiscovery:
enabled: true
annotations:
scrape: k8s.grafana.com/scrape
job: k8s.grafana.com/job
instance: k8s.grafana.com/instance
metricsPath: k8s.grafana.com/metrics.path
metricsPortName: k8s.grafana.com/metrics.portName
metricsPortNumber: k8s.grafana.com/metrics.portNumber
metricsScheme: k8s.grafana.com/metrics.scheme
metricsContainer: k8s.grafana.com/metrics.container
Suggestion provided to customer to use indexed annotations - as .1 and .2
annotations:
k8s.grafana.com/metrics.container: jupiter-core
k8s.grafana.com/metrics.container.1: envoy-sidecar
k8s.grafana.com/metrics.path: /metrics
k8s.grafana.com/metrics.path.1: /stats/prometheus
k8s.grafana.com/metrics.portNumber: "9100"
k8s.grafana.com/metrics.portNumber.1: "9901"
k8s.grafana.com/scrape: "true"
k8s.grafana.com/scrape.1: "true"
Alloy is discovering all indexed annotations:
__meta_kubernetes_pod_annotation_k8s_grafana_com_scrape_1 = "true"
__meta_kubernetes_pod_annotation_k8s_grafana_com_metrics_container_1 = "envoy-sidecar"
__meta_kubernetes_pod_annotation_k8s_grafana_com_metrics_portNumber_1 = "9901"
__meta_kubernetes_pod_annotation_k8s_grafana_com_metrics_path_1 = "/stats/prometheus"
It discovers them but doesn't process the suffix (.1) to create separate scrape jobs.
it's only creating ONE target:
address = "<IP_address>:9100" (only port 9100)
container = "jupiter-core" (only first container)
metrics_path = "/metrics" (only first path)