How to scrap metrics from multiple containers of same pod exposing metrics at different paths and different ports ?

[sjethvani]

I have django application pod having 3/4 containers. Out of them 2/3 are exposing metrics (one being uswgi exporter , another being django exporter) . I want to use annotation autodiscovery feature for scrapping metrics & visualizing it in grafana.

I want to know how can I use k8s.grafana.com/metrics... annotations on pod to scrap metrics from multiple containers of it exposing metrics on different paths (like /uwsgi/metrics , /django/metrics) & ports (like 9117 , 9118) etc ?

[sjethvani]

Update
I used Grafana cloud's Fleet management remote configuration to scrape prometheus metrics from my second container of pod .

So my pod having uwsgi container exposing prometheus metrics on port 9117 is being scrapped through grafana's annotation auto discovery settings (i.e k8s.grafana.com/scrape , k8s.grafana.com/metrics on POD) ,
while another container in that same pod which is exposing django metrics (using django-prometheus) on port 9118 is being scrapped through fleet management remote config

something like

// Discovery component to find all pods
discovery.kubernetes "all_pods" {
	role = "pod"
}

// Relabel component to filter to your specific pod and namespace
discovery.relabel "filter_mypods_pods_set_namespace_label" {
	targets = discovery.kubernetes.all_pods.targets

	// Filter to specific namespace (can be used for testing purpose)
	//rule {
	//source_labels = ["__meta_kubernetes_namespace"]
	//regex         = "mydjangometrics"
	//action        = "keep"
	//}

	// Filter to specific pod (by name pattern or label)
	rule {
		source_labels = ["__meta_kubernetes_pod_name"]
		regex         = "(mypodnameprefix.*|someotherpodnameprefix.*)"
		action        = "keep"
	}

	// Set the metrics path
	rule {
		target_label = "__metrics_path__"
		replacement  = "<somemetricpath> (for-ex /prometheus/metrics)"
		action       = "replace"
	}

	// Set namespace label
	rule {
		source_labels = ["__meta_kubernetes_namespace"]
		target_label  = "namespace"
		action        = "replace"
	}
}

// prometheus scrape component to collect the metrics
prometheus.scrape "scrape_mypod_django_metrics" {
	targets    = discovery.relabel.filter_mypods_pods_set_namespace_label.output
	forward_to = [prometheus.remote_write.grafana_cloud_prometheus_metrics.receiver]
}

// Remote write component to send metrics to Grafana Cloud
prometheus.remote_write "grafana_cloud_prometheus_metrics" {
	endpoint {
		url = "https://prometheus-prod-<someid>-prod-<region>.grafana.net/api/prom/push"

		basic_auth {
			username = "<prom_username>"
			password = sys.env("GCLOUD_RW_API_KEY")
		}
	}
	external_labels = {
		"cluster"          = sys.env("CLUSTER_NAME"),
		"k8s_cluster_name" = sys.env("CLUSTER_NAME"),
	}
}