Can't get Grafana to connect to prom-label-proxy, confusion

[jellyfish-headphones]

Apologies in advance for naivete. Not a Prometheus expert by any means.

I have a Grafana deployment that is used by multiple clients who are sorted into organizations so they can see dashboards in only their namespace.

I'd like to allow these users to edit their dashboards (they currently cannot) - but to also disallow them to change the namespace their dashboard is grabbing data from. This is where I am hoping prom-label-proxy will be useful. I have it deployed in the monitoring namespace, where Grafana and Prometheus also reside.

However, I am struggling to get this tool working properly with Grafana.

Here is what I have in prom-label-proxy.yaml (this is just a basic configuration, I'm just trying to get something going here):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prom-label-proxy
  namespace: monitoring
  labels:
    app: prom-label-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prom-label-proxy
  template:
    metadata:
      labels:
        app: prom-label-proxy
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
      - name: prom-label-proxy
        image: quay.io/prometheuscommunity/prom-label-proxy:v0.7.0
        args:
        - --label=namespace
        - --upstream=http://monitoring-monitoring-kube-prometheus.monitoring.svc:9090
        - --insecure-listen-address=0.0.0.0:8080
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: prom-label-proxy
  namespace: monitoring
spec:
  selector:
    app: prom-label-proxy
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

I listed the prom-label-proxy service as the datasource for Grafana for the Organizations I would like to apply this to.

grafana:
  values:
    datasources:
      datasources.yaml:
        apiVersion: 1
        datasources:
        - name: Alertmanager
          url: http://monitoring-monitoring-kube-alertmanager.monitoring.svc:9093
          type: alertmanager
          jsonData:
            implementation: 'prometheus'
        - name: Prometheus
          url: http://monitoring-monitoring-kube-prometheus.monitoring.svc:9090
          type: prometheus
          isDefault: true
          orgId: 1
        - name: Prometheus
          url: http://prom-label-proxy.monitoring.svc:8080
          type: prometheus
          orgId: 2 # <namespace-1>
        - name: Prometheus
          url: http://prom-label-proxy.monitoring.svc:8080
          type: prometheus
          orgId: 3 # <namespace-2>
        - name: Prometheus
          url: http://prom-label-proxy.monitoring.svc:8080
          type: prometheus
          orgId: 4 # <namespace-3>

However, when I go to these dashboards in the Grafana UI, a message pops up saying 404: page not found, and I see this in the response:

{
    "results": {
        "A": {
            "error": "ReadObject: expect { or , or } or n, but found \u0000, error found in #0 byte of ...||..., bigger context ...||...",
            "errorSource": "",
            "status": 502,
            "frames": [
                {
                    "schema": {
                        "refId": "A",
                        "meta": {
                            "typeVersion": [
                                0,
                                0
                            ],
                            "executedQueryString": "Expr: (\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_capacity_bytes{cluster=\"\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"<namespace-1>\", persistentvolumeclaim=\"\"})))\n  -\n  sum without(instance, node) (topk(1, (kubelet_volume_stats_available_bytes{cluster=\"\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"<namespace-1>\", persistentvolumeclaim=\"\"})))\n)\n\nStep: 20m0s"
                        },
                        "fields": []
                    },
                    "data": {
                        "values": []
                    }
                }
            ]
        },
        "B": {
            "error": "ReadObject: expect { or , or } or n, but found \u0000, error found in #0 byte of ...||..., bigger context ...||...",
            "errorSource": "",
            "status": 502,
            "frames": [
                {
                    "schema": {
                        "refId": "B",
                        "meta": {
                            "typeVersion": [
                                0,
                                0
                            ],
                            "executedQueryString": "Expr: sum without(instance, node) (topk(1, (kubelet_volume_stats_available_bytes{cluster=\"\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"<namespace-1>\", persistentvolumeclaim=\"\"})))\n\nStep: 20m0s"
                        },
                        "fields": []
                    },
                    "data": {
                        "values": []
                    }
                }
            ]
        }
    }
}

I see these logs in the prom-label-proxy container:

http: proxy error: read tcp <prom-label-proxy-ip>:39120-><prometheus-service-ip>:9090: read: connection reset by peer 

I have network policies that allow all intra-namespace traffic in the monitoring namespace, so prom-label-proxy should be able to reach Prometheus.

Trying to reach Prometheus via wget within the prom-label-proxy container:

/ $ wget -qO- "http://monitoring-monitoring-kube-prometheus.monitoring.svc:9090/api/v1/query?query=up"
wget: error getting response: Connection reset by peer
/ $ wget -qO- "http://monitoring-monitoring-kube-prometheus.monitoring.svc:9090/api/v1/query"
wget: error getting response: Connection reset by peer
/ $ wget -qO- "http://monitoring-monitoring-kube-prometheus.monitoring.svc:9090"
wget: error getting response: Connection reset by peer

I guess I'm just confused, perhaps a little out of my depth here. Any help would be very appreciated.

[jellyfish-headphones]

Hello? Is this thing on?

[squat]

Is it possible that your kube-prometheus stack has deployed NetworkPolicy resources protecting your Prometheus pods and your prom-label-proxy pods are not in the allow list?

[jellyfish-headphones]

Is it possible that your kube-prometheus stack has deployed NetworkPolicy resources protecting your Prometheus pods and your prom-label-proxy pods are not in the allow list?

Thanks for the response. All traffic within the monitoring namespace is permitted, and I've even explicitly allowed prom-label-proxy to reach Prometheus as an added measure, but to no avail.

I do have istio-proxy sidecars in each pod in my cluster, which provides mTLS. I wonder if this is disrupting the Grafana -> prom-label-proxy -> Prometheus traffic. I'm really not sure though.