Description
Hello! I apologize in advance if this is more of a question for the Istio community, but since this is a rather niche problem directly related to PLP, I figure I would ask this community because there is a chance someone here has some insight or experience with this.
I am trying to leverage prom-label-proxy to restrict my Grafana users to querying their namespace only, while allowing them to query Prometheus and edit/create Grafana dashboards. However, I'm having trouble with getting this to work under Istio.
For security compliance, we must use istio-proxy sidecars to facilitate STRICT TLS/mTLS for all cluster traffic. I can only get prom-label-proxy working if I circumvent security and set mTLS mode to PERMISSIVE. When using STRICT mode, the Grafana -> prom-label-proxy -> Prometheus traffic returns upstream connect error or disconnect/reset before headers. reset reason: connection termination.
All NetworkPolicies present allow this traffic to occur. Same with AuthorizationPolicies. Independently, Grafana can reach both prom-label-proxy and Prometheus with curl, prom-label-proxy can reach and query Prometheus with wget. But when, from Grafana, PLP is queried, which goes upstream to Prometheus, I'm ultimately met with the upstream connect error.
I can share config if needed, but nothing is necessarily misconfigured-- everything works correctly when istio is not there to facilitate STRICT mTLS.
If anyone has any experience with working with PLP and Istio I would greatly appreciate your insight. Thank you so much.