Bug: Custom Query Parameters on Prometheus Datasource Have No Effect When Using Thanos

[KUMNISIP]

helm version - v3
kubectl version - 1.33.2
chart : kube-prometheus-stack
chart version - v72.3.1

Even though custom query parameters are configured for the Prometheus datasource, they do not appear to have any effect on queries that are routed through Thanos. The same queries, when executed directly against Prometheus (bypassing Thanos), work as expected with the custom parameters applied.

When configuring a Prometheus datasource in Grafana with custom query parameters, these parameters should be passed correctly to the Prometheus instance (or Thanos if used). The query should be modified as per the custom parameters, resulting in correct and expected data retrieval

Reproducing
Set up kube-prometheus-stack with Thanos as the Prometheus remote storage.
Configure a Prometheus datasource in Grafana, specifying custom query parameters (e.g., time range, filters, etc.).
Create a panel in Grafana that uses the Prometheus datasource with the custom parameters.
Observe that the custom parameters do not affect the query when routed through Thanos.

You can prove it by setting any label in the custom query parameters box. And then do a query against a metric that has that label value as well as other values ie
a namespace label for any metric "namespace=kube-prom" but you will still get metrics back from all namespaces

Basically if you do the customQueryParamaters setting in the thanos datasource in grafana, adding a label like we did has no effect

ArgoCD was used

The lines added to the .YAML were:

customQueryParameters: "environment=dev-k8s"

[GiedriusS]

Is there any way we could simplify the reproduction steps so that we wouldn't have to run Grafana or something else?

[KUMNISIP]

I don't think so, the only way you can prove it is via grafana

Personally id just:
create grafana and prometheus via docker-compose file
Create prometheus datasource in grafana
Add custom query param label
Confirm that queries aren't filtered by label put in step 3

[GiedriusS]

Could you elaborate on what you mean by " Add custom query param label"?

[KUMNISIP]

something like setting the options to "customQueryParameters: "environment=dev-url"

[KUMNISIP]

Update :

It would be under this section

Where it says "custom query parameters"