-
Notifications
You must be signed in to change notification settings - Fork 273
Description
Summary
Prometheus query tools (query_prometheus, and related metadata tools) return 403 Forbidden when mcp-grafana calls the datasource proxy endpoint. The same Grafana service account token and the same URL return 200 when called via curl or wget from a pod in the same cluster. Only requests made by the mcp-grafana process get 403.
Environment
mcp-grafana: Running in Kubernetes, authenticated via GRAFANA_SERVICE_ACCOUNT_TOKEN
Grafana: Behind HTTPS; service account has datasource read/query permissions; dashboards and datasource list work (200)
Endpoint used by MCP: POST /api/datasources/uid//resources/api/v1/query
Steps to reproduce
Deploy mcp-grafana with a Grafana service account token that has datasources:read and datasources:query (or equivalent).
Use an MCP client to call query_prometheus (or list_prometheus_label_names, etc.) with a valid datasource UID.
Observe: Grafana returns 403 Forbidden for the request to /api/datasources/uid//resources/api/v1/query.
So the failure is specific to how mcp-grafana performs the request (or how Grafana treats that client), not the token or the URL.
What we verified
list_datasources, search_dashboards, and other non-proxy tools work.
Same token and same proxy URL from a laptop (curl) and from a pod in the same cluster (wget) return 200 or 422 (e.g. bad query params), never 403.
Only the mcp-grafana process receives 403 for the resources proxy call.
Possible cause
We suspect this may be related to grafana/grafana#51163 (403 on provisioned datasources via the resources proxy). The MCP server might be hitting a code path or sending headers that trigger the same (or a related) permission check in Grafana.