Skip to content

403 Forbidden on Prometheus/Loki query tools (datasource resources proxy) while same token works for /api/ds/query and from curl #524

@terryng

Description

@terryng

Summary
Prometheus query tools (query_prometheus, and related metadata tools) return 403 Forbidden when mcp-grafana calls the datasource proxy endpoint. The same Grafana service account token and the same URL return 200 when called via curl or wget from a pod in the same cluster. Only requests made by the mcp-grafana process get 403.

Environment
mcp-grafana: Running in Kubernetes, authenticated via GRAFANA_SERVICE_ACCOUNT_TOKEN
Grafana: Behind HTTPS; service account has datasource read/query permissions; dashboards and datasource list work (200)
Endpoint used by MCP: POST /api/datasources/uid//resources/api/v1/query
Steps to reproduce
Deploy mcp-grafana with a Grafana service account token that has datasources:read and datasources:query (or equivalent).
Use an MCP client to call query_prometheus (or list_prometheus_label_names, etc.) with a valid datasource UID.
Observe: Grafana returns 403 Forbidden for the request to /api/datasources/uid//resources/api/v1/query.
So the failure is specific to how mcp-grafana performs the request (or how Grafana treats that client), not the token or the URL.

What we verified
list_datasources, search_dashboards, and other non-proxy tools work.
Same token and same proxy URL from a laptop (curl) and from a pod in the same cluster (wget) return 200 or 422 (e.g. bad query params), never 403.
Only the mcp-grafana process receives 403 for the resources proxy call.
Possible cause
We suspect this may be related to grafana/grafana#51163 (403 on provisioned datasources via the resources proxy). The MCP server might be hitting a code path or sending headers that trigger the same (or a related) permission check in Grafana.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions