feat(tools): add Prometheus discovery tools (list_metrics, metadata, targets)

Three new read-only tools adapted from pab1it0/prometheus-mcp-server,
all backed by the standard Prometheus HTTP API so they work against
self-hosted Prom and Google Managed Prometheus alike (with one caveat,
noted below).

list_prom_metrics — list metric names known to the server with an
optional case-insensitive substring filter. Resolves the 'no series
returned' diagnostic loop where the agent can't tell whether the
metric name is wrong, the labels are wrong, or the workload is silent.
Backed by /api/v1/label/__name__/values.

get_prom_metric_metadata — return type/help/unit for a metric or all
metrics. Resolves rate/gauge/histogram confusion before writing
PromQL (e.g. counter needs rate(), gauge does not, histogram needs
histogram_quantile()). Backed by /api/v1/metadata.

get_prom_targets — scrape target health (up/down) with lastError on
failures. Detects the 'Prom isn't scraping the workload' failure mode
before blaming the workload. Backed by /api/v1/targets. NOTE: Google
Managed Prometheus gateways do not expose /api/v1/targets; the tool
returns an empty list there and the summary surfaces that explicitly
rather than treating it as an error.

PromClient interface gains listMetrics/metricMetadata/targets methods
(implemented in RealPromClient with native fetch, plus matching
fixture support). Three new evidence keys: prom_metrics,
prom_metric_metadata, prom_targets.

Planner adds a PROM_DISCOVERY_TOOLS group selected when the alert has
metric/rate/latency signals (matches the existing hasMetricSignal
heuristic), keeping tool budget tight for non-metric alerts.

Author: polo871209

README.md

@@ -116,3 +116,4 @@ See [`AGENTS.md`](./AGENTS.md) for development conventions.
 - [`googleapis/gcloud-mcp`](https://github.com/googleapis/gcloud-mcp) — tool-design reference.
 - [`JuliusBrussee/caveman`](https://github.com/JuliusBrussee/caveman) — prompt-compression reference.
 - [`k8sgpt-ai/k8sgpt`](https://github.com/k8sgpt-ai/k8sgpt) — deterministic k8s analyzer + failure-catalog reference.
+- [`pab1it0/prometheus-mcp-server`](https://github.com/pab1it0/prometheus-mcp-server) — Prometheus discovery-tool reference.

packages/clients/src/index.ts

@@ -35,6 +35,15 @@ const THROWING: ClientRegistry = {
     async alertRules() {
       throw new Error("Prom client not configured");
     },
+    async listMetrics() {
+      throw new Error("Prom client not configured");
+    },
+    async metricMetadata() {
+      throw new Error("Prom client not configured");
+    },
+    async targets() {
+      throw new Error("Prom client not configured");
+    },
     uiUrl() {
       return undefined;
     },

packages/clients/src/prom/fixture.ts

@@ -1,11 +1,27 @@
-import type { PromInstantResult, PromRuleState, PromSeriesResult } from "@opsremedy/core/types";
-import type { PromClient, PromInstantQuery, PromRangeQuery } from "../types.ts";
+import type {
+  PromInstantResult,
+  PromMetricMetadata,
+  PromRuleState,
+  PromSeriesResult,
+  PromTarget,
+} from "@opsremedy/core/types";
+import type {
+  PromClient,
+  PromInstantQuery,
+  PromMetadataQuery,
+  PromMetricsListQuery,
+  PromRangeQuery,
+  PromTargetsQuery,
+} from "../types.ts";
 
 export interface FixturePromPayload {
   /** Keyed by exact PromQL query string. */
   instant?: Record<string, PromInstantResult>;
   range?: Record<string, PromSeriesResult>;
   alertRules?: PromRuleState[];
+  metrics?: string[];
+  metadata?: PromMetricMetadata[];
+  targets?: PromTarget[];
 }
 
 const EMPTY_INSTANT: PromInstantResult = { resultType: "vector", series: [] };
@@ -31,6 +47,25 @@ export class FixturePromClient implements PromClient {
     return this.data.alertRules ?? [];
   }
 
+  async listMetrics(q: PromMetricsListQuery): Promise<string[]> {
+    let names = this.data.metrics ?? [];
+    if (q.contains) {
+      const needle = q.contains.toLowerCase();
+      names = names.filter((n) => n.toLowerCase().includes(needle));
+    }
+    return names.slice(0, q.limit ?? 200);
+  }
+
+  async metricMetadata(q: PromMetadataQuery): Promise<PromMetricMetadata[]> {
+    const all = this.data.metadata ?? [];
+    return q.metric ? all.filter((m) => m.metric === q.metric) : all;
+  }
+
+  async targets(q: PromTargetsQuery): Promise<PromTarget[]> {
+    const all = this.data.targets ?? [];
+    return q.job ? all.filter((t) => t.job === q.job) : all;
+  }
+
   uiUrl(): undefined {
     return undefined;
   }

packages/clients/src/prom/real.ts

@@ -1,5 +1,18 @@
-import type { PromInstantResult, PromRuleState, PromSeriesResult } from "@opsremedy/core/types";
-import type { PromClient, PromInstantQuery, PromRangeQuery } from "../types.ts";
+import type {
+  PromInstantResult,
+  PromMetricMetadata,
+  PromRuleState,
+  PromSeriesResult,
+  PromTarget,
+} from "@opsremedy/core/types";
+import type {
+  PromClient,
+  PromInstantQuery,
+  PromMetadataQuery,
+  PromMetricsListQuery,
+  PromRangeQuery,
+  PromTargetsQuery,
+} from "../types.ts";
 
 export interface RealPromClientOptions {
   baseUrl: string;
@@ -42,6 +55,30 @@ interface PromRulesData {
   }>;
 }
 
+/**
+ * `/api/v1/metadata` returns either Record<metric, MetadataItem[]> (vanilla
+ * Prom) or a flat array (some downstreams). We normalise to the array form.
+ */
+type PromMetadataRaw =
+  | Record<string, Array<{ type?: string; help?: string; unit?: string }>>
+  | Array<{ metric?: string; type?: string; help?: string; unit?: string }>;
+
+interface PromTargetRaw {
+  discoveredLabels?: Record<string, string>;
+  labels?: Record<string, string>;
+  scrapePool?: string;
+  scrapeUrl?: string;
+  globalUrl?: string;
+  lastError?: string;
+  lastScrape?: string;
+  health?: "up" | "down" | "unknown";
+}
+
+interface PromTargetsData {
+  activeTargets?: PromTargetRaw[];
+  droppedTargets?: PromTargetRaw[];
+}
+
 /**
  * Prometheus HTTP API client. Uses native fetch.
  * Supports bearer token or HTTP basic auth.
@@ -106,6 +143,51 @@ export class RealPromClient implements PromClient {
     return out;
   }
 
+  async listMetrics(q: PromMetricsListQuery): Promise<string[]> {
+    // Prom + GMP both expose label values via /api/v1/label/<label>/values.
+    const data = await this.get<string[]>("/api/v1/label/__name__/values", new URLSearchParams(), q.signal);
+    let names = Array.isArray(data) ? data : [];
+    if (q.contains) {
+      const needle = q.contains.toLowerCase();
+      names = names.filter((n) => n.toLowerCase().includes(needle));
+    }
+    const limit = q.limit ?? 200;
+    return names.slice(0, limit);
+  }
+
+  async metricMetadata(q: PromMetadataQuery): Promise<PromMetricMetadata[]> {
+    const params = new URLSearchParams();
+    if (q.metric) params.set("metric", q.metric);
+    if (q.perMetricLimit !== undefined) params.set("limit_per_metric", String(q.perMetricLimit));
+    const data = await this.get<PromMetadataRaw>("/api/v1/metadata", params, q.signal);
+    return normaliseMetadata(data);
+  }
+
+  async targets(q: PromTargetsQuery): Promise<PromTarget[]> {
+    const params = new URLSearchParams();
+    params.set("state", q.state ?? "active");
+    const data = await this.get<PromTargetsData>("/api/v1/targets", params, q.signal);
+    const sources: PromTargetRaw[] = [];
+    if (data.activeTargets) sources.push(...data.activeTargets);
+    if (q.state === "dropped" || q.state === "any") {
+      if (data.droppedTargets) sources.push(...data.droppedTargets);
+    }
+    const out: PromTarget[] = sources.map((t) => {
+      const labels = t.labels ?? {};
+      const target: PromTarget = {
+        job: labels.job ?? t.scrapePool ?? "",
+        instance: labels.instance ?? "",
+        health: t.health ?? "unknown",
+        labels,
+      };
+      if (t.scrapeUrl !== undefined) target.scrapeUrl = t.scrapeUrl;
+      if (t.lastError !== undefined && t.lastError.length > 0) target.lastError = t.lastError;
+      if (t.lastScrape !== undefined) target.lastScrape = t.lastScrape;
+      return target;
+    });
+    return q.job ? out.filter((t) => t.job === q.job) : out;
+  }
+
   uiUrl(kind: "graph" | "alerts", query?: string): string {
     if (kind === "alerts") return `${this.baseUrl}/alerts`;
     if (!query) return `${this.baseUrl}/graph`;
@@ -141,6 +223,33 @@ export class RealPromClient implements PromClient {
   }
 }
 
+function normaliseMetadata(raw: PromMetadataRaw): PromMetricMetadata[] {
+  const out: PromMetricMetadata[] = [];
+  if (Array.isArray(raw)) {
+    for (const item of raw) {
+      if (!item.metric) continue;
+      out.push({
+        metric: item.metric,
+        type: (item.type ?? "unknown") as PromMetricMetadata["type"],
+        help: item.help ?? "",
+        ...(item.unit !== undefined && item.unit.length > 0 && { unit: item.unit }),
+      });
+    }
+    return out;
+  }
+  for (const [metric, items] of Object.entries(raw)) {
+    const first = items?.[0];
+    if (!first) continue;
+    out.push({
+      metric,
+      type: (first.type ?? "unknown") as PromMetricMetadata["type"],
+      help: first.help ?? "",
+      ...(first.unit !== undefined && first.unit.length > 0 && { unit: first.unit }),
+    });
+  }
+  return out;
+}
+
 function pickInstantSample(s: { value?: [number, string]; values?: Array<[number, string]> }): {
   timestamp: number;
   value: number;

packages/clients/src/types.ts

@@ -3,8 +3,10 @@ import type {
   LogEntry,
   PodSummary,
   PromInstantResult,
+  PromMetricMetadata,
   PromRuleState,
   PromSeriesResult,
+  PromTarget,
   ServiceDep,
   TraceSummary,
 } from "@opsremedy/core/types";
@@ -50,10 +52,40 @@ export interface PromRangeQuery {
   signal?: AbortSignal;
 }
 
+export interface PromMetricsListQuery {
+  /** Optional case-insensitive substring filter applied to metric names. */
+  contains?: string;
+  /** Cap returned names; defaults to 200. */
+  limit?: number;
+  signal?: AbortSignal;
+}
+
+export interface PromMetadataQuery {
+  /** When set, return only metadata for this metric name. */
+  metric?: string;
+  /** Per-metric metadata limit returned by Prom (defaults to 1). */
+  perMetricLimit?: number;
+  signal?: AbortSignal;
+}
+
+export interface PromTargetsQuery {
+  /** When set, return only targets whose `job` label matches. */
+  job?: string;
+  /** When "active"|"dropped"|"any" — defaults to "active". */
+  state?: "active" | "dropped" | "any";
+  signal?: AbortSignal;
+}
+
 export interface PromClient {
   instant(q: PromInstantQuery): Promise<PromInstantResult>;
   range(q: PromRangeQuery): Promise<PromSeriesResult>;
   alertRules(signal?: AbortSignal): Promise<PromRuleState[]>;
+  /** List metric names known to the server (`/api/v1/label/__name__/values`). */
+  listMetrics(q: PromMetricsListQuery): Promise<string[]>;
+  /** Per-metric metadata: type/help/unit (`/api/v1/metadata`). */
+  metricMetadata(q: PromMetadataQuery): Promise<PromMetricMetadata[]>;
+  /** Scrape target health (`/api/v1/targets`). */
+  targets(q: PromTargetsQuery): Promise<PromTarget[]>;
   /** UI URL for a query in /graph; falls back to the bare /graph or /alerts. */
   uiUrl(kind: "graph" | "alerts", query?: string): string | undefined;
 }

packages/core/src/planner.ts

@@ -6,6 +6,9 @@ export const ALL_TOOL_NAMES = [
   "query_prom_instant",
   "query_prom_range",
   "get_prom_alert_rules",
+  "list_prom_metrics",
+  "get_prom_metric_metadata",
+  "get_prom_targets",
   "query_jaeger_traces",
   "get_jaeger_service_deps",
   "k8s_cluster_info",
@@ -34,6 +37,11 @@ const K8S_TOOLS: ToolName[] = [
   "k8s_triage_pod",
 ];
 const TRACE_TOOLS: ToolName[] = ["query_jaeger_traces", "get_jaeger_service_deps"];
+const PROM_DISCOVERY_TOOLS: ToolName[] = [
+  "list_prom_metrics",
+  "get_prom_metric_metadata",
+  "get_prom_targets",
+];
 
 export function planGatherTools(alert: Alert, loop: number, rerouteHint?: string): GatherPlanAudit {
   if (loop > 0 || rerouteHint) return selectAll("reroute can need any source", loop);
@@ -50,6 +58,11 @@ export function planGatherTools(alert: Alert, loop: number, rerouteHint?: string
     for (const tool of TRACE_TOOLS) selected.set(tool, "alert contains service/latency/dependency signal");
   }
 
+  if (hasMetricSignal(text)) {
+    for (const tool of PROM_DISCOVERY_TOOLS)
+      selected.set(tool, "metric/rate alert benefits from Prom discovery (names, types, scrape health)");
+  }
+
   if (hasInfraSignal(text)) selected.set("query_prom_instant", "alert asks for current infra state");
 
   return buildPlan(selected, loop);

packages/core/src/types.ts

@@ -54,6 +54,32 @@ export interface PromRuleState {
   lastTransition?: number;
 }
 
+/**
+ * Per-metric metadata from Prometheus `/api/v1/metadata`. Same shape across
+ * vanilla Prom + Google Managed Prometheus. Lets the LLM tell counter from
+ * gauge from histogram before writing rate() over a gauge.
+ */
+export interface PromMetricMetadata {
+  metric: string;
+  type: "counter" | "gauge" | "histogram" | "summary" | "untyped" | "info" | "stateset" | "unknown";
+  help: string;
+  unit?: string;
+}
+
+/**
+ * Scrape target health from Prometheus `/api/v1/targets`. Single entry per
+ * (job, instance). `lastError` is empty when the target is healthy.
+ */
+export interface PromTarget {
+  job: string;
+  instance: string;
+  health: "up" | "down" | "unknown";
+  scrapeUrl?: string;
+  lastError?: string;
+  lastScrape?: string;
+  labels: Record<string, string>;
+}
+
 export interface TraceSummary {
   traceId: string;
   rootService: string;
@@ -189,6 +215,9 @@ export interface Evidence {
   prom_instant?: Record<string, PromInstantResult>;
   prom_series?: Record<string, PromSeriesResult>;
   prom_alert_rules?: PromRuleState[];
+  prom_metrics?: string[];
+  prom_metric_metadata?: PromMetricMetadata[];
+  prom_targets?: PromTarget[];
 
   jaeger_traces?: TraceSummary[];
   jaeger_service_deps?: ServiceDep[];
@@ -312,6 +341,9 @@ export const ALL_EVIDENCE_KEYS = [
   "prom_instant",
   "prom_series",
   "prom_alert_rules",
+  "prom_metrics",
+  "prom_metric_metadata",
+  "prom_targets",
   "jaeger_traces",
   "jaeger_service_deps",
   "k8s_pods",

packages/tools/README.md

@@ -1,6 +1,6 @@
 # @opsremedy/tools
 
-The 14 read-only tools the gather agent calls. Each is a pi-mono `AgentTool`
+The 17 read-only tools the gather agent calls. Each is a pi-mono `AgentTool`
 factory that takes the `InvestigationContext` and returns a tool with a
 TypeBox parameter schema.
 
@@ -13,6 +13,9 @@ TypeBox parameter schema.
 | `query_prom_instant` | PromQL at a single timestamp |
 | `query_prom_range` | PromQL over a window (range query) |
 | `get_prom_alert_rules` | List alerting rules + state |
+| `list_prom_metrics` | Discover metric names by substring (resolves "no series returned" before guessing labels) |
+| `get_prom_metric_metadata` | Metric type/help/unit (counter vs gauge vs histogram) |
+| `get_prom_targets` | Scrape target health (up/down + lastError); empty under Google Managed Prometheus |
 | `query_jaeger_traces` | Find traces for a service |
 | `get_jaeger_service_deps` | Upstream/downstream edges |
 | `k8s_cluster_info` | Cluster-level facts: node ready count + namespace list |