|
| 1 | +# AI Service Metrics (Prometheus ServiceMonitor Discovery) |
| 2 | + |
| 3 | +**Cluster:** `EKS / p5.48xlarge / NVIDIA-H100-80GB-HBM3` |
| 4 | +**Generated:** 2026-03-24 13:46:00 UTC |
| 5 | +**Kubernetes Version:** v1.35 |
| 6 | +**Platform:** linux/amd64 |
| 7 | + |
| 8 | +--- |
| 9 | + |
| 10 | +Demonstrates that Prometheus discovers and collects metrics from AI workloads |
| 11 | +that expose them in Prometheus exposition format, using the ServiceMonitor CRD |
| 12 | +for automatic target discovery. |
| 13 | + |
| 14 | +## vLLM Inference Workload |
| 15 | + |
| 16 | +A vLLM inference server (serving Qwen/Qwen3-0.6B) exposes application-level |
| 17 | +metrics in Prometheus format at `:8000/metrics`. A ServiceMonitor enables |
| 18 | +Prometheus to automatically discover and scrape the endpoint. |
| 19 | + |
| 20 | +**vLLM workload pod** |
| 21 | +``` |
| 22 | +$ kubectl get pods -n vllm-metrics-test -o wide |
| 23 | +NAME READY STATUS RESTARTS AGE |
| 24 | +vllm-server 1/1 Running 0 3m |
| 25 | +``` |
| 26 | + |
| 27 | +**vLLM metrics endpoint (sampled)** |
| 28 | +``` |
| 29 | +$ kubectl exec -n vllm-metrics-test vllm-server -- python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8000/metrics').read().decode())" | grep vllm: |
| 30 | +vllm:num_requests_running{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0 |
| 31 | +vllm:num_requests_waiting{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0 |
| 32 | +vllm:kv_cache_usage_perc{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0 |
| 33 | +vllm:prefix_cache_queries_total{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0 |
| 34 | +vllm:prefix_cache_hits_total{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0 |
| 35 | +vllm:engine_sleep_state{engine="0",model_name="Qwen/Qwen3-0.6B",sleep_state="awake"} 1.0 |
| 36 | +vllm:estimated_flops_per_gpu_total{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0 |
| 37 | +``` |
| 38 | + |
| 39 | +## ServiceMonitor |
| 40 | + |
| 41 | +**ServiceMonitor for vLLM** |
| 42 | +``` |
| 43 | +$ kubectl get servicemonitor vllm-inference -n vllm-metrics-test -o yaml |
| 44 | +apiVersion: monitoring.coreos.com/v1 |
| 45 | +kind: ServiceMonitor |
| 46 | +metadata: |
| 47 | + labels: |
| 48 | + release: prometheus |
| 49 | + name: vllm-inference |
| 50 | + namespace: vllm-metrics-test |
| 51 | +spec: |
| 52 | + endpoints: |
| 53 | + - interval: 15s |
| 54 | + path: /metrics |
| 55 | + port: http |
| 56 | + selector: |
| 57 | + matchLabels: |
| 58 | + app: vllm-inference |
| 59 | +``` |
| 60 | + |
| 61 | +**Service endpoint** |
| 62 | +``` |
| 63 | +$ kubectl get endpoints vllm-inference -n vllm-metrics-test |
| 64 | +NAME ENDPOINTS AGE |
| 65 | +vllm-inference 10.0.170.78:8000 3m |
| 66 | +``` |
| 67 | + |
| 68 | +## Prometheus Target Discovery |
| 69 | + |
| 70 | +Prometheus automatically discovers the vLLM workload as a scrape target via |
| 71 | +the ServiceMonitor and actively collects metrics. |
| 72 | + |
| 73 | +**Prometheus scrape target (active)** |
| 74 | +``` |
| 75 | +$ kubectl exec -n monitoring prometheus-kube-prometheus-prometheus-0 -- \ |
| 76 | + wget -qO- 'http://localhost:9090/api/v1/targets?state=active' | \ |
| 77 | + jq '.data.activeTargets[] | select(.labels.job=="vllm-inference")' |
| 78 | +{ |
| 79 | + "job": "vllm-inference", |
| 80 | + "endpoint": "http://10.0.170.78:8000/metrics", |
| 81 | + "health": "up", |
| 82 | + "lastScrape": "2026-03-24T13:46:50.899967845Z" |
| 83 | +} |
| 84 | +``` |
| 85 | + |
| 86 | +## vLLM Metrics in Prometheus |
| 87 | + |
| 88 | +Prometheus collects vLLM application-level metrics including request counts, |
| 89 | +KV cache usage, prefix cache hit rates, and GPU utilization estimates. |
| 90 | + |
| 91 | +**vLLM metrics queried from Prometheus** |
| 92 | +``` |
| 93 | +$ kubectl exec -n monitoring prometheus-kube-prometheus-prometheus-0 -- \ |
| 94 | + wget -qO- 'http://localhost:9090/api/v1/query?query={job="vllm-inference",__name__=~"vllm:.*"}' |
| 95 | +vllm:num_requests_running{model_name="Qwen/Qwen3-0.6B"} 0 |
| 96 | +vllm:num_requests_waiting{model_name="Qwen/Qwen3-0.6B"} 0 |
| 97 | +vllm:kv_cache_usage_perc{model_name="Qwen/Qwen3-0.6B"} 0 |
| 98 | +vllm:prefix_cache_queries_total{model_name="Qwen/Qwen3-0.6B"} 0 |
| 99 | +vllm:prefix_cache_hits_total{model_name="Qwen/Qwen3-0.6B"} 0 |
| 100 | +vllm:engine_sleep_state{model_name="Qwen/Qwen3-0.6B",sleep_state="awake"} 1 |
| 101 | +vllm:estimated_flops_per_gpu_total{model_name="Qwen/Qwen3-0.6B"} 0 |
| 102 | +vllm:estimated_read_bytes_per_gpu_total{model_name="Qwen/Qwen3-0.6B"} 0 |
| 103 | +``` |
| 104 | + |
| 105 | +**Result: PASS** — Prometheus discovers the vLLM inference workload via ServiceMonitor and actively scrapes its Prometheus-format metrics endpoint. Application-level AI metrics (request queue depth, KV cache usage, prefix cache hits, GPU FLOPS estimates) are collected and queryable. |
| 106 | + |
| 107 | +## Cleanup |
| 108 | + |
| 109 | +**Delete test namespace** |
| 110 | +``` |
| 111 | +$ kubectl delete ns vllm-metrics-test |
| 112 | +``` |
0 commit comments