Skip to content

Commit 4479f70

Browse files
sudoaloksudoalok
committed
feat(monitoring): ship default EPP Prometheus alerting rules
Add a PrometheusRule under deploy/components/monitoring with a baseline set of EPP alerts (availability/errors + EPP self-health), wired into the monitoring kustomization next to the existing ServiceMonitor. Uses metric names stable today; latency SLO and flow-control saturation alerts follow in separate PRs. Also set ruleSelectorNilUsesHelmValues=false in the dev kube-prometheus-stack install so the rule loads like the ServiceMonitors. Part of llm-d#1635. Signed-off-by: Alok Behera <alokbeherak061@gmail.com>
1 parent ff37a55 commit 4479f70

3 files changed

Lines changed: 96 additions & 0 deletions

File tree

‎deploy/components/monitoring/epp-alerting-rules.yaml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
apiVersion: monitoring.coreos.com/v1
2+
kind: PrometheusRule
3+
metadata:
4+
name: epp-alerting-rules
5+
labels:
6+
app: epp-metrics
7+
spec:
8+
groups:
9+
# ---------------------------------------------------------------------------
10+
# Availability and errors
11+
# ---------------------------------------------------------------------------
12+
- name: epp.availability
13+
rules:
14+
- alert: EPPHighErrorRatio
15+
# Fraction of EPP requests returning an error. 0/0 yields no value, so this
16+
# stays silent when there is no traffic. If your workload is low-volume and
17+
# you see flapping at startup, add a request-rate floor with `and ... > N`.
18+
expr: |
19+
sum(rate(llm_d_router_epp_request_error_total[5m]))
20+
/ sum(rate(llm_d_router_epp_request_total[5m]))
21+
> 0.05
22+
for: 10m
23+
labels:
24+
severity: warning
25+
annotations:
26+
summary: "EPP error ratio above 5%"
27+
description: "EPP request error ratio has been above 5% for 10m (current {{ $value | humanizePercentage }})."
28+
- alert: EPPCriticalErrorRatio
29+
expr: |
30+
sum(rate(llm_d_router_epp_request_error_total[5m]))
31+
/ sum(rate(llm_d_router_epp_request_total[5m]))
32+
> 0.20
33+
for: 5m
34+
labels:
35+
severity: critical
36+
annotations:
37+
summary: "EPP error ratio above 20%"
38+
description: "EPP request error ratio has been above 20% for 5m (current {{ $value | humanizePercentage }}). The inference path is likely degraded."
39+
- alert: EPPNoReadyEndpoints
40+
# The pool has zero ready endpoints, so the EPP cannot route any request.
41+
expr: llm_d_router_epp_ready_endpoints == 0
42+
for: 2m
43+
labels:
44+
severity: critical
45+
annotations:
46+
summary: "EPP pool {{ $labels.name }} has no ready endpoints"
47+
description: "llm_d_router_epp_ready_endpoints has been 0 for pool {{ $labels.name }} for 2m. The pool is unroutable."
48+
- alert: EPPMetricsAbsent
49+
# ready_endpoints is emitted continuously by the pool collector regardless of
50+
# traffic, so its disappearance means the EPP is down or the ServiceMonitor
51+
# stopped scraping it (a silent observability gap).
52+
expr: absent(llm_d_router_epp_ready_endpoints)
53+
for: 5m
54+
labels:
55+
severity: critical
56+
annotations:
57+
summary: "EPP metrics are not being scraped"
58+
description: "No llm_d_router_epp_ready_endpoints samples for 5m. The EPP is down or its ServiceMonitor is broken."
59+
60+
# ---------------------------------------------------------------------------
61+
# EPP self-health
62+
# ---------------------------------------------------------------------------
63+
- name: epp.selfhealth
64+
rules:
65+
- alert: EPPDataLayerPollErrors
66+
# A data source failed to poll, so scheduling may be running on stale
67+
# endpoint state. This is otherwise invisible.
68+
expr: sum by (source_type) (rate(llm_d_router_epp_datalayer_poll_errors_total[5m])) > 0
69+
for: 10m
70+
labels:
71+
severity: warning
72+
annotations:
73+
summary: "EPP data layer poll errors for source {{ $labels.source_type }}"
74+
description: "llm_d_router_epp_datalayer_poll_errors_total is increasing for source {{ $labels.source_type }}. Scheduling may be using stale data."
75+
- alert: EPPDataLayerExtractErrors
76+
expr: sum by (source_type, extractor_type) (rate(llm_d_router_epp_datalayer_extract_errors_total[5m])) > 0
77+
for: 10m
78+
labels:
79+
severity: warning
80+
annotations:
81+
summary: "EPP data layer extract errors ({{ $labels.source_type }}/{{ $labels.extractor_type }})"
82+
description: "llm_d_router_epp_datalayer_extract_errors_total is increasing for {{ $labels.source_type }}/{{ $labels.extractor_type }}. Scheduling may be using stale data."
83+
# ext_proc stream metrics are opt-in (EPP flag --enable-grpc-stream-metrics).
84+
# When the flag is unset these series are absent and the alert below simply
85+
# never fires. "Abnormal" excludes OK and Canceled (the latter is a normal
86+
# client disconnect); tune the matcher for your environment.
87+
- alert: EPPExtProcStreamErrors
88+
expr: sum by (code) (rate(llm_d_router_epp_extproc_streams_total{code!~"OK|Canceled"}[5m])) > 0
89+
for: 10m
90+
labels:
91+
severity: warning
92+
annotations:
93+
summary: "EPP ext_proc streams terminating with {{ $labels.code }}"
94+
description: "ext_proc gRPC streams are completing with status {{ $labels.code }}. Envoy<->EPP request processing may be failing."

‎deploy/components/monitoring/kustomization.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,5 @@ kind: Kustomization
33

44
resources:
55
- epp-service-monitor.yaml
6+
- epp-alerting-rules.yaml
67
- vllm-sim-pod-monitor.yaml

‎scripts/kind-dev-env.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -451,6 +451,7 @@ if [ "${PROM_ENABLED}" == "true" ]; then
451451
--set kubeScheduler.enabled=false \
452452
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
453453
--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
454+
--set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false \
454455
--set prometheus.prometheusSpec.resources.requests.memory=512Mi \
455456
--set prometheus.prometheusSpec.resources.limits.memory=1Gi \
456457
--set prometheus.service.type=NodePort \

0 commit comments

Comments
 (0)