PrometheusRule Alerting Rules

[ev-shindin]

Add a PrometheusRule CRD with baseline alerting rules.

Alerts:

groups:
- name: wva.rules
  rules:
  - alert: WVAHighErrorRate
    expr: rate(wva_errors_total[5m]) > 0.1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "WVA error rate elevated"

  - alert: WVAOptimizationLoopStalled
    expr: rate(wva_models_processed_total[10m]) == 0
    for: 15m
    labels:
      severity: critical
    annotations:
      summary: "WVA optimization loop has stopped processing models"

  - alert: WVAMetricsCollectionFailing
    expr: rate(wva_metrics_collection_errors_total[5m]) > 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "WVA metrics collection failing frequently"

  - alert: WVAGPUResourceExhausted
    expr: wva_available_gpus == 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "No GPUs available for WVA scaling"

  - alert: WVAReplicaScalingThrashing
    expr: rate(wva_replica_scaling_total[10m]) > 2
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "WVA scaling decisions changing rapidly (possible thrashing)"

Implementation:

• Add PrometheusRule CRD in config/prometheus/
• Include in Helm chart as optional (enabled via values.yaml flag)
• Thresholds should be configurable in Helm values

Acceptance Criteria:

• PrometheusRule CRD deploys with make deploy and Helm chart
• Alerts fire correctly when conditions are met (manual verification)
• Helm chart allows enabling/disabling and threshold overrides

[hemanth9398]

I'm interested on working on this

[ev-shindin]

@hemanth9398 : this issue have pre-requistics. You would wait before preceding issues from this epic will be closed.