Add runbooks for GuestVCPUQueueHigh alerts

This PR adds runbooks for GuestVCPUQueueHighWarning
and GuestVCPUQueueHighCritical alerts.

Signed-off-by: Shirly Radco <sradco@redhat.com>

Author: sradco

docs/runbooks/GuestVCPUQueueHighCritical.md

@@ -0,0 +1,48 @@
+# GuestVCPUQueueHighCritical
+
+## Meaning
+A VirtualMachineInstance (VMI) reported a **guest CPU run‑queue length greater than 20** runnable or
+uninterruptible threads within the last scrape window (120s), indicating severe CPU contention.
+
+## Impact
+* Sustained backlog; high latency and throughput degradation are likely.
+* Risk of timeouts, watchdog resets, or I/O amplification.
+
+## Diagnosis
+Follow the below steps with extra focus on:
+* **Duration** – How long is the queue > 20
+* **Host saturation** – if node CPU is also > 90 %, migrate other VMs or mark the node as unschedulable
+   so that no new Pods/VMs are placed thereon it.
+
+1. **Confirm queue length**
+   ```promql
+   kubevirt_vmi_guest_vcpu_queue{namespace="$NS",name="$VM"}
+   ```
+2. **Check host CPU usage**
+   ```promql
+   rate(kubevirt_vmi_cpu_usage_seconds_total{namespace="$NS",name="$VM"}[2m])
+   ```
+3. **Inspect guest processes**
+   `virtctl console <vm>` → `top -H` or `pidstat -u 1`
+4. **Verify vCPU allocation**
+   ```bash
+   oc get vmi $VM -ojsonpath='{.spec.domain.cpu}'
+   ```
+
+## Mitigation
+| Horizon | Action |
+|---------|--------|
+| Immediate | **Prioritise**: live‑migrate VM; hot‑plug vCPUs; stop or throttle runaway processes. |
+| Short term | Permanently raise vCPU allocation or split workload across VMs. |
+| Long term | Adjust scheduling/placement policies; add autoscaling tied to queue length. |
+
+<!--USstart-->
+If you cannot resolve the issue, see the following resources:
+
+- [OKD Help](https://www.okd.io/help/)
+- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
+<!--USend-->
+
+<!--DS: If you cannot resolve the issue, log in to the
+[Customer Portal](https://access.redhat.com) and open a support case,
+attaching the artifacts gathered during the diagnosis procedure.-->

docs/runbooks/GuestVCPUQueueHighWarning.md

@@ -0,0 +1,44 @@
+# GuestVCPUQueueHighWarning
+
+## Meaning
+A VirtualMachineInstance (VMI) reported a **guest CPU run‑queue length greater than 10** runnable or \
+uninterruptible threads within the most‑recent scrape window (120s).
+The run‑queue length is derived from `guest_load_1m – vCPU_count`.
+
+## Impact
+* Moderate CPU contention inside the guest; latency may spike but workload still progresses.
+* Early signal that the VM might need additional vCPUs or that a short‑lived process is causing bursts.
+
+## Diagnosis
+1. **Confirm queue length**
+   ```promql
+   kubevirt_vmi_guest_vcpu_queue{namespace="$NS",name="$VM"}
+   ```
+2. **Check host CPU usage**
+   ```promql
+   rate(kubevirt_vmi_cpu_usage_seconds_total{namespace="$NS",name="$VM"}[2m])
+   ```
+3. **Inspect guest processes**
+   `virtctl console <vm>` → `top -H` or `pidstat -u 1`
+4. **Verify vCPU allocation**
+   ```bash
+   oc get vmi $VM -ojsonpath='{.spec.domain.cpu}'
+   ```
+
+## Mitigation
+| Horizon | Action |
+|---------|--------|
+| Immediate | Optionally live‑migrate the VM to a quieter node or throttle noisy processes. |
+| Short term | Hot‑plug / increase vCPUs; tune application thread pools. |
+| Long term | Implement horizontal scaling (HPA/KEDA, VMReplicaSet); review placement rules. |
+
+<!--USstart-->
+If you cannot resolve the issue, see the following resources:
+
+- [OKD Help](https://www.okd.io/help/)
+- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
+<!--USend-->
+
+<!--DS: If you cannot resolve the issue, log in to the
+[Customer Portal](https://access.redhat.com) and open a support case,
+attaching the artifacts gathered during the diagnosis procedure.-->