diff --git a/docs/runbooks/GuestVCPUQueueHighCritical.md b/docs/runbooks/GuestVCPUQueueHighCritical.md new file mode 100644 index 00000000..ac3cd62a --- /dev/null +++ b/docs/runbooks/GuestVCPUQueueHighCritical.md @@ -0,0 +1,51 @@ +# GuestVCPUQueueHighCritical + +## Meaning +A VirtualMachineInstance (VMI) reported a +**guest CPU run‑queue length greater than 20** runnable or +uninterruptible threads within the last scrape window (120s), +indicating severe CPU contention. + +## Impact +* Sustained backlog; high latency and throughput degradation are likely. +* Risk of timeouts, watchdog resets, or I/O amplification. + +## Diagnosis +Follow the below steps with extra focus on: +* **Duration** – How long is the queue > 20 +* **Host saturation** – if node CPU is also > 90 %, migrate other VMs or mark the + node as unschedulable so that no new Pods/VMs are placed thereon it. + +1. **Confirm queue length** + ```promql + kubevirt_vmi_guest_vcpu_queue{namespace="$NS",name="$VM"} + ``` +2. **Check host CPU usage** + ```promql + rate(kubevirt_vmi_cpu_usage_seconds_total{namespace="$NS",name="$VM"}[2m]) + ``` +3. **Inspect guest processes** + `virtctl console ` → `top -H` or `pidstat -u 1` +4. **Verify vCPU allocation** + ```bash + oc get vmi $VM -ojsonpath='{.spec.domain.cpu}' + ``` + +## Mitigation +| Horizon | Action | +|----------|------------------------------------------------------------------| +| Immediate| **Prioritise**: live-migrate VM; hot-plug vCPUs; stop or throttle hot threads. | +| Short term| Raise vCPU limit or split workload across additional VMs. | +| Long term| Adjust placement rules; add autoscaling tied to run-queue length.| + + + +If you cannot resolve the issue, see the following resources: + +- [OKD Help](https://www.okd.io/help/) +- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization) + + + diff --git a/docs/runbooks/GuestVCPUQueueHighWarning.md b/docs/runbooks/GuestVCPUQueueHighWarning.md new file mode 100644 index 00000000..42d87db6 --- /dev/null +++ b/docs/runbooks/GuestVCPUQueueHighWarning.md @@ -0,0 +1,47 @@ +# GuestVCPUQueueHighWarning + +## Meaning +A VirtualMachineInstance (VMI) reported a +**guest CPU run‑queue length greater than 10** runnable or +uninterruptible threads within the most‑recent scrape window (120s). +The run‑queue length is derived from `guest_load_1m – vCPU_count`. + +## Impact +* Moderate CPU contention inside the guest; + latency may spike but workload still progresses. +* Early signal that the VM might need additional vCPUs or + that a short‑lived process is causing bursts. + +## Diagnosis +1. **Confirm queue length** + ```promql + kubevirt_vmi_guest_vcpu_queue{namespace="$NS",name="$VM"} + ``` +2. **Check host CPU usage** + ```promql + rate(kubevirt_vmi_cpu_usage_seconds_total{namespace="$NS",name="$VM"}[2m]) + ``` +3. **Inspect guest processes** + `virtctl console ` → `top -H` or `pidstat -u 1` +4. **Verify vCPU allocation** + ```bash + oc get vmi $VM -ojsonpath='{.spec.domain.cpu}' + ``` + +## Mitigation +| Horizon | Action | +|---------|--------| +| Immediate | Optionally live‑migrate the VM to a quieter node or throttle noisy processes. | +| Short term | Hot‑plug / increase vCPUs; tune application thread pools. | +| Long term | Implement horizontal scaling (HPA/KEDA, VMReplicaSet); review placement rules. | + + +If you cannot resolve the issue, see the following resources: + +- [OKD Help](https://www.okd.io/help/) +- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization) + + +