Skip to content

Commit 96836bb

Browse files
sradcosradco
committed
Build: (4ed2d64) Merge pull request #356 from Ronilerr/updateRunbooks
Adding runbooks for virt handler and virt api new alerts and fix low ready virt controllers count runbook
1 parent 0175ec6 commit 96836bb

8 files changed

Lines changed: 349 additions & 4 deletions

‎runbooks/LowReadyVirtAPICount.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# LowReadyVirtAPICount
2+
3+
## Meaning
4+
5+
This alert fires when one or more `virt-api` pods are running, but not
6+
all of them have been in a `Ready` state for the last 10 minutes.
7+
8+
The `virt-api` serves the KubeVirt API. The deployment typically runs two
9+
replicas for high-availability.
10+
11+
## Impact
12+
13+
Reduced capacity or redundancy for the KubeVirt API. If the condition
14+
persists, it can lead to the `NoReadyVirtAPI` alert and API unavailability.
15+
16+
## Diagnosis
17+
18+
1. Set the `NAMESPACE` environment variable:
19+
20+
```bash
21+
$ export NAMESPACE="$(kubectl get kubevirt -A -o jsonpath='{.items[].metadata.namespace}')"
22+
```
23+
24+
2. Check the status of the `virt-api` pods:
25+
26+
```bash
27+
$ kubectl -n $NAMESPACE get pods -l kubevirt.io=virt-api
28+
```
29+
30+
3. Check the `virt-api` deployment and its events:
31+
32+
```bash
33+
$ kubectl -n $NAMESPACE describe deploy virt-api
34+
```
35+
36+
4. Check pod readiness and conditions for non-ready pods:
37+
38+
```bash
39+
$ kubectl -n $NAMESPACE get pods -l kubevirt.io=virt-api -o wide
40+
$ kubectl -n $NAMESPACE describe pod -l kubevirt.io=virt-api
41+
```
42+
43+
5. If pods are in `CrashLoopBackOff` or to inspect runtime failures, check
44+
`virt-api` pod logs and look for errors:
45+
46+
```bash
47+
$ kubectl -n $NAMESPACE logs -l kubevirt.io=virt-api
48+
```
49+
50+
6. Check for node issues, such as a `NotReady` state:
51+
52+
```bash
53+
$ kubectl get nodes
54+
```
55+
56+
## Mitigation
57+
58+
Identify why some `virt-api` pods are not ready (e.g. failed readiness probe,
59+
resource pressure, image pull issues) and resolve the underlying cause.
60+
61+
<!--DS: If you cannot resolve the issue, log in to the
62+
link:https://access.redhat.com[Customer Portal] and open a support case,
63+
attaching the artifacts gathered during the diagnosis procedure.-->
64+
<!--USstart-->
65+
If you cannot resolve the issue, see the following resources:
66+
67+
- [OKD Help](https://okd.io/docs/community/help/)
68+
- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
69+
<!--USend-->

‎runbooks/LowReadyVirtControllersCount.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
## Meaning
44

5-
This alert fires when one or more `virt-controller` pods are running, but none
6-
of these pods has been in the `Ready` state for the last 5 minutes.
5+
This alert fires when one or more `virt-controller` pods are running, but not
6+
all of them have been in a `Ready` state for the last 10 minutes.
77

88
A `virt-controller` device monitors the custom resource definitions (CRDs) of a
99
virtual machine instance (VMI) and manages the associated pods. The device

‎runbooks/LowReadyVirtHandlerCount.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# LowReadyVirtHandlerCount
2+
3+
## Meaning
4+
5+
This alert fires when one or more `virt-handler` pods are running, but not
6+
all of them have been in a `Ready` state for the last 10 minutes.
7+
8+
The `virt-handler` runs on every node that can schedule VMIs (as a
9+
DaemonSet). Each node typically has one `virt-handler` pod.
10+
11+
## Impact
12+
13+
Some nodes may have a running but not ready `virt-handler`. VMIs running on those
14+
nodes might not be fully managed (e.g. domain updates, network or storage
15+
changes). If the condition persists, it can lead to the `NoReadyVirtHandler`
16+
alert for affected nodes.
17+
18+
## Diagnosis
19+
20+
1. Set the `NAMESPACE` environment variable:
21+
22+
```bash
23+
$ export NAMESPACE="$(kubectl get kubevirt -A -o custom-columns="":.metadata.namespace)"
24+
```
25+
26+
2. Check the status of the `virt-handler` pods:
27+
28+
```bash
29+
$ kubectl -n $NAMESPACE get pods -l kubevirt.io=virt-handler -o wide
30+
```
31+
32+
3. For pods that are running but not ready, inspect pod conditions and events:
33+
34+
```bash
35+
$ kubectl -n $NAMESPACE describe pod -l kubevirt.io=virt-handler
36+
```
37+
38+
4. If pods are in `CrashLoopBackOff` or to inspect runtime failures, check
39+
non-ready `virt-handler` pod logs and look for errors:
40+
41+
```bash
42+
$ kubectl -n $NAMESPACE logs -l kubevirt.io=virt-handler
43+
```
44+
45+
Note: With multiple pods (DaemonSet), `-l` streams one pod's logs; use a
46+
pod name from step 2 to target a specific non-ready pod.
47+
48+
5. If needed, check the `virt-handler` DaemonSet and its events:
49+
50+
```bash
51+
$ kubectl -n $NAMESPACE describe daemonset virt-handler
52+
```
53+
54+
6. Check for node issues on nodes where `virt-handler` is not ready:
55+
56+
```bash
57+
$ kubectl get nodes
58+
```
59+
60+
## Mitigation
61+
62+
Identify why some `virt-handler` pods are not ready (e.g. failed readiness
63+
probe, resource pressure, node issues) and resolve the underlying cause so
64+
all schedulable nodes have a ready `virt-handler`.
65+
66+
<!--DS: If you cannot resolve the issue, log in to the
67+
link:https://access.redhat.com[Customer Portal] and open a support case,
68+
attaching the artifacts gathered during the diagnosis procedure.-->
69+
<!--USstart-->
70+
If you cannot resolve the issue, see the following resources:
71+
72+
- [OKD Help](https://okd.io/docs/community/help/)
73+
- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
74+
<!--USend-->

‎runbooks/LowReadyVirtOperatorsCount.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
## Meaning
44

5-
This alert fires when one or more `virt-operator` pods are running, but none of
6-
these pods has been in a `Ready` state for the last 10 minutes.
5+
This alert fires when one or more `virt-operator` pods are running, but not
6+
all of them have been in a `Ready` state for the last 10 minutes.
77

88
The `virt-operator` is the first Operator to start in a cluster. The
99
`virt-operator` deployment has a default replica of two `virt-operator` pods.

‎runbooks/NoReadyVirtAPI.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# NoReadyVirtAPI
2+
3+
## Meaning
4+
5+
This alert fires when no `virt-api` pod in a `Ready` state has been detected
6+
for 10 minutes.
7+
8+
The `virt-api` serves the KubeVirt API. Without a ready `virt-api`, API
9+
requests for virtual machines and other KubeVirt resources cannot be served.
10+
11+
## Impact
12+
13+
KubeVirt API is effectively unavailable. Users and controllers cannot perform
14+
API operations such as creating, updating, or deleting virtual machine
15+
instances (VMIs) or other KubeVirt resources.
16+
17+
## Diagnosis
18+
19+
1. Set the `NAMESPACE` environment variable:
20+
21+
```bash
22+
$ export NAMESPACE="$(kubectl get kubevirt -A -o custom-columns="":.metadata.namespace)"
23+
```
24+
25+
2. Check the status of the `virt-api` pods:
26+
27+
```bash
28+
$ kubectl -n $NAMESPACE get pods -l kubevirt.io=virt-api
29+
```
30+
31+
3. Check the `virt-api` deployment and events:
32+
33+
```bash
34+
$ kubectl -n $NAMESPACE describe deploy virt-api
35+
```
36+
37+
4. Review logs of any `virt-api` pod that is running but not ready:
38+
39+
```bash
40+
$ kubectl -n $NAMESPACE logs <virt-api-pod-name> --previous
41+
$ kubectl -n $NAMESPACE logs <virt-api-pod-name>
42+
```
43+
44+
5. Check for node issues:
45+
46+
```bash
47+
$ kubectl get nodes
48+
```
49+
50+
## Mitigation
51+
52+
Identify the root cause (e.g. all replicas crashing, readiness probe failures,
53+
node or resource issues) and restore at least one ready `virt-api` pod.
54+
55+
<!--DS: If you cannot resolve the issue, log in to the
56+
link:https://access.redhat.com[Customer Portal] and open a support case,
57+
attaching the artifacts gathered during the diagnosis procedure.-->
58+
<!--USstart-->
59+
If you cannot resolve the issue, see the following resources:
60+
61+
- [OKD Help](https://okd.io/docs/community/help/)
62+
- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
63+
<!--USend-->

‎runbooks/NoReadyVirtHandler.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# NoReadyVirtHandler
2+
3+
## Meaning
4+
5+
This alert fires when no `virt-handler` pod in a `Ready` state has been
6+
detected for 10 minutes.
7+
8+
The `virt-handler` runs on every node that can schedule VMIs (as a
9+
DaemonSet). It is responsible for domain lifecycle and node-level operations
10+
for virtual machine instances.
11+
12+
## Impact
13+
14+
No node has a ready `virt-handler`. Virtual machine instances cannot be
15+
properly managed: domain updates, migrations, and node-level operations will
16+
fail or be delayed until at least one `virt-handler` becomes ready.
17+
18+
## Diagnosis
19+
20+
1. Set the `NAMESPACE` environment variable:
21+
22+
```bash
23+
$ export NAMESPACE="$(kubectl get kubevirt -A -o custom-columns="":.metadata.namespace)"
24+
```
25+
26+
2. Check the status of the `virt-handler` DaemonSet and pods:
27+
28+
```bash
29+
$ kubectl -n $NAMESPACE get daemonset virt-handler
30+
$ kubectl -n $NAMESPACE get pods -l kubevirt.io=virt-handler -o wide
31+
```
32+
33+
3. Check DaemonSet and pod events:
34+
35+
```bash
36+
$ kubectl -n $NAMESPACE describe daemonset virt-handler
37+
$ kubectl -n $NAMESPACE describe pod -l kubevirt.io=virt-handler
38+
```
39+
40+
4. Review logs of any running but not ready `virt-handler` pod:
41+
42+
```bash
43+
$ kubectl -n $NAMESPACE logs <virt-handler-pod-name> --previous
44+
$ kubectl -n $NAMESPACE logs <virt-handler-pod-name>
45+
```
46+
47+
5. Check for cluster-wide node or scheduling issues:
48+
49+
```bash
50+
$ kubectl get nodes
51+
```
52+
53+
## Mitigation
54+
55+
Identify the root cause (e.g. DaemonSet not scheduling, all pods crashing or
56+
failing readiness, node or image issues) and restore at least one ready
57+
`virt-handler` pod on a schedulable node.
58+
59+
<!--DS: If you cannot resolve the issue, log in to the
60+
link:https://access.redhat.com[Customer Portal] and open a support case,
61+
attaching the artifacts gathered during the diagnosis procedure.-->
62+
<!--USstart-->
63+
If you cannot resolve the issue, see the following resources:
64+
65+
- [OKD Help](https://okd.io/docs/community/help/)
66+
- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
67+
<!--USend-->

‎runbooks/VirtHandlerDown.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# VirtHandlerDown
2+
3+
## Meaning
4+
5+
This alert fires when all `virt-handler` pods are down (no `virt-handler` pod
6+
has been reported up for 10 minutes).
7+
8+
The `virt-handler` runs on every node that can schedule VMIs. It is
9+
responsible for domain lifecycle, network configuration, and other
10+
node-level operations for virtual machine instances.
11+
12+
## Impact
13+
14+
Virtual machine instances (VMIs) on affected nodes cannot be managed properly.
15+
New VMIs may not start on nodes without a running `virt-handler`, and
16+
existing VMIs may not receive updates or clean shutdowns.
17+
18+
## Diagnosis
19+
20+
1. Set the `NAMESPACE` environment variable:
21+
22+
```bash
23+
$ export NAMESPACE="$(kubectl get kubevirt -A -o custom-columns="":.metadata.namespace)"
24+
```
25+
26+
2. Check the status of the `virt-handler` DaemonSet and pods:
27+
28+
```bash
29+
$ kubectl -n $NAMESPACE get daemonset virt-handler -o yaml
30+
$ kubectl -n $NAMESPACE get pods -l kubevirt.io=virt-handler
31+
```
32+
33+
3. Check DaemonSet events and pod status:
34+
35+
```bash
36+
$ kubectl -n $NAMESPACE describe daemonset virt-handler
37+
$ kubectl -n $NAMESPACE describe pod -l kubevirt.io=virt-handler
38+
```
39+
40+
4. Check for node issues (e.g. nodes not ready or taints):
41+
42+
```bash
43+
$ kubectl get nodes
44+
```
45+
46+
5. If any `virt-handler` pod exists, review its logs:
47+
48+
```bash
49+
$ kubectl -n $NAMESPACE logs <virt-handler-pod-name> --previous
50+
$ kubectl -n $NAMESPACE logs <virt-handler-pod-name>
51+
```
52+
53+
## Mitigation
54+
55+
Identify why `virt-handler` pods are down (e.g. DaemonSet not scheduling, pods
56+
crashing, node issues, image pull failures) and restore the DaemonSet so
57+
`virt-handler` runs on schedulable nodes.
58+
59+
<!--DS: If you cannot resolve the issue, log in to the
60+
link:https://access.redhat.com[Customer Portal] and open a support case,
61+
attaching the artifacts gathered during the diagnosis procedure.-->
62+
<!--USstart-->
63+
If you cannot resolve the issue, see the following resources:
64+
65+
- [OKD Help](https://okd.io/docs/community/help/)
66+
- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
67+
<!--USend-->

‎runbooks_index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
- [HCOGoldenImageWithNoArchitectureAnnotation.md](runbooks/HCOGoldenImageWithNoArchitectureAnnotation.md)
44
- [KubeVirtCRModified.md](runbooks/KubeVirtCRModified.md)
5+
- [NoReadyVirtAPI.md](runbooks/NoReadyVirtAPI.md)
56
- [NoReadyVirtController.md](runbooks/NoReadyVirtController.md)
67
- [OutdatedVirtualMachineInstanceWorkloads.md](runbooks/OutdatedVirtualMachineInstanceWorkloads.md)
78
- [CnaoDown.md](runbooks/CnaoDown.md)
@@ -14,6 +15,7 @@
1415
- [CDINoDefaultStorageClass.md](runbooks/CDINoDefaultStorageClass.md)
1516
- [VirtHandlerDaemonSetRolloutFailing.md](runbooks/VirtHandlerDaemonSetRolloutFailing.md)
1617
- [VirtControllerDown.md](runbooks/VirtControllerDown.md)
18+
- [LowReadyVirtHandlerCount.md](runbooks/LowReadyVirtHandlerCount.md)
1719
- [KubemacpoolDown.md](runbooks/KubemacpoolDown.md)
1820
- [HAControlPlaneDown.md](runbooks/HAControlPlaneDown.md)
1921
- [NoLeadingVirtOperator.md](runbooks/NoLeadingVirtOperator.md)
@@ -23,8 +25,11 @@
2325
- [SSPFailingToReconcile.md](runbooks/SSPFailingToReconcile.md)
2426
- [NodeNetworkInterfaceDown.md](runbooks/NodeNetworkInterfaceDown.md)
2527
- [CDIStorageProfilesIncomplete.md](runbooks/CDIStorageProfilesIncomplete.md)
28+
- [VirtHandlerDown.md](runbooks/VirtHandlerDown.md)
29+
- [LowReadyVirtAPICount.md](runbooks/LowReadyVirtAPICount.md)
2630
- [LowVirtOperatorCount.md](runbooks/LowVirtOperatorCount.md)
2731
- [HPPNotReady.md](runbooks/HPPNotReady.md)
32+
- [NoReadyVirtHandler.md](runbooks/NoReadyVirtHandler.md)
2833
- [VirtAPIDown.md](runbooks/VirtAPIDown.md)
2934
- [VMStorageClassWarning.md](runbooks/VMStorageClassWarning.md)
3035
- [LowVirtControllersCount.md](runbooks/LowVirtControllersCount.md)

0 commit comments

Comments
 (0)