Skip to content

Commit c7d2fb2

Browse files
nunnatsanunnatsa
committed
Add runbook for the HCODICTWithNoSupportedArchitecture alert
Signed-off-by: Nahshon Unna-Tsameret <nunnatsa@redhat.com>
1 parent 0c5c8b9 commit c7d2fb2

1 file changed

Lines changed: 211 additions & 0 deletions

File tree

‎docs/runbooks/HCODICTWithNoSupportedArchitecture.md‎

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# HCODICTWithNoSupportedArchitecture
2+
3+
## Meaning
4+
5+
When running on a heterogeneous cluster, a cluster with nodes of different
6+
architectures, the DataImportCronTemplates (DICTs; also known as golden
7+
images), in the hyperconverged cluster operator (HCO) should be annotated with
8+
the `ssp.kubevirt.io/dict.architectures` annotation, where the value is the
9+
list of the architectures supported by the image, that is defined in each DICT.
10+
11+
For pre-defined DICTs, this annotation is already set, but for custom DICTs
12+
(user defined DICTs), this annotation must be set by the user in the
13+
HyperConverged custom resource (CR).
14+
15+
For each DICT, if the annotation does not include any architecture that is
16+
supported by the cluster (which mean, there is no node in the cluster that with
17+
any of the architectures listed in the DICT annotation), Then HCO will trigger
18+
the `HCODICTWithNoSupportedArchitecture` alert for this specific DICT.
19+
20+
## Impact
21+
22+
When this alert is triggered, it means that the DICT is not supported by any of
23+
the nodes in the cluster. HCO will not populate the SSP CR with this DICT, and
24+
so this golden image will not be available for use in the cluster.
25+
26+
## Diagnosis
27+
28+
Read the HyperConverged CR:
29+
30+
```bash
31+
# Get the namespace of the HyperConverged CR
32+
$ NAMESPACE="$(kubectl get hyperconverged -A --no-headers | awk '{print $1}')"
33+
34+
#Read the HyperConverged CR
35+
$ kubectl get hyperconverged -n "${NAMESPACE}" -o yaml
36+
```
37+
38+
There are a few fields in the HyperConverged CR status that can be used to
39+
diagnose this issue:
40+
41+
1. The `status.nodeInfo.workloadsArchitectures` shows the list of architectures
42+
supported by the cluster.
43+
2. The `status.dataImportCronTemplates` field shows the list of DICTs that are
44+
defined in the cluster.
45+
1. For each DICT, in this list, check the
46+
`ssp.kubevirt.io/dict.architectures` annotation. Unlike the annotation
47+
in the spec field, this annotation contain only the architectures that
48+
are supported by the image **and** byt the cluster.
49+
50+
If the annotation is missing, then it was not set in the source DICT.
51+
If the annotation is empty, then there is no architecture supported by
52+
the image and by the cluster.
53+
2. In this case, the status field of this DICT will include the
54+
`conditions` field, with the `Deployed` condition set to `False`, and
55+
the `reason` field set to `UnsupportedArchitectures`.
56+
3. The DICT's `status.workloadsArchitectures` field shows the list of
57+
architectures supported by the image, as was set in the
58+
`ssp.kubevirt.io/dict.architectures` annotation in the source DICT.
59+
60+
If this field is empty, it means that the `ssp.kubevirt.io/dict.architectures`
61+
annotation was not set in the source DICT.
62+
63+
### Example
64+
65+
```yaml
66+
apiVersion: hco.kubevirt.io/v1beta1
67+
kind: HyperConverged
68+
...
69+
status:
70+
...
71+
dataImportCronTemplates:
72+
- metadata:
73+
annotations:
74+
ssp.kubevirt.io/dict.architectures: ""
75+
name: my-image
76+
spec:
77+
...
78+
status:
79+
conditions:
80+
- message: DataImportCronTemplate has no supported architectures for the current
81+
cluster
82+
reason: UnsupportedArchitectures
83+
status: "False"
84+
type: Deployed
85+
originalSupportedArchitectures: someUnsupportedArch,otherUnsupportedArch
86+
```
87+
88+
## Mitigation
89+
90+
### Pre-defined DataImportCronTemplates
91+
92+
These DICTs are not defined in the HyperConverged CR, but are defined in the
93+
HCO itself.
94+
95+
All pre-defined DICTs are annotated with the `ssp.kubevirt.io/dict.architectures`
96+
annotation, and all of them supports the `amd64`, `arm64`, and `s390x`
97+
architectures. In the unlikely case that the cluster does not support any of
98+
these architectures, there is no way to use these pre-defined DICTs in the
99+
cluster.
100+
101+
To mitigate this issue, (if adding supported nodes to the cluster is not an
102+
option), you can either:
103+
104+
1. Disable the pre-defined DICTs in the HyperConverged CR, to turn this alert
105+
off:
106+
1. Find the DICT(s) you want to disable, in the HyperConverged `status.dataImportCronTemplates`
107+
field, as described
108+
[above](#diagnosis).
109+
2. Add the DICT to the `spec.dataImportCronTemplates` field in the
110+
HyperConverged CR. Add the `dataimportcrontemplate.kubevirt.io/enable`
111+
annotation with the value `false` to the DICT. Only the DICT name and
112+
the annotation are required, in this case
113+
114+
For example, to disable the `centos-stream10-image-cron` DICT:
115+
```yaml
116+
apiVersion: hco.kubevirt.io/v1beta1
117+
kind: HyperConverged
118+
metadata:
119+
name: kubevirt-hyperconverged
120+
spec:
121+
dataImportCronTemplates:
122+
- metadata:
123+
name: centos-stream10-image-cron
124+
annotations:
125+
dataimportcrontemplate.kubevirt.io/enable: 'false'
126+
```
127+
2. If you have the self-built desired image, that is supported by the nodes in
128+
the cluster, you can modify the pre-defined DICT to use your image, adding
129+
the DICT to the `spec.dataImportCronTemplates` field in the HyperConverged
130+
CR, and modify its `spec.source.registry` field.
131+
132+
> Tip: you can find the pre-defined DICTs in HyperConverged CR `status.dataImportCronTemplates`
133+
> field, as described [above](#diagnosis). Then you can copy the DICT from
134+
> there, and modify it in the HyperConverged CR
135+
> `spec.dataImportCronTemplates` field.
136+
137+
Don't forget to set the `ssp.kubevirt.io/dict.architectures` annotation to
138+
include all the architectures supported by your image.
139+
140+
In this case, you'll need to add all the fields of the DICT.
141+
142+
For example:
143+
```yaml
144+
apiVersion: hco.kubevirt.io/v1beta1
145+
kind: HyperConverged
146+
metadata:
147+
name: kubevirt-hyperconverged
148+
spec:
149+
dataImportCronTemplates:
150+
- metadata:
151+
annotations:
152+
cdi.kubevirt.io/storage.bind.immediate.requested: "true"
153+
ssp.kubevirt.io/dict.architectures: arch1,arch2
154+
name: centos-stream10-image-cron
155+
spec:
156+
garbageCollect: Outdated
157+
managedDataSource: centos-stream10
158+
schedule: "0 */12 * * *"
159+
template:
160+
spec:
161+
source:
162+
registry:
163+
url: docker://your-registry/your-image:latest
164+
storage:
165+
resources:
166+
requests:
167+
storage: 10Gi
168+
```
169+
170+
### User-defined DataImportCronTemplates
171+
172+
User-defined DICTs are defined in the HyperConverged CR, in the
173+
`spec.dataImportCronTemplates` field.
174+
175+
First, check what architectures are supported by the image. You can use the
176+
following command:
177+
178+
```bash
179+
$ podman manifest inspect your-registry/your-image:latest
180+
```
181+
182+
See here for
183+
the [podman manifest inspect documentation](https://docs.podman.io/en/latest/markdown/podman-manifest-inspect.1.html).
184+
185+
If the image is multi architecture manifest (fat manifest), it will include the
186+
`manifests` field, which is a list of architectures supported by the image. If
187+
the image is not a multi architecture manifest, you will need to find out what
188+
is its architecture.
189+
190+
Then, check that the `ssp.kubevirt.io/dict.architectures` annotation is set,
191+
and with the correct value. If not, Add or fix the annotation. The format of
192+
the annotation is a comma-separated list of architectures; e.g., `amd64,arm64,s390x`.
193+
194+
If the image does not support any of the architectures supported by the
195+
cluster, you will need to either rebuild the image for one or more of
196+
the architectures supported by the cluster, or remove the DICT from the
197+
HyperConverged CR.
198+
199+
Find some more information about building multi-architecture images, see the
200+
[podman documentation](https://docs.podman.io/en/latest/markdown/podman-manifest-create.1.html).
201+
202+
<!--DS: If you cannot resolve the issue, log in to the
203+
link:https://access.redhat.com[Customer Portal] and open a support case,
204+
attaching the artifacts gathered during the diagnosis procedure.-->
205+
<!--USstart-->
206+
If you cannot resolve the issue, see the following resources:
207+
208+
- [OKD Help](https://okd.io/docs/community/help/)
209+
- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization)
210+
211+
<!--USend-->

0 commit comments

Comments
 (0)