TELCODOCS#2230: Coordinating reboots for configuration changes #91723
+51
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openshift-ci
bot
added
the
size/M
sr1kar99
commented
Apr 7, 2025
xenolinux
reviewed
Apr 7, 2025
sr1kar99
force-pushed
the
2230-controlled-reboots
branch
2 times, most recently
from
00e1010 to
d439288
Compare
xenolinux
reviewed
Apr 8, 2025
sr1kar99
force-pushed
the
2230-controlled-reboots
branch
from
d439288 to
735afd2
Compare
| @@ -15,6 +15,8 @@ include::modules/accessing-an-example-cluster-node-tuning-operator-specification | |||
|
|
|||
| include::modules/cluster-node-tuning-operator-default-profiles-set.adoc[leveloffset=+1] | |||
|
|
|||
| include::modules/ztp-coordinating-reboots-for-config-changes.adoc[leveloffset=+1] | |||
There was a problem hiding this comment.
Reboot section should come after
defer-application-of-tuning-changes_node-tuning-operator
in this document.
I am not sure if it needs to be a section, It can be a link to original section.
There was a problem hiding this comment.
Sure! I added a link to the original section in this module.
sr1kar99
force-pushed
the
2230-controlled-reboots
branch
from
735afd2 to
5112fd9
Compare
| You can use {cgu-operator-full} to coordinate reboots across a fleet of spoke clusters when configuration changes require a reboot. These configuration changes include updates to tuning profiles that modify kernel parameters or system behavior.{cgu-operator} ensures that only nodes with a degraded tuned profile, indicating a reboot is needed, are rebooted. Instead of rebooting nodes after each individual change, you can apply all configuration updates through policies and then trigger a single, coordinated reboot. | ||
|
|
||
| For more information about coordinated reboots, see xref:../../edge_computing/policygenerator_for_ztp/ztp-configuring-managed-clusters-policygenerator.adoc#ztp-coordinating-reboots-for-config-changes_ztp-configuring-managed-clusters-policygenerator[Coordinating reboots for configuration changes] |
There was a problem hiding this comment.
🤖 [error] OpenShiftAsciiDoc.NoXrefInModules: Do not include xrefs in modules, only assemblies.
sr1kar99
force-pushed
the
2230-controlled-reboots
branch
from
5112fd9 to
8563776
Compare
|
@sr1kar99: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| <3> The `machineConfigLabels` field is used to target the `worker-cnf` role. Configure a `MachineConfigPool` resource to ensure the profile is applied only to the correct nodes. | ||
|
|
||
| You can use {cgu-operator-full} to coordinate reboots across a fleet of spoke clusters when configuration changes require a reboot. These configuration changes include updates to tuning profiles that modify kernel parameters or system behavior.{cgu-operator} ensures that only nodes with a degraded tuned profile, indicating a reboot is needed, are rebooted. Instead of rebooting nodes after each individual change, you can apply all configuration updates through policies and then trigger a single, coordinated reboot. |
There was a problem hiding this comment.
I don't think this explanation is needed here.
I was thinking more like:
Note: You can use TALM to perform a controlled reboot across a fleet of cluster to apply a deferred tuning change[link]
Comment on lines
+11
to
+15
| The following types of configuration changes typically require a reboot: | ||
|
|
||
| * Updates to tuning profiles that modify kernel parameters or system behavior. | ||
| * Kubelet configuration updates. | ||
| * Node-level changes delivered through `MachineConfig`, such as `sysctl` settings or system service changes. |
There was a problem hiding this comment.
I am not sure how accurate this is. Also I don't think this is necessary.
It can be just
You can use {cgu-operator-full} to coordinate reboots across a fleet of spoke clusters when configuration changes require a reboot, such as defered tuning changes.
| * Kubelet configuration updates. | ||
| * Node-level changes delivered through `MachineConfig`, such as `sysctl` settings or system service changes. | ||
|
|
||
| {cgu-operator} ensures that only nodes with a degraded tuned profile, indicating a reboot is needed, are rebooted. |
There was a problem hiding this comment.
this is not true. TALM does not check for this. It reboot the nodes in the selcted mcp on the selected clusters.
Comment on lines
+29
to
+33
| . Create the configuration policies and policy bindings for the tuning or configuration changes. | ||
|
|
||
| . Create a reboot policy on the hub cluster. This policy checks for degraded tuned profiles that indicate a reboot is needed. | ||
|
|
||
| . Create and apply the `ClusterGroupUpgrade` (CGU) custom resource (CR). In the `spec.managedPolicies` list, include all relevant configuration policies first, followed by the reboot policy and the optional `mcp-validator` policy. |
There was a problem hiding this comment.
The procedure should be mentioned:
- The policy can be generated using PolicyGenerator.
- Select
out/argocd/example/acmpolicygenerator/acm-example-sno-rebootorout/argocd/example/acmpolicygenerator/acm-example-multinode-reboot. - change
policyDefaults.placement.labelSelectorto match clusters that you want to reboot. - modify other placeholder values according to your use case.
- in case of rebooting for applying defered tuning change, make sure mcp value matches Tuned.spec.recommended
- follow the procedure (Customizing a managed cluster with PolicyGenerator CRs)[link to section above in the same document]
- After ArgoCD syncs successfully, you can rollout the reboot using TALM by creating a CGU
example:
apiVersion: ran.openshift.io/v1alpha1
kind: ClusterGroupUpgrade
metadata:
name: reboot
namespace: default
spec:
clusterLabelSelectors:
- matchLabels:
# select clusters that you want to reboot
enable: true
managedPolicies:
# you add other configuration policies here, before the reboot policy
- example-reboot
remediationStrategy:
timeout: 300 # timeout across all clusters; you should consider the worst case
maxConcurrency: 10
Comment on lines
+39
to
+40
| * Confirm that all nodes in the `MachineConfigPool` have rebooted. | ||
| * Verify that the `MachineConfigPool` reaches the `Updated` state. |
There was a problem hiding this comment.
Verify successful reboot on a specific node:
oc get mcp master +
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-be5785c3b98eb7a1ec902fef2b81e865 True False False 3 3 3 0 72d
| After you apply the CGU custom resource, {cgu-operator} rolls out the configuration policies in order. When all policies are compliant, {cgu-operator} applies the reboot policy. This triggers a reboot of all nodes in the specified `MachineConfigPool`. | ||
|
|
||
| .Verification | ||
|
|
There was a problem hiding this comment.
You can monitor the rollout on the hub by watching CGU's status. To verify all successful rollout of the reboot:
oc get cgu -A
NAMESPACE NAME AGE STATE DETAILS
default reboot 1d Completed All clusters are compliant with all the managed policies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Version(s):
4.19
Issue:
TELCODOCS-2230
Link to docs preview:
QE review: