Adding taint to block deployment to drained nodes

Cilium-operator pod automatically rescheduled onto drained mode.
This can cause to block some kubernete upgrades.

Signed-off-by: Murat Parlakisik <parlakisik@gmail.com>

Author: parlakisik

Documentation/helm-values.rst

@@ -418,6 +418,9 @@ Helm Options
   ``ciliumEndpointSlice.enabled`` instead to enable CiliumEndpointSlices.
 * ``localRedirectPolicy`` helm option has been deprecated. Set ``localRedirectPolicies.enabled`` instead.
 * The new ``localRedirectPolicies.addressMatcherCIDRs`` option can be used to limit what addresses are allowed in an address match of a CiliumLocalRedirectPolicy.
+* The default value for ``operator.tolerations`` has been narrowed to only include the following tolerations:
+   ``node-role.kubernetes.io/control-plane`` , ``node-role.kubernetes.io/master`` , ``node.kubernetes.io/not-ready`` and ``node.cilium.io/agent-not-ready``. This will 
+   block the operator running on drained nodes. 
 
 Agent Options
 ~~~~~~~~~~~~~

Documentation/operations/upgrade.rst

@@ -4790,14 +4790,50 @@
             "anyOf": [
               {
                 "properties": {
+                  "key": {
+                    "type": "string"
+                  },
+                  "operator": {
+                    "type": "string"
+                  }
+                }
+              },
+              {
+                "properties": {
+                  "key": {
+                    "type": "string"
+                  },
+                  "operator": {
+                    "type": "string"
+                  }
+                }
+              },
+              {
+                "properties": {
+                  "key": {
+                    "type": "string"
+                  },
+                  "operator": {
+                    "type": "string"
+                  }
+                }
+              },
+              {
+                "properties": {
+                  "key": {
+                    "type": "string"
+                  },
                   "operator": {
                     "type": "string"
                   }
                 }
               }
             ]
           },
-          "type": "array"
+          "type": [
+            "null",
+            "array"
+          ]
         },
         "topologySpreadConstraints": {
           "items": {},

install/kubernetes/cilium/README.md

@@ -2905,12 +2905,18 @@ operator:
     kubernetes.io/os: linux
   # -- Node tolerations for cilium-operator scheduling to nodes with taints
   # ref: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
-  tolerations:
-    - operator: Exists
-      # - key: "key"
-      #   operator: "Equal|Exists"
-      #   value: "value"
-      #   effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"
+  # @schema
+  # type: [null, array]
+  # @schema
+  tolerations: 
+    - key: "node-role.kubernetes.io/control-plane"
+      operator: Exists
+    - key: "node-role.kubernetes.io/master" #deprecated
+      operator: Exists
+    - key: "node.kubernetes.io/not-ready"
+      operator: Exists
+    - key: "node.cilium.io/agent-not-ready"
+      operator: Exists
   # -- Additional cilium-operator container arguments.
   extraArgs: []
   # -- Additional cilium-operator environment variables.