New Feature - Downscaling Jobs Using Admission Controller (Kyverno And Gatekeeper) (#42)

* added downscaling jobs feature using admission controller (kyverno and gatekeeper). Refactored docs in order to explain how to use the new feature and how to use matching_labels args

* added downscaling jobs feature using admission controller (kyverno and gatekeeper). Refactored docs in order to explain how to use the new feature and how to use matching_labels. Rebased args

* small syntax fix after rebase

* added doc for downscaling daemonset feature

Author: samuel-esp

README.md

@@ -267,7 +267,7 @@ Available command line options:
 `--include-resources`
 
 :   Downscale resources of this kind as comma separated list.
-    \[deployments, statefulsets, stacks, horizontalpodautoscalers, cronjobs, daemonsets, rollouts, scaledobjects\]
+    \[deployments, statefulsets, stacks, horizontalpodautoscalers, cronjobs, daemonsets, rollouts, scaledobjects, jobs\]
     (default: deployments)
 
 `--grace-period`
@@ -340,6 +340,134 @@ Available command line options:
     For backwards compatibility, if this argument is not specified,
     py-kube-downscaler will apply to all resources.
 
+`--admission-controller`
+
+:   Optional: admission controller used by the kube-downscaler to downscale and upscale
+    jobs. Required only if "jobs" are specified inside "--include-resources" arg. 
+    Supported Admission Controllers are
+    \[gatekeeper, kyverno*\]
+
+    *Make sure to read the dedicated section below to understand how to use the
+    --admission-controller feature correctly
+
+### Scaling Jobs
+
+Before scaling jobs make sure the Admission Controller of your choice is correctly installed inside the cluster. 
+Kube-Downscaler performs some health checks that are displayed inside logs when the `--debug` arg is present. 
+If you are using Gatekeeper, Kube-Downscaler will install a new Custom Resource Definition
+called `kubedownscalerjobsconstraint`
+
+**When using this feature you need to exclude Kyverno or Gatekeeper resources from downscaling otherwise 
+the admission controller pods won't be able to donwscale jobs.** 
+You can use `EXCLUDE_NAMESPACES` environment variable or `--exclude-namespaces` arg to exclude `"kyverno"` or `"gatekeeper-system"` namespaces. 
+To have a more fine-grained control you can use `EXCLUDE_DEPLOYMENTS` environment variable
+or `--exclude-deployments` arg to exclude only certain resources inside `"kyverno"` or `"gatekeeper-system"` namespaces
+
+**<u>Important</u>:** Jobs started from CronJobs are excluded by default unless you have included `cronjobs` inside `--include-resources` argument
+
+**Annotations:** both the `downscaler/exclude` and `downscaler/exclude-until` annotations are fully supported 
+inside jobs to exclude them from downscaling. However, when using `downscaler/exclude-until`, the time <u>**must**</u> be specified in the RFC format `YYYY-MM-DDT00:00:00Z`
+otherwise the exclusion won't work. 
+Please check the example below
+
+```yaml {.sourceCode .yaml}
+apiVersion: batch/v1
+kind: Job
+metadata:
+  namespace: default
+  name: testjob
+  annotations:
+    downscaler/exclude-until: "2024-01-31T00:00:00Z"   
+spec:
+  template:
+    spec:
+      containers:
+      - image: nginx
+        name: testjob
+      restartPolicy: Never
+```
+
+**Arguments and Env:** you can also use `EXCLUDE_DEPLOYMENTS` environment variable or the argument `--exclude-deployments`
+to exclude jobs. As described above, despite their names, these variables work for any type of workload
+
+**<u>Important</u>:** 
+`downscaler/downscale-period`, `downscaler/downtime`, `downscaler/upscale-period`, `downscaler/uptime` 
+annotations are not supported if specified directly inside the Job definition due to limitations 
+on computing days of the week inside the policies. However you can still use 
+these annotations at Namespace level to downscale/upscale Jobs 
+
+
+**Deleting Policies:** if for some reason you want to delete all resources blocking jobs, you can use these commands:
+
+Gatekeeper
+
+``` {.sourceCode .sh}
+kubectl delete constraints -A -l origin=kube-downscaler
+```
+
+Kyverno
+
+``` {.sourceCode .sh}
+kubectl delete policies -A -l origin=kube-downscaler
+```
+
+### Scaling DaemonSet
+
+The feature to scale DaemonSets can be very useful for reducing the base occupancy of a node. If enabled, the DaemonSets downscaling algorithm works like this:
+
+1) Downtime Hours: Kube Downscaler will add to each targeted DaemonSet a Node Selector that cannot be satisfied `kube-downscaler-non-existent=true`
+2) Uptime Hours: Kube Downscaler will remove the `kube-downscaler-non-existent=true` Node Selector from each targeted DaemonSet
+
+### Matching Labels Argument
+
+Labels, in Kubernetes, are key-value pairs that can be used to identify and group resources.
+
+You can use the `--matching-labels` argument to include only certain resources in the namespaces
+that are targeted by the Kube Downscaler. inside this argument you can specify:
+- labels written in this format [key=value]
+- regular expressions that target this format [key=value].
+
+Each entry must be separated by a comma (`,`). If multiple entries are specified, the Kube Downscaler evaluates them as an OR condition
+
+To make it more clear, given the following resource
+
+```yaml {.sourceCode .yaml}
+kind: Deployment
+metadata:
+  labels:
+    app: nginx
+    type: example
+  name: nginx
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: nginx
+  template:
+    metadata:
+      labels:
+        app: nginx
+    spec:
+      containers:
+      - image: nginx
+        name: nginx
+```
+
+Kube-Downscaler will evaluate the input of the `--matching-labels` argument against _app=nginx_
+and _type=example_. If at least one of the two key-value pairs matches the resource will be downscaled
+
+Example of valid inputs are:
+
+`--matching-labels=hello=world`: if the resource has a label "hello" equals to "world" it will be downscaled
+
+`--matching-labels=hello=world,version=2.0`: if the resource has a label "hello" equals to "world" 
+or a label "version" equal to "2.0" it will be downscaled
+
+`--matching-labels=^it-plt.*`: if the resource has a label that starts with "it-plt" it will be downscaled
+
+`--matching-labels=^it-plt.*,not-critical=true`: if the resource has a label that starts with "it-plt" or a label
+"not-critical" equals to "true" it will be downscaled
+
 ### Namespace Defaults
 
 `DEFAULT_UPTIME`, `DEFAULT_DOWNTIME`, `FORCE_UPTIME` and exclusion can

chart/templates/rbac.yaml

@@ -100,6 +100,59 @@ rules:
     - list
     - update
     - patch
+- apiGroups:
+  - constraints.gatekeeper.sh
+  resources:
+  - kubedownscalerjobsconstraint
+  verbs:
+  - get
+  - create
+  - watch
+  - list
+  - update
+  - patch
+  - delete
+- apiGroups:
+  - kyverno.io
+  resources:
+  - policies
+  resourceNames:
+  - kube-downscaler-jobs-policy
+  verbs:
+  - get
+  - create
+  - watch
+  - list
+  - update
+  - patch
+  - delete
+- apiGroups:
+  - kyverno.io
+  resources:
+  - policies
+  verbs:
+  - get
+  - create
+  - watch
+  - list
+- apiGroups:
+  - templates.gatekeeper.sh
+  resources:
+  - constrainttemplate
+  verbs:
+  - create
+  - get
+  - list
+  - watch
+- apiGroups:
+  - apiextensions.k8s.io
+  resources:
+  - customresourcedefinitions
+  verbs:
+  - create
+  - get
+  - list
+  - watch
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding

kube_downscaler/cmd.py

@@ -10,6 +10,7 @@
         "horizontalpodautoscalers",
         "rollouts",
         "scaledobjects",
+        "jobs",
         "daemonsets",
         "poddisruptionbudgets",
     ]
@@ -106,4 +107,9 @@ def get_parser():
         default=os.getenv("MATCHING_LABELS", ""),
         help="Apply downscaling to resources with the supplied labels. This is a comma-separated list of regex patterns. This is optional, downscaling will be applied to all resources by default.",
     )
+    parser.add_argument(
+        "--admission-controller",
+        default=os.getenv("ADMISSION_CONTROLLER", ""),
+        help="Apply downscaling to jobs using the supplied admission controller. Jobs should be included inside --include-resources if you want to use this parameter. kyverno and gatekeeper are supported.",
+    )
     return parser

kube_downscaler/main.py

@@ -31,6 +31,7 @@ def main(args=None):
         args.namespace,
         args.include_resources,
         args.matching_labels,
+        args.admission_controller,
         args.upscale_period,
         args.downscale_period,
         args.default_uptime,
@@ -51,6 +52,7 @@ def run_loop(
     namespace,
     include_resources,
     matching_labels,
+    admission_controller,
     upscale_period,
     downscale_period,
     default_uptime,
@@ -80,6 +82,7 @@ def run_loop(
                 exclude_deployments=frozenset(exclude_deployments.split(",")),
                 dry_run=dry_run,
                 grace_period=grace_period,
+                admission_controller=admission_controller,
                 downtime_replicas=downtime_replicas,
                 deployment_time_annotation=deployment_time_annotation,
                 enable_events=enable_events,

kube_downscaler/resources/constraint.py

@@ -0,0 +1,38 @@
+from pykube.objects import APIObject
+
+
+class KubeDownscalerJobsConstraint(APIObject):
+
+    """Support the Gatakeeper Admission Controller Custom CRDs (https://open-policy-agent.github.io/gatekeeper/website/docs)."""
+
+    version = "constraints.gatekeeper.sh/v1beta1"
+    endpoint = "kubedownscalerjobsconstraint"
+    kind = "KubeDownscalerJobsConstraint"
+
+    @property
+    def namespaces_list(self):
+        return self.obj["spec"]["match"].get("namespaces", [])
+
+    @namespaces_list.setter
+    def namespaces_list(self, updated_list):
+        self.obj["spec"]["match"]["namespaces"] = updated_list
+
+    @staticmethod
+    def create_job_constraint(resource_name):
+        obj = {
+            "apiVersion": "constraints.gatekeeper.sh/v1beta1",
+            "kind": "KubeDownscalerJobsConstraint",
+            "metadata": {
+                "name": resource_name,
+                "labels": {
+                    "origin": "kube-downscaler"
+                }
+            },
+            "spec": {
+                "match": {
+                    "namespaces": [resource_name]
+                }
+            }
+        }
+
+        return obj

kube_downscaler/resources/constrainttemplate.py

@@ -0,0 +1,105 @@
+from pykube.objects import APIObject
+
+from kube_downscaler.helper import logger
+
+
+class ConstraintTemplate(APIObject):
+
+    """Support the Gatakeeper Admission Controller Custom CRDs (https://open-policy-agent.github.io/gatekeeper/website/docs)."""
+
+    version = "templates.gatekeeper.sh/v1"
+    endpoint = "constrainttemplates"
+    kind = "ConstraintTemplate"
+
+    @staticmethod
+    def create_constraint_template_crd(excluded_jobs, matching_labels):
+
+        excluded_jobs_regex = '^(' + '|'.join(excluded_jobs) + ')$'
+
+        # For backwards compatibility, if the matching_labels FrozenSet has an empty string as the first element,
+        # we don't ignore anything
+        first_element = next(iter(matching_labels), None)
+        first_element_str = first_element.pattern
+
+        if first_element_str == "":
+            logger.debug("Matching_labels arg set to empty string: all resources are considered in the scaling process")
+            matching_labels_arg_is_present = False
+        else:
+            matching_labels_arg_is_present = True
+
+        if matching_labels_arg_is_present:
+            matching_labels_rego_string: str = "\n"
+            for pattern in matching_labels:
+                matching_labels_rego_string = matching_labels_rego_string + "    has_matched_labels(\"" + pattern.pattern + "\", input.review.object.metadata.labels)\n"
+        else:
+            matching_labels_rego_string: str = ""
+
+        rego = """
+        package kubedownscalerjobsconstraint
+
+        violation[{"msg": msg}] {
+            input.review.kind.kind == "Job"
+            not exist_owner_reference
+            not exact_match(\"""" + excluded_jobs_regex + """\", input.review.object.metadata.name)
+            not has_exclude_annotation
+            not is_exclude_until_date_reached""" + matching_labels_rego_string + """
+            msg := "Job creation is not allowed in this namespace during a kube-downscaler downtime period."
+        }
+
+        exact_match(pattern, name) {
+            regex.match(pattern, name)
+        }
+
+        exist_owner_reference {
+	        input.review.object.metadata.ownerReferences
+        }
+
+        has_exclude_annotation {
+            input.review.object.metadata.annotations["downscaler/exclude"] = "true"
+        }
+
+        is_exclude_until_date_reached {
+            until_date_str := input.review.object.metadata.annotations["downscaler/exclude-until"]
+            parsed_until_date := time.parse_rfc3339_ns(until_date_str)
+            current_utc := time.now_ns()
+            parsed_until_date >= current_utc
+        }
+
+        has_matched_labels(pattern, labels) {
+            some k
+            value := labels[k]
+            key_equals_contact := concat("", [k, "="])
+            equals_value_contact := concat("", [key_equals_contact, value])
+            regex.match(pattern, equals_value_contact)
+        }        
+        """
+
+        obj = {
+            "apiVersion": "templates.gatekeeper.sh/v1",
+            "kind": "ConstraintTemplate",
+            "metadata": {
+                "name": "kubedownscalerjobsconstraint",
+                "annotations": {
+                    "metadata.gatekeeper.sh/title": "Kube Downscaler Jobs Constraint",
+                    "metadata.gatekeeper.sh/version": "1.0.0",
+                    "description": "Policy to downscale jobs in certain namespaces."
+                }
+            },
+            "spec": {
+                "crd": {
+                    "spec": {
+                        "names": {
+                            "kind": "KubeDownscalerJobsConstraint"
+                        }
+                    }
+                },
+                "targets": [
+                    {
+                        "target": "admission.k8s.gatekeeper.sh",
+                        "rego": rego
+                    }
+                ]
+            }
+        }
+
+        return obj