OpenShift Cronjobs Automation

Understanding the CronJob Schedule

Every CronJob in OpenShift or Kubernetes uses a cron expression to define how often it runs.

This schedule is specified in the spec.schedule field of the CronJob manifest.

schedule: "* * * * *"

A cron expression consists of five time fields, each representing a unit of time:

*    *    *    *    *
│    │    │    │    └─ Day of the week (0–7, 0 or 7 = Sunday)
│    │    │    └───── Month (1–12)
│    │    └───────── Day of the month (1–31)
│    └───────────── Hour (0–23)
└───────────────── Minute (0–59)

Each field can include:

• * → Any value (every minute, every hour, etc.)

• / → Step values (e.g., */10 for every 10 units)

• , → Lists (e.g., 1,15,30)

• - → Ranges (e.g., 1-5)

Common examples:

Here are some commonly used cron schedules and what they mean:

Schedule	Description
* * * * *	Every minute
*/5 * * * *	Every 5 minutes
*/15 * * * *	Every 15 minutes
0 * * * *	Every hour
0 0 * * *	Every day at midnight
0 9 * * 1-5	Every weekday at 9 AM
0 0 1 * *	On the first day of every month
0 12 */2 * *	Every two days at 12 PM
30 2 * * 0	Every Sunday at 2:30 AM

Tip

You can verify and experiment with cron expressions using crontab.guru, an easy-to-use online cron expression editor.

deployment-culler

This folder contains yaml files that set up a CronJob to scale replicas of Deployment to 0 running in a user's project namespace after a specified period of time.

This CronJob is designed to run periodically according to the schedule defined in spec.schedule in the cronjob.yaml file. It is applied exclusively to deployments within the user's specified namespace and does not affect deployments in other namespaces.

This CronJob performs the following actions:

• This will run every 1 hour (at minute 0), throughout the day, every day.

  spec:
  schedule: '0 * * * *'   # ⏰ Run every 1 hour (at minute 0)

• Scale down Deployed Pods exceeding X hours of runtime: Any pods running for more than X hours under a deployment will be gracefully shut down to conserve resources. The duration is controlled by the CUTOFF_TIME environment variable (in seconds) specified in the cronjob.yaml file - for example, 3600 seconds (1 hour).

  env:
  ...
  - name: CUTOFF_TIME
      value: "3600"   # 1 hour = 3600 seconds

Deployment procedure - STOP PODS BY SCALING REPLICAS FOR DEPLOYMENTS

1. Clone or navigate to this repository.

   To get started, clone the repository using:

   git clone https://github.com/OCP-on-NERC/openshift-cronjobs-automation.git
   cd openshift-cronjobs-automation/cronjobs

2. In the deployment-culler folder, you will find the following YAML files that allow you to easily setup CronJob to your own namespace:

   • serviceaccount.yaml: Defines a ServiceAccount that provides the necessary permissions for the CronJob to run within your namespace.

   • rolebinding.yaml: Creates a RoleBinding that assigns the required roles to the ServiceAccount, allowing it to perform actions on resources in your namespace.

   • cronjob.yaml: Sets up a CronJob that periodically stops RHOAI workbenches in your namespace according to the defined schedule.

   VERY IMPORTANT NOTE: Ensure that the schedule defined under spec.schedule and the environment variable CUTOFF_TIME (in seconds) are correctly set or adjusted as needed in the cronjob.yaml file.

3. To add resources to your own namespace:

   i. Ensure you are logged in to your OpenShift account via the CLI and have access to your namespace.

   ii. Switch to your project namespace where Deployment is running by running the following oc command:

   oc project <YOUR_NAMESPACE>

   iii. Verify that the following oc command displays the correct project namespace, i.e., <YOUR_NAMESPACE> is correct:

   oc project
   
   Using project "<YOUR_NAMESPACE>" on server "https://api.shift.nerc.mghpcc.org:6443".

4. From cronjobs directory, you can run this oc command: oc apply -f ./deployment-culler/. to execute all of the above described YAML files located in the deployment-culler folder at once.

   oc apply -f ./deployment-culler/.
   
   cronjob.batch/deployment-culler created
   rolebinding.rbac.authorization.k8s.io/deployment-culler created
   serviceaccount/deployment-culler created

   This will deploy all the necessary resources for the CronJob to run on the specified schedule.

   NOTE: To delete all resources if not necessary just run oc delete -f ./deployment-culler/..

Alternatively, to run the script immediately:

1. Ensure you followed the steps above.

2. Verify the cronjob deployment-culler exists:

   oc get cronjob deployment-culler

3. Run:

   kubectl create job --from=cronjob/deployment-culler deployment-culler

   This will trigger the cronjob to spawn a job manually.

nb-culler

This folder contains yaml files that set up a CronJob to stop RHOAI Workbenches running in a user's project namespace after a specified period of time.

This CronJob is designed to run periodically according to the schedule defined in spec.schedule in the cronjob.yaml file. It is applied exclusively to notebooks within the user's specified namespace and does not affect notebooks in other namespaces.

This CronJob performs the following actions:

• This will run every 15 minutes, throughout the day, every day.

  spec:
  schedule: '*/15 * * * *'  # ⏰ Run every 15 minutes

• Scale down Deployed Pods exceeding X hours of runtime: Any pods running for more than X hours under a deployment will be gracefully shut down to conserve resources. The duration is controlled by the CUTOFF_TIME environment variable (in seconds) specified in the cronjob.yaml file - for example, 900 seconds (15 minutes). Persistent Volume Claims (PVCs) ensure that all data is preserved during the shutdown process.

  env:
  ...
  - name: CUTOFF_TIME
      value: "900"  # 15 minutes = 900 seconds

Deployment procedure - STOP RHOAI WORKBENCH

1. Clone or navigate to this repository.

   To get started, clone the repository using:

   git clone https://github.com/OCP-on-NERC/openshift-cronjobs-automation.git
   cd openshift-cronjobs-automation/cronjobs

2. In the nb-culler folder, you will find the following YAML files that allow you to easily setup CronJob to your own namespace:

   • serviceaccount.yaml: Defines a ServiceAccount that provides the necessary permissions for the CronJob to run within your namespace.

   • rolebinding.yaml: Creates a RoleBinding that assigns the required roles to the ServiceAccount, allowing it to perform actions on resources in your namespace.

   • cronjob.yaml: Sets up a CronJob that periodically stops RHOAI workbenches in your namespace according to the defined schedule.

   VERY IMPORTANT NOTE: Ensure that the schedule defined under spec.schedule and the environment variable CUTOFF_TIME (in seconds) are correctly set or adjusted as needed in the cronjob.yaml file.

3. To add resources to your own namespace:

   i. Ensure you are logged in to your OpenShift account via the CLI and have access to your namespace.

   ii. Switch to your project namespace for the Data Science Project (DSP) where RHOAI workbenches are running by running the following oc command:

   oc project <YOUR_NAMESPACE>

   iii. Verify that the following oc command displays the correct project namespace, i.e., <YOUR_NAMESPACE> is correct:

   oc project
   
   Using project "<YOUR_NAMESPACE>" on server "https://api.shift.nerc.mghpcc.org:6443".

4. From cronjobs directory, you can run this oc command: oc apply -f ./nb-culler/. to execute all of the above described YAML files located in the nb-culler folder at once.

   oc apply -f ./nb-culler/.
   
   cronjob.batch/nb-culler created
   rolebinding.rbac.authorization.k8s.io/nb-culler created
   serviceaccount/nb-culler created

   This will deploy all the necessary resources for the CronJob to run on the specified schedule.

   NOTE: To delete all resources if not necessary just run oc delete -f ./nb-culler/..

Alternatively, to run the script immediately:

1. Ensure you followed the steps above.

2. Verify the cronjob nb-culler exists:

   oc get cronjob nb-culler

3. Run:

   kubectl create job --from=cronjob/nb-culler nb-culler

   This will trigger the cronjob to spawn a job manually.

---